Electronic apparatus and control method thereof

Abstract

Disclosed are an electronic device and a control method thereof. The electronic device may include: a communication unit; a camera; a projection unit; and a processor which receives state information indicating the operation state of an external device through the communication unit, acquires related information about the external device from a memory, identifies at least one area among an area on a surface of the external device and an area around the external device on the basis of an image of the external device acquired through the camera, identifies a projection area, on which a screen related to the received state information is to be projected, among the at least one area on the basis of the acquired related information about the external device, and controls the projection unit to project the screen on the identified projection area.

Description

Electronic device and method of controlling the same

[0001] The present disclosure relates to an electronic device and a method for controlling the electronic device, and more particularly, to an electronic device capable of projecting an image without location constraints and a method for controlling the same.

[0002] Recently, technological advancements have been made in electronic devices capable of projecting images, so-called projectors, and in particular, movable projectors have recently been attracting attention.

[0003] Because portable projectors can be moved freely, they can project images without any location constraints. However, because they can project images without any location constraints, determining when, where, and how to project the image becomes a crucial issue.

[0004] Meanwhile, since mobile projectors can project images without location constraints, they can also project images containing information about home appliances onto other devices within the space where they are located, such as the surfaces of home appliances or their surroundings. However, since the types of information about home appliances vary widely, and the types of home appliances themselves also vary widely, when projecting images containing information about home appliances, it is necessary to consider both the types of information about home appliances and the types of home appliances themselves.

[0005] For example, even if an appliance contains a display, projecting information about the appliance onto its surface or surroundings may not be desirable. Conversely, if an appliance contains a display, but its size and the type of information it can display are limited, projecting information about the appliance onto its surface or surroundings may be desirable. Furthermore, the appropriate timing, projection area, and method for projecting information may vary depending on the type of information about the appliance being projected.

[0006] Meanwhile, since there may be various other devices and more than one user within the space where the mobile projector is located, the presence of the user and the interaction between the user and the various devices within the space where the mobile projector is located can also be considered as considerations related to the projection of the image.

[0007] Despite technological advancements in portable projectors, conventional technologies have not sufficiently addressed the aforementioned issues, and thus, they are criticized for their limitations in effectively providing images and information that align with the user's intentions and intuition.

[0008] The present disclosure is intended to overcome the limitations of the prior art as described above, and an object of the present disclosure is to provide an electronic device and a control method thereof that can provide information that matches the user's intention and intuition in an effective and efficient manner while projecting an image without location restrictions.

[0009] Aspects of embodiments of the present disclosure will be set forth in part in the description that follows, and in part will be obvious from the description or may be learned by practice of the embodiments presented.

[0010] According to one embodiment of the present disclosure, an electronic device includes a communication unit, a camera, a projection unit, and a processor that receives status information indicating an operating status of an external device through the communication unit, acquires related information of the external device from a memory, identifies at least one area among an area on a surface of the external device and an area around the external device based on an image of the external device acquired through the camera, identifies a projection area on which to project a screen related to the received status information among the at least one area based on the acquired related information of the external device, and controls the projection unit to project the screen on the identified projection area.

[0011] According to one embodiment of the present disclosure, the processor identifies the projection area among the at least one area based on the acquired related information and the received status information, and the acquired related information may include at least one of information for identifying the external device, information on the type of the external device, information on the surface of the external device, information on the main operation of the external device, and information on the configuration included in the external device.

[0012] According to one embodiment of the present disclosure, the processor may identify a threshold size for the identified projection area based on at least one of the amount of the received status information and the importance of the received status information, and if the size of the surface of the external device is greater than or equal to the identified threshold size, the processor may identify an area on the surface of the external device as the identified projection area.

[0013] According to one embodiment of the present disclosure, the processor may identify, if the size of the surface of the external device is less than the identified threshold size, and if the external device includes a display, an area adjacent to the display among the at least one area around the external device as the identified projection area, and if the size of the surface is less than the threshold size, and if the external device does not include the display, an area adjacent to a component of the external device that performs the primary operation among the at least one area around the external device may be identified as the projection area.

[0014] According to one embodiment of the present disclosure, the processor can identify a configuration related to the received status information among the configurations included in the external device based on information about the configuration included in the acquired related information, and control the projection unit to project the screen at a position corresponding to the identified configuration in the identified projection area.

[0015] According to one embodiment of the present disclosure, the processor identifies whether to project the screen through the projection unit based on the received status information and the acquired related information, and if it is determined not to project the screen, controls the communication unit to transmit the received status information to the user terminal, and if it is determined to project the screen, it can acquire an image of the external device through the camera.

[0016] According to one embodiment of the present disclosure, the processor can identify whether to project the screen through the projection unit based on the received status information, depending on whether the importance of the received status information corresponds to an emergency.

[0017] According to one embodiment of the present disclosure, the device further comprises a driving unit, and the processor can control the driving unit and the projection unit to project the screen onto the identified projection area.

[0018] According to one embodiment of the present disclosure, the processor acquires an image of a user through the camera, acquires information about the user based on the image of the user, and identifies the identified projection area among the at least one area based on the acquired information about the user, the acquired related information, and the received status information, wherein the acquired information about the user may include at least one of information about a location of the user, information about a direction in which the user's head is looking, information about a distance between the user and the external device, and information about a lifestyle pattern of the user.

[0019] According to one embodiment of the present disclosure, if the direction in which the user's head is looking is maintained for a preset period of time, the processor identifies an area corresponding to the direction in which the user's head is looking as the projection area, identifies whether the external device is in a state of performing a preset operation based on the received state information, and if the external device is in a state of performing the operation, controls the driving unit and the projection unit to project the screen including information on the progress of the operation, and if the external device is in a state of not performing the operation, controls the driving unit and the projection unit to project the screen including information for controlling the operation of the external device.

[0020] According to one embodiment of the present disclosure, when the processor identifies that the received status information is to be displayed on a plurality of screens based on the received status information, the processor controls the driving unit and the projection unit to project a first screen among the plurality of screens onto the identified projection area, and when the processor identifies that the user has performed an action corresponding to the first screen based on the image of the user acquired while the first screen among the plurality of screens is projected, the processor controls the driving unit and the projection unit to project a second screen subsequent to the first screen among the plurality of screens.

[0021] According to one embodiment of the present disclosure, a control method of an electronic device including a communication unit, a camera, and a projection unit includes the steps of receiving status information indicating an operating status of an external device through the communication unit, obtaining related information of the external device from a memory, identifying at least one area among an area on a surface of the external device and an area around the external device based on an image of the external device obtained through the camera, identifying a projection area on which to project a screen related to the received status information among the at least one area based on the obtained related information of the external device, and projecting the screen on the identified projection area.

[0022] According to one embodiment of the present disclosure, the step of identifying the identified projection area includes the step of identifying the identified projection area among the at least one area based on the acquired related information and the received status information, wherein the acquired related information may include at least one of information for identifying the external device, information on the type of the external device, information on the surface of the external device, information on the main operation of the external device, and information on the configuration included in the external device.

[0023] According to one embodiment of the present disclosure, the step of identifying the identified projection area may further include the step of identifying a threshold size for the identified projection area based on at least one of the amount of the received status information and the importance of the received status information, and the step of identifying an area on the surface of the external device as the identified projection area if the size of the surface of the external device is greater than or equal to the identified threshold size.

[0024] According to one embodiment of the present disclosure, the step of identifying the identified projection area may further include: if the size of the surface of the external device is less than the identified threshold size, and if the external device includes a display, identifying an area adjacent to the display among the at least one area around the external device as the identified projection area; and if the size of the surface of the external device is less than the threshold size, and if the external device does not include the display, identifying an area adjacent to a component of the external device that performs the main operation among the at least one area around the external device as the identified projection area.

[0025] According to one or more embodiments of the present disclosure for achieving the above-described object, a method for controlling an electronic device may include: receiving status information indicating an operating status of the external device; identifying at least one area among an area on a surface of the external device and at least one area around the external device based on an image of the external device; identifying a projection area for providing a screen related to the status information among the at least one area based on related information of the external device; and projecting the screen onto the projection area.

[0026] Meanwhile, the step of identifying the projection area includes a step of identifying the projection area among the at least one area based on the related information and the status information, and the related information may include at least one of information for identifying the external device, information on the type of the external device, information on the surface of the external device, information on the main operation of the external device, and information on the configuration included in the external device.

[0027] Meanwhile, the step of identifying the projection area may further include a step of identifying a threshold size for the projection area based on at least one of the amount of the state information and the importance of the state information, and a step of identifying an area on the surface among the at least one area as the projection area if the size of the surface is greater than or equal to the threshold size.

[0028] Meanwhile, the step of identifying the projection area may further include a step of identifying an area adjacent to the display among the at least one area as the projection area if the size of the surface is less than the threshold size and the external device includes a display, and a step of identifying an area adjacent to a configuration of the external device that performs the main operation among the at least one area as the projection area if the size of the surface is less than the threshold size and the external device does not include the display.

[0029] Meanwhile, the control method of the electronic device may further include a step of identifying a configuration of the external device related to the status information based on the status information and information about a configuration included in the external device, and a step of projecting the screen at a position corresponding to the identified configuration in the projection area.

[0030] Meanwhile, the control method of the electronic device may further include a step of identifying whether to project the screen based on the status information and the related information, a step of transmitting the status information to a user terminal if the screen is identified as not being projected, and a step of acquiring the image if the screen is identified as being projected.

[0031] Meanwhile, the control method of the electronic device further includes a step of obtaining an image of a user and a step of obtaining information about the user based on the image of the user, and the step of identifying the projection area includes a step of identifying the projection area among the at least one area based on the information about the user, the related information, and the status information, and the information about the user may include at least one of information about the location of the user, information about the direction of the user's head, information about the distance between the user and the external device, and information about the user's lifestyle pattern.

[0032] Meanwhile, the step of identifying the projection area may further include a step of identifying an area corresponding to the direction of the user's head as the projection area when the direction of the user's head is maintained for a preset period of time, and the control method of the electronic device may further include a step of identifying whether the external device is in a state of performing a preset operation based on the state information, a step of projecting the screen including information on the progress of the operation when the external device is in a state of performing the operation, and a step of projecting the screen including information for controlling the operation of the external device when the external device is in a state of not performing the operation.

[0033] Meanwhile, the control method of the electronic device may further include a step of projecting a first screen among the plurality of screens onto the projection area if the state information is identified as being displayed on a plurality of screens based on the state information, and a step of projecting a second screen subsequent to the first screen if the user is identified as having performed an action corresponding to the first screen based on an image of the user acquired while the first screen is being projected.

[0034] Other aspects, features and advantages of one or more embodiments according to the present disclosure will become more apparent from the following detailed description taken in conjunction with the accompanying drawings.

[0035] FIG. 1 is a drawing for explaining an electronic device and an external device according to the present disclosure;

[0036] FIG. 2 is a block diagram briefly illustrating the configuration of an electronic device according to one or more embodiments;

[0037] FIG. 3 is a drawing illustrating an operation of projecting a screen onto an area on a surface of an external device according to one or more embodiments;

[0038] FIG. 4 is a flowchart illustrating one or more embodiments related to identifying a projection area based on related information and status information;

[0039] FIG. 5 is a diagram illustrating an operation of projecting a screen onto an area on a surface adjacent to an external device among at least one area according to one or more embodiments;

[0040] FIG. 6 is a flowchart illustrating one or more embodiments related to identifying whether to project a screen based on status information and related information;

[0041] FIG. 7 is a flowchart illustrating one or more embodiments related to identifying a projection area and projecting a screen based on relevant information and information about a user;

[0042] Figures 8 and 9 are diagrams showing a process of identifying a projection area based on information about the user's head direction.

[0043] FIG. 10 and FIG. 11 are drawings for explaining one or more embodiments related to the size of the screen and the shape of the screen;

[0044] FIGS. 12 to 14 are drawings for explaining one or more embodiments related to sequentially projecting multiple screens and identifying projection positions of the screens within a projection area;

[0045] FIG. 15 is a block diagram detailing the configuration of an electronic device according to one or more embodiments; and

[0046] FIG. 16 is a flowchart illustrating a method of controlling an electronic device according to one or more embodiments.

[0047] The present embodiments may be modified and have various embodiments. Specific embodiments are illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the scope to specific embodiments, but should be understood to encompass various modifications, equivalents, and / or alternatives of the embodiments of the present disclosure. In connection with the description of the drawings, similar reference numerals may be used for similar components.

[0048] In describing the present disclosure, if it is determined that a specific description of a related known function or configuration may unnecessarily obscure the gist of the present disclosure, a detailed description thereof will be omitted.

[0049] Additionally, the following embodiments may be modified in various other forms, and the scope of the technical concepts of the present disclosure is not limited to the following embodiments. Rather, these embodiments are provided to further faithfully and completely convey the technical concepts of the present disclosure to those skilled in the art.

[0050] The terminology used in this disclosure is for the purpose of describing specific embodiments only and is not intended to limit the scope of the rights. Singular expressions include plural expressions unless the context clearly dictates otherwise.

[0051] In this disclosure, expressions such as “has,” “can have,” “includes,” or “may include” indicate the presence of a corresponding feature (e.g., a component such as a number, function, operation, or part), and do not exclude the presence of additional features.

[0052] In this disclosure, expressions such as “A or B,” “at least one of A and / or B,” or “one or more of A or / and B” can include all possible combinations of the listed items. For example, “A or B,” “at least one of A and B,” or “at least one of A or B” can all refer to instances where (1) at least one A is included, (2) at least one B is included, or (3) both at least one A and at least one B is included. As another example, “at least one of A, B, or C” or “at least one of A, B, and C” can all refer to instances where (1) at least one A is included, (2) at least one B is included, (3) at least one C is included, (4) at least one A and at least one B is included, (5) at least one A and at least one C is included, (6) at least one B and at least one C is included, or (7) at least one A and at least one B and at least one C is included.

[0053] The expressions “first,” “second,” “first,” or “second,” etc., used in this disclosure can describe various components, regardless of order and / or importance, and are only used to distinguish one component from another, but do not limit the components.

[0054] When it is said that a component (e.g., a first component) is “(operatively or communicatively) coupled with / to” or “connected to” another component (e.g., a second component), it should be understood that said component may be directly coupled to said other component, or may be coupled via another component (e.g., a third component).

[0055] On the other hand, when it is said that a component (e.g., a first component) is "directly connected" or "directly connected" to another component (e.g., a second component), it can be understood that no other component (e.g., a third component) exists between said component and said other component.

[0056] The expression "configured to" as used in the present disclosure may be used interchangeably with, for example, "suitable for," "having the capacity to," "designed to," "adapted to," "made to," or "capable of." The term "configured to" may not necessarily mean only "specifically designed to" in terms of hardware.

[0057] Instead, in some contexts, the phrase "a device configured to" may mean that the device, in conjunction with other devices or components, is "capable of" performing A, B, and C. For example, the phrase "a processor configured (or set) to perform A, B, and C" may refer to a dedicated processor (e.g., an embedded processor) for performing those operations, or a general-purpose processor (e.g., a CPU or application processor) that can perform those operations by executing one or more software programs stored in a memory device.

[0058] In the embodiments, a 'module' or 'part' performs at least one function or operation, and may be implemented as hardware or software, or as a combination of hardware and software. Furthermore, a plurality of 'modules' or 'parts' may be integrated into at least one module and implemented as at least one processor, except for a 'module' or 'part' that needs to be implemented as a specific hardware.

[0059] Meanwhile, the various elements and areas in the drawings are schematically drawn. Therefore, the technical concept of the present invention is not limited by the relative sizes or spacing depicted in the attached drawings.

[0060] FIG. 1 is a drawing for explaining an electronic device (100) and an external device (200) according to the present disclosure.

[0061] The term "electronic device (100)" refers to a device that is movable and capable of projecting images. Specifically, the electronic device (100) can autonomously navigate a space (e.g., a user's home) in which the electronic device (100) is located, and can also move to a destination selected by the user or identified by the electronic device (100). In addition, the electronic device (100) can project an image onto a projection area (or projection surface) selected by the user or identified by the electronic device (100) within the space in which the electronic device (100) is located. Accordingly, the electronic device (100) may also be referred to as a "movable projector," a "projection robot," or an "autonomous driving projector."

[0062] Meanwhile, the fact that the electronic device (100) is 'movable' may include not only that the electronic device (100) can autonomously drive or move to a destination using the driving unit (130) of the electronic device (100), but also that the location of the electronic device (100) can be moved by the user. In other words, the electronic device (100) can project an image onto a projection area within the space where the electronic device (100) is located after the location is moved by the user. Accordingly, the electronic device (100) may be a 'portable projector' or a 'small projector' rather than a 'fixed projector', and may also be a 'portable terminal including a projector'.

[0063] FIG. 1 illustrates an electronic device (100) having a body similar to a spherical shape, including two wheels on both sides, and including a configuration for projecting an image on the front, but the shape of the electronic device (100) is not limited to the shape illustrated in FIG. 1.

[0064] An 'external device (200)' refers to a device other than the electronic device (100) placed in the space where the electronic device (100) is located. There may be one or more external devices (200) placed in the space where the electronic device (100) is located. In particular, the external device (200) may be a home appliance such as a TV, an air conditioner, a refrigerator, a washing machine, a dryer, a dishwasher, a microwave oven, a vacuum cleaner, a speaker, an air purifier, etc. placed in the space where the electronic device (100) is located. In FIG. 1, an air conditioner (200-1) and a speaker (200-2) are illustrated as examples of the external device (200), but there is no particular limitation on the type of the external device (200).

[0065] The electronic device (100) can move through the space where the electronic device (100) is located and then project an image onto the projection area. Here, the 'projection area' refers to an area where an image is projected within the space where the electronic device (100) is located. Specifically, the projection area may include a wall surface of the space where the electronic device (100) is located, an object (e.g., a projection screen, a curtain, etc.) placed in the space where the electronic device (100) is located, a surface of an external device (200), etc. A more specific description of the projection area will be described in more detail with reference to FIGS. 2 and 3, etc.

[0066] Here, the image projected by the electronic device (100) may be referred to as a screen. The term "screen" collectively refers to the images projected by the electronic device (100), and may be implemented as one or more screens. The screen may include a user interface (UI), and thus the term "screen" may be replaced with the term "UI." Here, the UI may include text and graphic elements for providing information to the user, items corresponding to actions that the user can select, and may be replaced with terms such as "graphical user interface."

[0067] In particular, the 'screen' may be related to status information indicating the operating status of the external device (200). Specifically, the electronic device (100) may receive status information indicating the operating status of the external device (200) and project a screen for providing the received status information on an area on the surface of the external device (200) or an area around the external device (200).

[0068] For example, as illustrated in FIG. 1, the electronic device (100) can project a screen on the surface of an air conditioner (200-1) at position 1, and then project the screen on the wall surface above a speaker (200-2) after moving from position 1 to position 2.

[0069] However, since the electronic device (100) can move freely and project the screen to various locations, determining when, where, and in what projection area the electronic device (100) will project the screen is an important task. In particular, in order to provide images and information that meet the user's intentions in an effective and efficient manner, it may be desirable to consider the information to be projected, information about the external device (200), and information about the user. Various embodiments for solving this task will be described in detail below with reference to FIGS. 2 to 16.

[0070] FIG. 2 is a block diagram schematically illustrating the configuration of an electronic device (100) according to one or more embodiments. FIG. 3 is a diagram illustrating an operation of projecting a screen onto an area on the surface of an external device (200) according to one or more embodiments. The following description will be made with reference to FIGS. 2 and 3 together.

[0071] As illustrated in FIG. 2, the electronic device (100) may include a communication unit (110), a camera (120), a driving unit (130), a projection unit (140), a memory (150), and a processor (160). As described above, the electronic device (100) may not only be moved using the driving unit (130), but may also be moved by a user. Hereinafter, embodiments implemented by the electronic device (100) including the driving unit (130) will be described, but the embodiments described below may also be applied to a case where the electronic device (100) does not use the driving unit (130) included in the electronic device (100) or the electronic device (130) does not include the driving unit (130).

[0072] The communication unit (110) includes a circuit and can communicate with various devices including an external device (200). For example, the processor (160) can receive various data or information from an external device (200) connected through the communication unit (110) and can also transmit various data or information to the external device (200).

[0073] The communication unit (110) may include at least one of a Wi-Fi module, a Bluetooth module, a wireless communication module, an NFC module, and an Ultra-Wide Band (UWB) module. Specifically, the Wi-Fi module and the Bluetooth module may each perform communication in the Wi-Fi or Bluetooth manner. When using a Wi-Fi module or a Bluetooth module, various connection information, such as an SSID, may be first transmitted and received, and then communication may be established using this, after which various pieces of information may be transmitted and received.

[0074] In addition, the wireless communication module can perform communication according to various communication standards such as IEEE, Zigbee, 3G (3rd Generation), 3GPP (3rd Generation Partnership Project), LTE (Long Term Evolution), 5G (5th Generation), etc. And, the NFC module can perform communication in the NFC (Near Field Communication) method using the 13.56MHz band among various RF-ID frequency bands such as 135kHz, 13.56MHz, 433MHz, 860~960MHz, 2.45GHz, etc. In addition, the UWB module can accurately measure ToA (Time of Arrival), which is the time it takes for a pulse to reach a target, and AoA (Ange of Arrival), which is the pulse arrival angle at the transmitting device, through communication between UWB antennas, and accordingly, precise distance and location recognition is possible within an error range of several tens of centimeters indoors.

[0075] In particular, in one or more embodiments, the processor (160) may receive status information indicating the operating status of the external device (200) from the external device (200) through the communication unit (110). The processor (160) may receive information related to the external device (200), information about the user, etc. from the external device (200) through the communication unit (110). The processor (160) may control the communication unit (110) to transmit the status information received from the external device (200) to the user terminal.

[0076] The processor (130) may receive status information indicating the operating status of the external device (200), related information of the external device (200), information about the user, etc. from a device other than the external device (200), such as a server, through the communication unit (110). For example, the server, as an IoT (Internet of Things) server, may obtain and store status information of IoT devices including the external device (200), and the electronic device (100) may receive status information indicating the operating status of the external device (200) from the IoT server through the communication unit (110). In particular, when the power of the external device (200) is turned off or a communication connection between the electronic device (100) and the external device (200) is not established, the processor (130) can receive status information, etc. from the server. The processor (130) can receive status information indicating the operating status of the external device (200), related information of the external device (200), information about the user, etc. from both the external device (200) and the server through the communication unit (110), and can also update the status information, related information of the external device (200), information about the user, etc.

[0077] The camera (120) can acquire an image of at least one object. Specifically, the camera (120) includes an image sensor, which can convert light entering through a lens into an electrical image signal. The term "camera (120)" may be replaced with the term "image sensor."

[0078] The camera (120) may be an RGB camera (120) that can acquire images using red, green, and blue color channels, and may also be an RGB-D camera (120) that can acquire depth information along with color information that can be acquired through the RGB camera (120). In addition, the camera (120) may be implemented as a 360-degree camera (120), a panoramic camera (120), etc. The camera (120) may be arranged in an opposite direction to the projection unit (140), and the number of cameras may be one. However, there is no particular limitation on the type, location, or number of cameras (120) according to the present disclosure.

[0079] In particular, in one or more embodiments, the processor (160) may acquire images of an external device (200), images of a user, etc., through the camera (120).

[0080] The driving unit (130) can control the driving of the electronic device (100). The driving unit (130) can include at least one wheel and at least one motor. When physical force is generated by at least one motor, the generated force can be directly or through a connecting member to a hardware component included in the electronic device (100), and thus the hardware included in the electronic device (100) can be driven. The at least one motor included in the driving unit (130) can be implemented as various types of motors, such as a DC (Direct Current) motor, an AC (Alternative Current) motor, and a BLDC (Brushless DC) motor.

[0081] In one or more embodiments, the processor (160) may control the driving unit (130) to move the electronic device (100) or a hardware component included in the electronic device (100). For example, the processor (160) may control the direction and speed of at least one wheel to move the electronic device (100) to a destination. Here, the destination may include not only the final destination but also intermediate destinations, and may include not only a destination selected by a user but also a destination identified by the electronic device (100). In addition, the processor (160) may control the projection direction and projection angle of the projection unit (140) by controlling the driving unit (130) so that the projection unit (140) may project a screen onto a projection area.

[0082] The projection unit (140) refers to a configuration that emits light corresponding to an image in order to project an image. Here, the term 'emission' can be replaced with the term 'radiate', etc. The projection unit (140) can include a light source and at least one lens. The light source can emit light corresponding to an image and can be implemented as, for example, a light emitting diode (LED) or a laser. At least one lens is for projecting the light emitted from the light source in a desired direction and can include a lens that can be converted into a general lens or an ultra short throw (UST) lens. There can be one or more projection units (140). The term 'projection' can be replaced with the term 'provision' or 'display'.

[0083] In one or more embodiments, the processor (160) may control the projection unit (140) to project a screen onto a projection area. The projection unit (140) may include various detailed components such as a light source, a projection lens, and a reflector, and may perform various functions related to the projected image. The specific configuration and various functions of the projection unit (140) will be described in more detail with reference to FIG. 15.

[0084] At least one instruction regarding the electronic device (100) may be stored in the memory (150). In addition, an O / S (Operating System) for driving the electronic device (100) may be stored in the memory (150). In addition, various software programs or applications for operating the electronic device (100) according to various embodiments of the present disclosure may be stored in the memory (150). In addition, the memory (150) may include a semiconductor memory such as a flash memory or a magnetic storage medium such as a hard disk.

[0085] Specifically, various software modules for operating the electronic device (100) according to various embodiments of the present disclosure may be stored in the memory (150), and the processor (160) may control the operation of the electronic device (100) by executing the various software modules stored in the memory (150). That is, the memory (150) is accessed by the processor (160), and data reading / recording / modifying / deleting / updating, etc. may be performed by the processor (160).

[0086] Meanwhile, in the present disclosure, the term memory (150) may be used to mean a memory (150), a ROM, a RAM in a processor (160), or a memory card (e.g., a micro SD card, a memory stick) mounted in an electronic device (100).

[0087] In particular, in one or more embodiments, the memory (150) may store related information of the external device (200), status information indicating the operating status of the external device (200), and information about the user. The term "related information" is used as a general term for information related to the external device (200). A method for obtaining related information and what information the related information specifically includes will be described later in the description of the processor (160).

[0088] Additionally, the memory (150) may store various information, such as information about the screen, information about at least one identified area, information about the identified projection area, information about the critical size for the projection area, information about the criteria for defining the importance of state information, information about the neural network model, etc.

[0089] In addition, various information necessary within the scope of achieving the purpose of the present disclosure may be stored in the memory (150), and the information stored in the memory (150) may be updated as received from an external device (200) or input by a user.

[0090] The processor (160) controls the overall operation of the electronic device (100). Specifically, the processor (160) is connected to the configuration of the electronic device (100) including a communication unit (110), a camera (120), a driving unit (130), a projection unit (140), and a memory (150), and can control the overall operation of the electronic device (100) by executing at least one instruction stored in the memory (150) as described above.

[0091] The processor (160) may be implemented in various ways. For example, the processor (160) may be implemented as at least one of an application-specific integrated circuit (ASIC), an embedded processor, a microprocessor, hardware control logic, a hardware finite state machine (FSM), and a digital signal processor (DSP). Meanwhile, the term "processor (160)" in the present disclosure may be used to mean a central processing unit (CPU), a graphic processing unit (GPU), and a microprocessor unit (MPU).

[0092] In particular, in one or more embodiments, the processor (160) may identify at least one of an area on a surface of the external device (200) and an area around the external device (200), identify a projection area for providing a screen among the identified at least one area, and project the screen onto the identified projection area. Hereinafter, various embodiments implemented by the processor (160) will be described. In the present disclosure, the term 'identification' may be replaced with the term 'determination'.

[0093] The processor (160) can receive status information indicating the operating status of the external device (200) through the communication unit (110). Specifically, the processor (160) can receive status information indicating the operating status of the external device (200) based on at least one of the occurrence of a preset event and a user input. For example, the processor (160) can receive status information whenever a preset cycle arrives. As described above, the processor (130) can receive status information indicating the operating status of the external device (200) from the external device (200) through the communication unit, or can receive status information indicating the operating status of the external device (200) from a device other than the external device (200), such as a server, through the communication unit. The term 'status information' is used as a general term for information on the operating status of the external device (200). Specifically, the status information may include at least one of information about whether the external device (200) is performing an operation, detailed information about the operation being performed by the external device (200), and information about necessary actions for the operation of the external device (200).

[0094] The status information may represent information about the operational status of the external device (200) at the time the external device (200) transmits the status information to the electronic device (100). In addition, the status information may be continuously received at preset intervals and may be updated each time it is received.

[0095] The processor (160) can identify at least one area among an area on the surface of the external device (200) and an area around the external device (200) based on an image of the external device (200) obtained through the camera (120).

[0096] The term 'image for the external device (200)' refers to an image that includes information about the external device (200), and may include information about not only the external device (200) but also objects, walls, etc. around the external device (200). The image for the external device (200) may refer to an image acquired in real time through the camera (120) while the electronic device (100) is moving, as well as an image that has already been acquired through the camera (120) and stored in the memory (150).

[0097] The processor (160) can obtain an image of an external device (200) through the camera (120). The processor (160) can obtain an image of the external device (200) through the camera (120) while the electronic device (100) is moving. The processor (160) can also obtain an image of the external device (200) by receiving image data from another external device including the camera (120) through the communication unit (110).

[0098] When an image for an external device (200) is received, the processor (160) can analyze the image for the external device (200) to identify at least one area among an area on the surface of the external device (200) and an area around the external device (200).

[0099] The term "at least one region" is used to collectively refer to a region that can be a projection region according to the present disclosure. Hereinafter, the term "at least one region" refers to a region identified based on an image of an external device (200), and may be replaced with terms such as "candidate region," "projection candidate region," etc.

[0100] At least one region may include at least one of a region on the surface of the external device (200) and a region around the external device (200). Specifically, at least one region may include both a region on the surface of the external device (200) and a region around the external device (200). At least one region may include only a region on the surface of the external device (200) and may not include a region around the external device (200). At least one region may not include a region on the surface of the external device (200) and may include only a region around the external device (200).

[0101] The 'area on the surface of the external device (200)' may be any area on the surface of the external device (200), and may include multiple areas on the surface of the external device (200). The 'area around the external device (200)' may be an area on the surface of various objects such as furniture and curtains, as well as a wall around the external device (200), and may include multiple areas around the external device (200). How far away the area around the external device (200) is from the external device (200) is included in the area around the external device (200) may vary depending on how a user or developer sets a threshold distance to define the 'periphery' of the external device (200).

[0102] At least one area may be, but is not limited to, vertical planes perpendicular to the floor of the space where the electronic device (100) is located. For example, the floor of the space where the electronic device (100) is located, the ceiling plane horizontal to the floor, the surface of an external device (200) having a curved surface, the surface of an object having a curved surface, etc. may also be identified as at least one area. There is no particular limitation on the number of at least one area.

[0103] For example, when an image of an air conditioner, which is an external device (200), is acquired as in the example of FIG. 3, the processor (160) can identify an area (31) on the front surface of the air conditioner, a wall area (32) on the left side of the air conditioner, and a wall area (33) on the right side of the air conditioner as at least one area based on the acquired image.

[0104] Hereinafter, the case where the electronic device (100) identifies at least one area in real time based on an image of an external device (200) will be mainly described, but the processor (160) may also identify at least one area based on a spatial map previously stored in the memory (150).

[0105] Specifically, the processor (160) may acquire a plurality of images of a space in which the electronic device (100) is located, and acquire a map of the space in which the electronic device (100) is located based on the acquired plurality of images. The map of the space may be a three-dimensional map including information on horizontal planes and vertical planes of the space in which the electronic device (100) is located, and may be updated as images of the space in which the electronic device (100) is located are acquired. When the map of the space is updated and stored in the memory (150), the processor (160) may identify at least one area based on the updated space map without acquiring an image of the external device (200).

[0106] The processor (160) can identify a projection area for providing a screen related to status information among at least one of the identified areas based on the relevant information of the external device (200). Once the projection area is identified, the processor (160) can project a screen on the projection area.

[0107] The term 'related information of an external device' is a general term for information related to an external device (200) and may be simply referred to as 'related information'. In addition, the specific information related to an external device may be replaced with terms such as 'external device (200) information', 'type information', etc. Specifically, the related information may include at least one of information for identifying an external device (200), information on the type of the external device (200), information on the surface of the external device (200), information on the main operation of the external device (200), and information on the configuration included in the external device (200).

[0108] 'Information on the type of external device (200)' may include information on the type of operation of the external device (200) (e.g., air conditioner, washing machine, etc.), information on the model of the external device (200) (e.g., S Company ABC-111 model), information on detailed specifications of the external device (200), etc.

[0109] ‘Information on the surface of the external device (200)’ may include information on the area of ​​the surface of the external device (200), information on the shape, information on the shape, information on the color, information on the material, etc.

[0110] 'Information on the main operation of the external device (200)' may include information on what the main operation is among the operations performed by the external device (200). For example, if the external device (200) is an air conditioner, the information on the main operation of the external device (200) may include information on the fact that the main operation of the air conditioner is 'blowing'.

[0111] 'Information on the configuration included in the external device (200)' may include information on whether the external device (200) includes a specific hardware configuration (component), information on the specifications of the hardware configuration included in the external device (200), etc. For example, information on the configuration included in the external device (200) may include information on whether the external device (200) includes a display, information on the size and position of the display included in the external device (200), etc.

[0112] In addition to the information described above, the related information may include information related to identifying a projection area for providing a screen as information about an external device (200), and may be acquired by the processor (160) and then stored in the memory (150).

[0113] The processor (160) can obtain related information about the external device (200) based on an image of the external device (200). For example, the processor (160) can obtain an image of the external device (200) through the camera (120), analyze the image of the external device (200), or input the image of the external device (200) into a learned neural network model, thereby obtaining information about the type of the external device (200), information about the surface of the external device (200), etc.

[0114] The processor (160) may also obtain relevant information about the external device (200) by receiving relevant information about the external device (200) from at least one of the external device (200) and other external devices such as a server. For example, the electronic device (100) may receive information for identifying the external device (200), information about the type of the external device (200), etc. from the external device (200) while performing a communication connection with the external device (200).

[0115] The processor (160) may also receive, from the server, information related to an external device (200) transmitted to the server by the external device (200) through a communication connection with the server. In this case, the server may be a server for providing an IoT (Internet of Things) service, and the external device (200) may be a device registered as a device constituting an IoT network, i.e., an IoT device.

[0116] In the description of the present disclosure, for convenience of explanation, it will be explained on the assumption that the related information described above is stored in the memory (150), but the related information of the external device (200) can be acquired and updated in real time while the electronic device (100) performs an operation.

[0117] The processor (160) can identify a projection area based on relevant information of the external device (200). Specifically, as described below, the processor (160) can identify a projection area among at least one area based on at least some of various pieces of relevant information. In the present disclosure, identifying a projection area based on relevant information of the external device (200) does not exclude the possibility that the projection area can be identified using information other than the relevant information (e.g., status information, information about the user) together with the relevant information.

[0118] A "projection area" refers to an area within at least one area on which a screen related to status information is projected. The projection area is merely an area distinct from an area within at least one area on which a screen is not projected, and the screen does not need to be projected throughout the projection area. That is, the screen may be displayed in all or part of the projection area. For example, as illustrated in FIG. 3, if the projection area is identified as an area (31) on the surface of the electronic device, the screen may be displayed in a part within the area (31).

[0119] If at least one identified area is a single area, the processor (160) can identify the identified area as a projection area. However, below, embodiments of identifying a projection area among multiple areas will be described on the premise that multiple areas, including an area on the surface of the external device (200) and an area around the external device (200), are identified as at least one area.

[0120] The processor (160) can identify a projection area based on information about the type of the external device (200). Specifically, the memory (150) can store priority information about which area among the area on the surface of the external device (200) and the area around the external device (200) is to be preferentially identified as a projection area, depending on the type of the external device (200). Then, the processor (160) can identify the area as a projection area by using information that matches the information about the type of the external device (200) among the priority information stored in the memory (150). For example, if the external device (200) is a “standing air conditioner,” the processor (160) can identify the area on the surface of the external device (200) as a projection area, and if the external device (200) is a “wall-mounted air conditioner,” the processor (160) can identify the area around the external device (200) as a projection area.

[0121] The processor (160) can identify a projection area based on information about the surface of the external device (200). For example, the processor (160) can identify whether it is appropriate to project a screen onto the surface of the external device (200) based on at least one of the area, shape, appearance, and material of the surface of the external device (200). If it is determined that it is appropriate to project a screen onto the surface of the external device (200), the processor (160) can identify the surface of the external device (200) among at least one area as a projection area. If it is determined that it is not appropriate to project a screen onto the surface of the external device (200), the processor (160) can identify the area around the external device (200) among at least one area as a projection area.

[0122] The processor (160) can identify a projection area based on information about the main operation of the external device (200). For example, if the external device (200) is an air conditioner, the processor (160) can identify an area adjacent to the air vent of the air conditioner among at least one area as a projection area based on the fact that the main operation of the air conditioner is to blow air.

[0123] The processor (160) can identify a projection area based on information about the configuration included in the external device (200). For example, if the external device (200) includes a display, the processor (160) can identify an area adjacent to the display among at least one area as a projection area. If the external device (200) includes a button for user input, the processor (160) can identify an area adjacent to the button among at least one area as a projection area.

[0124] Once the projection area is identified, the processor (160) can control the projection unit (140) to project a screen onto the projection area. In addition, the processor (160) can control the driving unit (130) and the projection unit (140) to project a screen onto the projection area. The processor (160) can also control the projection unit (140) to project a screen onto the projection area after at least one of the position and projection angle of the electronic device (100) is changed by the user.

[0125] The processor (160) can control the driving unit (130) to move the electronic device (100) to a position where the screen can be projected onto the identified projection area. In addition, the processor (160) can control the projection angle of the projection unit (140) to project the screen onto the identified projection area. If the electronic device (100) is already in a position where the screen can be projected onto the projection area, the processor (160) can control the electronic device (100) to maintain that position and only change the projection angle of the projection unit (140).

[0126] After the electronic device (100) is in a state where it can project a screen onto the identified projection area by the driving unit (130), the processor (160) can control the projection unit (140) to project the screen onto the identified projection area. The processor (160) can also control the driving unit (130) and the projection unit (140) to project the screen onto the projection area while the electronic device (100) moves.

[0127] For example, if an area (31) on the front surface of the air conditioner, an area (32) on the wall surface on the left side of the air conditioner, and an area (33) on the wall surface on the right side of the air conditioner are identified as at least one area, as in the example of FIG. 3, the processor (160) may control the driving unit (130) and the projection unit (140) to identify the area (31) on the front surface of the air conditioner among the at least one area as a projection area and project a screen (34) onto the projection area according to one of the various embodiments described above. As illustrated in FIG. 3, the screen (34) may include information indicating that the filter of the air conditioner needs to be cleaned, information about the set temperature of the air conditioner, and information about the wind speed of the air conditioner.

[0128] According to the embodiments described above with reference to FIGS. 2 and 3, the electronic device (100) can provide status information about the external device (200) in a manner that matches the user's intention and intuition by using relevant information about the external device (200). In particular, the electronic device (100) can effectively provide status information about the external device (200) by projecting the status information about the external device (200) that does not have a display or has a limited display size onto an area on the surface of the external device (200) or an area around the external device (200).

[0129] Meanwhile, the above has described an embodiment related to identifying a projection area based on related information of an external device (200), but the processor (160) may also identify a projection area based on status information, or may identify a projection area based on information about a user. An embodiment related to identifying a projection area based on status information will be described with reference to FIGS. 4 and 5, and an embodiment of identifying a projection area based on information about a user will be described with reference to FIGS. 7 and 8.

[0130] FIG. 4 is a flowchart illustrating one or more embodiments related to identifying a projection area based on related information and status information. Furthermore, FIG. 5 is a diagram illustrating an operation of projecting a screen onto an area on a surface adjacent to an external device (200) among at least one area according to one or more embodiments. The following description will be made with reference to FIGS. 4 and 5 together.

[0131] Although the description of FIGS. 2 and 3 describes embodiments of identifying a projection area based on related information of an external device (200), the processor (160) may also identify a projection area based on related information and status information.

[0132] The processor (160) can identify at least one area including an area on the surface of the external device (200) and at least one area around the external device (200) based on an image of the external device (200) (S410). Specifically, the processor (160) can identify at least one area based on the position and size of the external device (200) included in the image of the external device (200) and the position and size of an object around the external device (200).

[0133] For example, when an image of an air conditioner, which is an external device (200), is acquired as in the example of FIG. 5, the processor (160) can identify at least one area of ​​an area on the front surface of the air conditioner (51), a wall area on the left side of the air conditioner (52), a wall area on the top of the air conditioner (53), and a wall area on the bottom of the air conditioner (54) based on the acquired image.

[0134] Since a picture frame is located on the wall surface below the air conditioner, the processor (160) can identify the wall surface area (54) below the air conditioner as at least one of the areas within a range that does not overlap with the picture frame. Since a clock is located on the wall surface to the right of the air conditioner in a position adjacent to the air conditioner, the processor (160) may not identify the wall surface area to the right of the air conditioner as at least one area.

[0135] The processor (160) can identify a critical size for the projection area based on at least one of the amount of state information and the importance of the state information (S420). Specifically, the processor (160) can identify a larger critical size for the projection area as the amount of state information increases, and the processor (160) can identify a larger critical size for the projection area as the importance of the state information increases.

[0136] Here, the "importance of status information" can be identified based on predefined criteria. Specifically, the status information may include, in order of importance, first status information indicating that immediate action is required, second status information indicating that action is required in the near future, third status information that is continuously updated information, and fourth status information indicating the general operating status of the external device (200). The processor (160) may also obtain a score, which is a quantitative value corresponding to the importance of the status information.

[0137] For example, the first status information may indicate that a refrigerator door is open, a washing machine's laundry is complete, or a malfunction has been detected in the device. The second status information may indicate the need for an air conditioner filter replacement, the need for an air purifier's filter replacement, or a suspected device malfunction. The third status information may indicate a fine dust alarm, a software update notification, or the results of an operating status monitoring. The fourth status information may indicate indoor temperature, indoor humidity, fine dust concentration, or the temperature inside the refrigerator.

[0138] Referring to FIG. 5, the status information may include information indicating that the air conditioner filter requires cleaning, that the indoor temperature is 15 degrees Celsius, and that the air conditioner's wind speed is strong. In this case, the processor (160) may identify a threshold size for the projection area based on the fact that there are three types of information to be provided through the screen, and that the information indicating that the air conditioner filter requires cleaning is the second status information with the second highest importance.

[0139] If the size of the surface of the external device (200) is greater than or equal to a threshold size (S430-Y), the processor (160) may identify an area on the surface of the external device (200) among at least one area as a projection area (S440). If the size of the surface of the external device (200) is less than or equal to a threshold size (S430-N) and the external device (200) includes a display (S450-Y), the processor (160) may identify an area adjacent to the display among at least one area as a projection area (S460). If the size of the surface is less than or equal to a threshold size (S430-N) and the external device (200) does not include a display (S450-N), the processor (160) may identify an area adjacent to a component of the external device (200) that performs a main operation of the external device (200) among at least one area as a projection area (S470).

[0140] Referring to the example of FIG. 3, the processor (160) can identify that the size of the surface of the external device (200) is greater than or equal to a threshold size, and can identify an area (31) on the surface of the external device (200) among at least one area (31, 32, 33) as a projection area.

[0141] On the other hand, referring to the example of FIG. 5, the processor (160) may identify that the size of the surface of the external device (200) is less than a critical size, and may not identify the area (51) on the surface of the external device (200) among at least one area (51, 52, 53, 54) as a projection area. The processor (160) may identify that the external device (200) does not include a display, and may identify the area adjacent to the 'air vent (55)' that performs the 'blowing' operation, which is the main operation of the external device (200), among at least one area, i.e., the wall area (54) at the bottom of the air conditioner, as a projection area.

[0142] According to the embodiments described above with reference to FIGS. 4 and 5, the electronic device (100) can provide status information about the external device (200) in a manner that matches the user's intention and intuition by using not only related information of the external device (200) but also status information.

[0143] FIG. 6 is a flowchart illustrating one or more embodiments related to identifying whether to project a screen based on status information and related information.

[0144] The above description of various embodiments assumes that the screen is identified as being projected to provide status information. However, the processor (160) can also identify whether to project the screen. Specifically, the processor (160) can identify whether to project the screen through the projection unit (140) based on the status information and related information (S610).

[0145] In one or more embodiments, the processor (160) may determine whether to project a screen through the projection unit (140) based on the status information, depending on whether the importance of the status information corresponds to an emergency.

[0146] Specifically, the processor (160) can identify whether an action is urgent regarding the operational status of the external device (200) based on the status information, and can identify whether to project the screen through the projection unit (140) based on the identification result. If the importance of the status information is identified as corresponding to an emergency, the processor (160) can identify not to project the screen. On the other hand, if the importance of the status information is identified as not corresponding to an emergency, the processor (160) can identify to project the screen.

[0147] As described above, the status information may include, in order of importance, first status information indicating that immediate action is required, second status information indicating that action is required in the near future, third status information which is continuously updated information, and fourth status information indicating a general operating status.

[0148] For example, if the status information is first status information, the processor (160) can identify that the importance of the status information corresponds to urgency and can identify that the screen is not projected. If the status information is not first status information, the processor (160) can identify that the importance of the status information does not correspond to urgency and can identify that the screen is projected.

[0149] In one or more embodiments, the processor (160) may identify whether to project a screen through the projection unit (140) based on information about a configuration included in the external device (200) among information related to the external device (200).

[0150] For example, if the external device (200) includes a display, the processor (160) may identify it as not projecting a screen. Conversely, if the external device (200) does not include a display, the processor (160) may identify it as projecting a screen.

[0151] For another example, if the external device (200) includes a display, if the size of the display included in the external device (200) is greater than or equal to a threshold size, the processor (160) may determine not to project the screen. On the other hand, if the size of the display included in the external device (200) is less than the threshold size, the processor (160) may determine to project the screen.

[0152] Additionally, the processor (160) may also identify whether the screen is projected based on whether the status information is a type of information that can be displayed through a display included in the external device (200).

[0153] In one or more embodiments, when status information is received, the processor (160) may identify whether the operational state of the external device (200) has changed based on the status information, and may project the screen only when the operational state of the external device (200) has changed.

[0154] If it is determined that the screen is not being projected (S620-Y), the processor (160) can control the communication unit (110) to transmit status information to the user terminal (S630). In other words, the processor (160) can provide status information to the user by transmitting the status information to the user terminal when the importance of the status information corresponds to an emergency or cannot be displayed through a display included in the external device (200). The user terminal can output the received status information as an image or voice.

[0155] When it is identified that a screen is being projected (S620-N), the processor (160) can acquire an image through the camera (120). When an image for the external device (200) is acquired, as described above with reference to FIG. 1, the processor (160) can identify at least one area including an area on the surface of the external device (200) and at least one area around the external device (200) based on the image for the external device (200) (S650), and can identify a projection area for providing a screen related to status information among the at least one area based on related information of the external device (200) (S660), and can control the driving unit (130) and the projection unit (140) to project the screen on the projection area (S670).

[0156] In other words, the processor (160) may provide the user with status information by projecting the screen without transmitting the status information to the user terminal, when the action regarding the operating status of the external device (200) is not urgent or can be displayed through a display included in the external device (200). Since steps S640 to S670 have been described above, a duplicate description of the same content will be omitted.

[0157] According to the embodiments described above with reference to FIG. 6, the electronic device (100) can provide status information to an external device (200) in a manner that conforms to the user's intention and intuition by projecting the screen or providing the status information through a user terminal based on the status information and related information.

[0158] FIG. 7 is a flowchart illustrating one or more embodiments related to identifying a projection area and projecting a screen based on relevant information and information about the user. FIGS. 8 and 9 are diagrams illustrating a process for identifying a projection area based on information about the user's head direction.

[0159] In the above, embodiments for identifying a projection area based on related information of an external device (200) and embodiments for identifying a projection area based on related information and status information have been described, but the processor (160) may also identify a projection area using information about a user. That is, the processor (160) may identify a projection area based on related information of an external device (200) and information about a user, or may identify a projection area based on related information, status information, and information about a user of an external device (200).

[0160] Below, embodiments of first obtaining information about a user and identifying whether a screen is projected based on the information about the user are described, and then embodiments of identifying a projection area using the information about the user are described.

[0161] The processor (160) can obtain information about the user (S710). The term "user information" is used to collectively refer to information related to the user. Specifically, the user information may include at least one of information about the user's location, information about the user's head direction, and information about the distance between the user and the external device (200). In addition to the information described above, the user information may identify a projection area for providing a screen as information about the user and include information related to projecting the screen.

[0162] The processor (160) may obtain information about the user based on at least one of an image of the user obtained through a camera (120), information about the location of the user obtained through a sensor unit (170) (e.g., a GPS sensor, an infrared sensor, an ultrasonic sensor, etc.), information about the history of the user using the electronic device (100), information about the history of the user using the external device (200), information about an automated routine registered by the user, and information about the establishment / termination of a communication (Wi-Fi communication, Bluetooth communication, etc.) connection between the user's user terminal (e.g., the user's smartphone, smartwatch, etc.) and the electronic device (100) / external device (200).

[0163] The processor (160) can acquire information about the direction of the user's head by acquiring an image of the user through the camera (120) and inputting the image of the user into a neural network model trained to identify the direction of the user's head. The processor (160) can also acquire information about the direction of the user's head by identifying the direction of the user's eyes or pupils, etc. based on the image of the user.

[0164] When information about a user is collected for a certain period of time or longer, and the information about the user shows a certain pattern according to a certain condition or time based on the collected information, the processor (160) can obtain information about the certain pattern as information about the user's lifestyle pattern.

[0165] The processor (160) can identify whether the screen is projected based on information about the user as well as related information and status information.

[0166] In one or more embodiments, the processor (160) may identify whether the user is present in the space where the electronic device (100) is located based on information about the user. If the user is identified as being present in the space where the electronic device (100) is located, the processor (160) may identify the user as projecting a screen. If the user is identified as not being present in the space where the electronic device (100) is located, the processor (160) may identify the user as not projecting a screen. In addition, if the user is identified as not projecting a screen, the processor (160) may control the communication unit (110) to transmit status information to the user terminal.

[0167] In one or more embodiments, the processor (160) may identify whether the user is located in proximity to an external device (200) based on information about the user. Here, whether the user is located in proximity to the external device (200) may mean whether the user is located in a location where the user can visually recognize the external device (200).

[0168] If the user is identified as being located adjacent to the external device (200), the processor (160) may determine that the screen is being projected. If the user is identified as not being located adjacent to the external device (200), the processor (160) may determine that the screen is not being projected. In addition, if the user is identified as not being projected, the processor (160) may control the communication unit (110) to transmit status information to the user terminal.

[0169] In one or more embodiments, the processor (160) may determine whether to project a screen based on information about the user's lifestyle and status information. In other words, the processor (160) may determine whether to project a screen based on whether projecting a screen related to the status information aligns with the user's lifestyle.

[0170] For example, if it is identified that the user is planning to go out based on information about the user's lifestyle pattern, the processor (160) can identify the user by projecting a screen to guide the user to change the operating state of the external device (200) (e.g., power off the external device (200)) to correspond to the user's going out situation.

[0171] As another example, if it is identified that the user is about to return after going out based on information about the user's lifestyle pattern, the processor (160) can identify the user by projecting a screen to guide that the operating state of the external device (200) has been changed to correspond to the user's return situation.

[0172] As another example, if the operational status of an external device (200) changes while the user is out, the processor (160) can identify whether to transmit information about the changed operational status to the user terminal or to project a screen including information about the operational status based on the importance of the status information.

[0173] Hereinafter, in the description of FIG. 7, it is assumed that the screen has been identified as being projected according to various embodiments as described above (S720), and a projection area is identified and the screen is projected on the identified projection area.

[0174] The processor (160) can identify a projection area based on information about the user, related information about the external device (200), and status information. In particular, the processor (160) can identify a projection area among at least one area based on information about the direction of the user's head.

[0175] The processor (160) can obtain an image of the user through the camera (120) and input the image of the user into a neural network model trained to identify the direction of the user's head, thereby obtaining information about the direction of the user's head.

[0176] The processor (160) can identify whether the user's head orientation is maintained for a preset period of time (S730). Specifically, the processor (160) acquires an image of the user at preset intervals and obtains information about the user's head orientation, thereby identifying whether the user's head orientation is maintained for a preset period of time.

[0177] If the user's head direction is maintained for a preset period of time (S730-Y), the processor (160) can identify an area corresponding to the user's head direction as a projection area (S740). Specifically, the processor (160) can identify at least one area corresponding to the user's head direction, and can also identify an area corresponding to the user's head direction among the at least one area as a projection area.

[0178] In one or more embodiments, when the user's head direction is maintained for a preset period of time, the processor (160) can identify at least one area corresponding to the user's head direction, and can identify a projection area among the at least one area based on information about the user, related information of the external device (200), and status information.

[0179] As illustrated in FIG. 8, if it is determined that the user's head direction is maintained in the first direction for a preset period of time, the processor (160) may identify an area (81) on the surface of the external device (200) corresponding to the first direction and an area (82) around the external device (200) corresponding to the first direction as at least one area. Then, the processor (160) may identify one area among the at least one area as a projection area based on information about the user, related information of the external device (200), and status information.

[0180] In one or more embodiments, if the user's head direction is maintained for a preset period of time, the processor (160) may identify at least one area based on information about the user, related information and status information of the external device (200), and identify an area corresponding to the user's head direction among the at least one area as a projection area.

[0181] As illustrated in FIG. 9, the processor (160) can identify at least one area as an area (91) on the surface of the external device (200) corresponding to the first direction, an area (92) around the external device (200) corresponding to the first direction, an area (93) on the surface of the external device (200) corresponding to the second direction, and an area (94) around the external device (200) corresponding to the second direction. Then, when it is determined that the direction of the user's head is maintained in the second direction for a preset time, the processor (160) can identify one area among the area (93) on the surface of the external device (200) corresponding to the second direction and the area (94) around the external device (200) corresponding to the second direction as a projection area.

[0182] If the user's head direction is not maintained for a preset period of time (S730-N), the processor (160) can identify a projection area among at least one area based on information about the user, related information, or status information, rather than information about the user's head direction.

[0183] Once the projection area is identified, the processor (160) can identify whether the external device (200) is in a state of performing a preset operation based on the status information (S750). The preset operation can be changed according to the user or developer's settings, and may be the entire operation of the external device (200).

[0184] When the external device (200) is in a state of performing a preset operation (S750-Y), the processor (160) can control the driving unit (130) and the projection unit (140) to project a screen containing information on the progress of the preset operation (S760). This is because the user's intention of looking at the external device (200) performing the preset operation for a preset period of time or longer is often to know the progress of the preset operation of the external device (200).

[0185] For example, when the washing machine, which is an external device (200), is performing a preset operation of “washing,” the processor (160) can control the driving unit (130) and the projection unit (140) to project a screen containing information such as the washing mode, the expected end time of washing, etc.

[0186] If the external device (200) is not performing a preset operation (S750-N), the processor (160) can control the driving unit (130) and the projection unit (140) to project a screen including information for controlling the operation of the external device (200) (S770). This is because the intention of a user who looks at an external device (200) that is not performing a preset operation for a preset period of time or longer is often to control the external device (200) so that it performs a preset operation.

[0187] For example, if the washing machine, which is an external device (200), is not performing a preset operation, such as a “washing” operation, the processor (160) can control the driving unit (130) and the projection unit (140) to project a screen containing information such as the setting of the washing mode.

[0188] According to the embodiments described above with reference to FIGS. 7 to 9, the electronic device (100) can provide status information about the external device (200) to an area on the surface of the external device (200) or an area around the external device (200) in a manner that matches the user's intention and intuition by using information about the user as well as information related to the external device (200).

[0189] FIG. 10 and FIG. 11 are drawings for explaining one or more embodiments related to the size of the screen and the shape of the screen.

[0190] In the above, various embodiments have been described focusing on an embodiment of identifying whether a screen is projected and an embodiment of identifying a projection area of ​​a screen. However, in one or more embodiments, the processor (160) may identify at least one of the size of the screen to be projected on the projection area and the shape of the screen.

[0191] The processor (160) can obtain information about the distance between the user and the external device (200). Specifically, the processor (160) can obtain information about the distance between the user and the external device (200) using the communication unit (110). The processor (160) can obtain information about the distance between the user and the external device (200) using the sensor unit (170). In addition, the processor (160) can obtain information about the distance between the user and the external device (200) based on an image obtained through the camera (120).

[0192] For example, the processor (160) can obtain information about the distance between the user and the external device (200) based on the difference in time between the arrival of the signal transmitted through the communication unit (110) and the arrival angle between the user and the external device (200).

[0193] For another example, the processor (160) can obtain information about the distance between the electronic device (100) and the user and information about the distance between the electronic device (100) and the external device (200) using an infrared sensor or an ultrasonic sensor, etc. In addition, the processor (160) can obtain information about the distance between the user and the external device (200) based on the information about the distance between the electronic device (100) and the user and the information about the distance between the electronic device (100) and the external device (200).

[0194] When information about the distance between the user and the external device (200) is obtained, the processor (160) can identify at least one of the size of the screen and the shape of the screen based on at least one of the status information and the information about the distance between the user and the external device (200).

[0195] In one or more embodiments, the processor (160) may identify the size of the screen and the sizes of elements included in the screen based on the status information. Specifically, the processor (160) may identify the size of the screen and the sizes of elements included in the screen more significantly as the amount of status information increases, and the processor (160) may identify the size of the screen and the sizes of elements included in the screen more significantly as the importance of the status information increases. In addition, the processor (160) may configure the screen layout by adjusting the arrangement of elements included in the screen based on the status information.

[0196] As described above, the status information may include, in order of importance, first status information indicating that immediate action is required, second status information indicating that action is required in the near future, third status information which is continuously updated information, and fourth status information indicating the general operating status of the external device (200).

[0197] Referring to the examples of FIGS. 3 and 5, the information that the air conditioner filter needs to be cleaned is the second state information and is the second most important state information, so the size of the screen can be identified according to a preset standard, and the size of the element including the information “filter cleaning needed” can be identified as larger than the element including the information “15 degrees Celsius” and the element including the information “High” according to the preset standard.

[0198] In one or more embodiments, the processor (160) may identify the size of the screen and the size of the elements included in the screen as being larger as the distance between the user and the external device (200) increases. Specifically, the processor (160) may identify the size of the screen and the size of the elements included in the screen as being larger in proportion to the distance between the user and the external device (200) within a threshold size for the projection area. In addition, if the distance between the user and the external device (200) is greater than the threshold distance, the processor (160) may change the arrangement and shape of the elements included in the screen to increase the user's visibility.

[0199] Referring to FIG. 10, for example, the distance (101) between the user and the external device (200) may be 1 m. In this case, the processor (160) may identify a screen in which elements included in the screen are arranged horizontally and the vertical length (102) is 30 cm.

[0200] Referring to FIG. 11, for example, the distance between the user and the external device (200) (111) may be 3 m, which may be a distance exceeding the critical distance of 2.5 m. In this case, the processor (160) may identify a screen having a vertical length (112) of 1 m, in which the elements included in the screen are arranged vertically to correspond to the shape of the projection area so that the screen can be projected in a critical size for the projection area.

[0201] According to the embodiments described above with reference to FIGS. 10 and 11, the electronic device (100) can provide status information about the external device (200) in a manner that matches the user's intention and intuition by controlling the size and shape of the screen based on related information, status information, and information about the user of the external device (200).

[0202] FIGS. 12 to 14 are drawings illustrating one or more embodiments related to sequentially projecting multiple screens and identifying projection positions of the screens within a projection area.

[0203] In FIGS. 12 to 14, the description is given on the assumption that an area on the surface of an air conditioner, which is an external device (200), is identified as a projection area.

[0204] In one or more embodiments, the processor (160) may project not only a single screen related to the status information, but may also sequentially project multiple screens related to the status information. Specifically, the processor (160) may determine whether to project multiple screens based on at least one of the amount of status information and the importance of the status information. If it is determined that multiple screens are to be projected, the processor (160) may control the driving unit (130) and the projection unit (140) to sequentially project the multiple screens.

[0205] Referring to FIGS. 12 to 14, when the status information indicates the need to replace the filter of the air conditioner, the processor (160) can identify by projecting a plurality of screens corresponding to each of a plurality of actions related to replacing the filter of the air conditioner.

[0206] In one or more embodiments, the processor (160) may control the driving unit (130) and the projection unit (140) to sequentially project the plurality of screens based on whether the user has performed an action corresponding to each of the plurality of screens. Specifically, the processor (160) may identify whether the user has performed an action corresponding to the first screen while projecting the first screen among the plurality of screens. Then, if it is identified that the user has performed an action corresponding to the first screen, the processor (160) may control the driving unit (130) and the projection unit (140) to project the second screen subsequent to the first screen.

[0207] Here, 'whether the user performed an action corresponding to the first screen' can be identified based on at least one of status information, an image of the user acquired through the camera (120), and information about the user's movement or pose acquired through a sensor.

[0208] For example, the processor (160) can control the driving unit (130) and the projection unit (140) to project a first screen (121) including the text “Please turn off the power before replacing the filter” as shown in FIG. 12.

[0209] If it is determined that the air conditioner is turned off as a result of identifying whether the user has performed an action corresponding to the first screen, the processor (160) can control the driving unit (130) and the projection unit (140) to project a second screen (131) including the text “Please replace the filter” and “an arrow indicating the filter position” as shown in FIG. 13.

[0210] If it is determined that the air conditioner filter has been replaced as a result of identifying whether the user has performed an action corresponding to the second screen, the processor (160) can control the driving unit (130) and the projection unit (140) to project a third screen (141) including the text “Please turn off the power before replacing the filter” as shown in FIG. 14.

[0211] In one or more embodiments, the processor (160) can identify a projection position of a screen to be projected onto a projection area and project the screen onto the identified projection position. Here, the term 'projection position' can be distinguished from the term 'projection area' which indicates an area among at least one area identified as being intended to provide a screen, in that it refers to a position within the projection area where the screen is projected when the projection area is identified.

[0212] The processor (160) can identify a projection position of a screen to be projected on a projection area based on the status information and information about a configuration included in the external device (200). Specifically, the processor (160) can identify a configuration of the external device (200) related to the status information based on the status information and information about a configuration included in the external device (200). Here, the configuration of the external device (200) related to the status information may be a configuration related to an operating state of the external device (200) indicated by the status information or a configuration related to a necessary action indicated by the status information. Once the configuration of the external device (200) related to the status information is identified, the processor (160) can identify a position corresponding to the identified configuration in the projection area as a projection position. Then, the processor (160) can control the driving unit (130) and the projection unit (140) to project the screen on the identified projection position. Here, the screen may include a graphic element indicating the projection position.

[0213] Referring to FIG. 13, when the status information indicates the need for replacing the air conditioner filter, the processor (160) can identify the "filter of the air conditioner", which is a component related to the necessary action indicated by the status information, and can identify a location corresponding to the "filter of the air conditioner" in the projection area as a projection location. Then, the processor (160) can control the driving unit (130) and the projection unit (140) to project a screen including an "arrow indicating the filter location", which is a graphic element indicating the projection location, at the identified projection location.

[0214] According to the embodiments described above with reference to FIGS. 12 to 14, the electronic device (100) can sequentially project multiple screens, and can sequentially project multiple screens while taking into account interaction with the user. Accordingly, the electronic device (100) can effectively provide multiple screens that meet the user's needs, particularly during the initial installation or repair process of the external device (200).

[0215] Additionally, the electronic device (100) can provide status information about the external device (200) in a manner that matches the user's intention and intuition by identifying a projection location within the projection area using status information and related information of the external device (200).

[0216] FIG. 15 is a block diagram detailing the configuration of an electronic device (100) according to one or more embodiments.

[0217] As illustrated in FIG. 15, the electronic device (100) may further include a communication unit (110), a camera (120), a driving unit (130), a projection unit (140), a memory (150), and a processor (160), as well as a sensor unit (170), an input unit (180), and an output unit (190). However, the configurations illustrated in FIGS. 2 and 15 are merely exemplary, and it is to be understood that new configurations may be added or some configurations may be omitted in addition to the configurations illustrated in FIGS. 2 and 15 when implementing the present disclosure.

[0218] First, the projection unit (140) described with reference to FIG. 2 according to the present disclosure will be described in more detail below, and then the communication unit (110), input unit (180), and output unit (190) will be described.

[0219] The projection unit (140) can be implemented using various projection methods (e.g., CRT (cathode-ray tube) method, LCD (Liquid Crystal Display) method, DLP (Digital Light Processing) method, laser method, etc.). For example, the CRT method is basically the same principle as a CRT monitor. The CRT method magnifies the image with a lens in front of the cathode-ray tube (CRT) and displays the image on the screen. Depending on the number of cathode-ray tubes, it is divided into a single-tube type and a three-tube type, and in the case of a three-tube type, the red, green, and blue cathode-ray tubes can be implemented separately.

[0220] Another example is the LCD method, which displays images by passing light from a light source through liquid crystals. LCD methods are divided into single-panel and three-panel types. In the case of the three-panel type, light from a light source is separated into red, green, and blue by a dichroic mirror (a mirror that reflects only a specific color of light and transmits all others). After passing through the liquid crystals, the light can be refocused into a single point.

[0221] As another example, the DLP method is a method of displaying images using a DMD (Digital Micromirror Device) chip. The DLP projection unit (140) may include a light source, a color wheel, a DMD chip, a projection lens, etc. Light output from the light source may be colored as it passes through the rotating color wheel. The light passing through the color wheel is input to the DMD chip. The DMD chip includes numerous micromirrors and reflects the light input to the DMD chip. The projection lens may play a role in magnifying the light reflected from the DMD chip to an image size.

[0222] Another example is a laser system that uses a Diode Pumped Solid State (DPSS) laser and a galvanometer. A multi-color laser uses three DPSS lasers, one for each RGB color, with their optical axes overlapped by a special mirror. The galvanometer includes a mirror and a high-power motor that moves the mirror at high speeds. For example, the galvanometer can rotate the mirror at up to 40 kHz / sec. The galvanometer is mounted according to the scan direction, but since projectors typically scan in a planar manner, the galvanometer can also be positioned separately along the x and y axes.

[0223] Meanwhile, the projection unit (140) may include various types of light sources. For example, the projection unit (140) may include at least one light source among a lamp, an LED, and a laser.

[0224] The projection unit (140) can output images with a 4:3 screen ratio, a 5:4 screen ratio, or a 16:9 wide screen ratio depending on the purpose of the electronic device (100) or the user's settings, and can output light corresponding to the image with various resolutions such as WVGA (854*480), SVGA (800*600), XGA (1024*768), WXGA (1280*720), WXGA (1280*800), SXGA (1280*1024), UXGA (1600*1200), and Full HD (1920*1080) depending on the screen ratio.

[0225] Meanwhile, the projection unit (140) can perform various functions to adjust the output image (i.e., the image projected onto the screen). For example, the projection unit (140) can adjust the size and shape of the output image by performing functions such as zoom, keystone, quick corner (4-corner) keystone, and lens shift.

[0226] Specifically, the projection unit (140) can enlarge or reduce the image depending on the distance from the screen (projection distance). That is, the zoom function can be performed depending on the distance from the screen. At this time, the zoom function may include a hardware method that adjusts the screen size by moving the lens and a software method that adjusts the screen size by cropping the image, etc. Meanwhile, when the zoom function is performed, the focus of the image needs to be adjusted. For example, the method of adjusting the focus includes a manual focus method, an electric focus method, etc. The manual focus method refers to a method of focusing manually, and the electric focus method refers to a method of automatically focusing using a motor built into the projector when the zoom function is performed. When performing the zoom function, the projection unit (140) may provide a digital zoom function through software, and may provide an optical zoom function that performs the zoom function by moving the lens through the driving unit (130).

[0227] In addition, the projection unit (140) can perform a keystone correction function. If the height is not right for front projection, the screen may be distorted upwards or downwards. The keystone correction function refers to a function that corrects a distorted screen. For example, if distortion occurs in the left and right directions of the screen, it can be corrected using horizontal keystone, and if distortion occurs in the up and down directions, it can be corrected using vertical keystone. The quick corner (4 corner) keystone correction function is a function that corrects the screen when the center area of ​​the screen is normal but the corner areas are not balanced. The lens shift function is a function that moves the screen as it is when the screen is off the screen.

[0228] Meanwhile, the projection unit (140) can automatically analyze the surrounding environment and projection environment without user input to provide zoom / keystone / focus functions. Specifically, the projection unit (140) can automatically provide zoom / keystone / focus functions based on the distance between the electronic device (100) and the screen detected through a sensor (depth camera (120), distance sensor, infrared sensor, light sensor, etc.), information about the space where the electronic device (100) is currently located, information about the amount of ambient light, etc.

[0229] In addition, the projection unit (140) can provide a lighting function using a light source. In particular, the projection unit (140) can provide a lighting function by outputting a light source using an LED. According to various embodiments, the projection unit (140) can include one LED, and according to other embodiments, the electronic device (100) can include a plurality of LEDs. Meanwhile, the projection unit (140) can output a light source using a surface-emitting LED according to an implementation example. Here, the surface-emitting LED can mean an LED having a structure in which an optical sheet is arranged on the upper side of the LED so that the light source is evenly distributed and output. Specifically, when a light source is output through the LED, the light source can be evenly distributed through the optical sheet, and the light source distributed through the optical sheet can be incident on the display panel.

[0230] Meanwhile, the projection unit (140) may provide the user with a dimming function for adjusting the intensity of the light source. Specifically, when a user input for adjusting the intensity of the light source is received from the user through an operation interface (e.g., a touch display button or dial), the projection unit (140) may control the LED to output the intensity of the light source corresponding to the received user input.

[0231] Additionally, the projection unit (140) can provide a dimming function based on content analyzed by the processor (160) without user input. Specifically, the projection unit (140) can control the LED to output the intensity of the light source based on information about the currently provided content (e.g., content type, content brightness, etc.).

[0232] Meanwhile, the projection unit (140) can control the color temperature under the control of the processor (160). Here, the processor (160) can control the color temperature based on the content. Specifically, when the content is identified to be output, the processor (160) can obtain frame-by-frame color information of the content whose output has been determined. Then, the processor (160) can control the color temperature based on the obtained frame-by-frame color information. Here, the processor (160) can obtain at least one primary color of the frame based on the frame-by-frame color information. Then, the processor (160) can adjust the color temperature based on the obtained at least one primary color. For example, the color temperature that the processor (160) can adjust can be classified into a warm type or a cold type. Here, it is assumed that the frame to be output (hereinafter, referred to as the output frame) includes a scene where a fire has occurred. The processor (160) can identify (or obtain) that the primary color is red based on the color information included in the current output frame. In addition, the processor (160) can identify a color temperature corresponding to the identified primary color (red). Here, the color temperature corresponding to red may be a warm type. Meanwhile, the processor (160) may utilize an artificial intelligence model to obtain color information or a primary color of the frame. According to various embodiments, the artificial intelligence model may be stored in the electronic device (100) (e.g., memory (150)). According to another embodiment, the artificial intelligence model may be stored in an external server capable of communicating with the electronic device (100).

[0233] The sensor unit (170) can detect various information inside and outside the electronic device (100). Specifically, the sensor may include at least one of an infrared sensor, an ultrasonic sensor, a GPS (Global Positioning System) sensor, a gyro sensor (gyroscope), an acceleration sensor (accelerometer), a LiDAR sensor, an inertial measurement unit (IMU), and a motion sensor. In addition, the sensor may include various types of sensors, such as an illuminance sensor, a temperature sensor, a humidity sensor, and a biosensor.

[0234] In particular, in one or more embodiments, the processor (160) may obtain information about the location of the external device (200), information about the distance between the external device (200) and the electronic device (100), information about the distance between the external device (200) and the user, etc. through the sensor unit (170). The processor (160) may obtain information about whether a user is present in the space where the electronic device (100) is located through the sensor unit (170). In addition, the processor (160) may obtain information about the movement or pose of the user through the sensor unit (170).

[0235] The processor (160) can implement various embodiments as described above by using various information acquired through the sensor unit (170) together with related information of the external device (200), status information, information about the user, an image about the external device (200), and an image about the user.

[0236] The input unit (180) includes a circuit, and the processor (160) can receive a user command to control the operation of the electronic device (100) through the input unit (180). Specifically, the input unit (180) can be configured with components such as a microphone, a camera (120), and a remote control signal receiving unit. In addition, the input unit (180) can also be implemented in a form included in a display as a touch screen. In particular, the microphone can receive a voice signal and convert the received voice signal into an electrical signal.

[0237] In particular, in one or more embodiments, the processor (160) may receive, through the input unit (180), a user input for projecting a screen, a user input for receiving status information about an external device (200), a user input for adjusting the size and shape of a projected screen, a user input for controlling movement of the electronic device (100), a user input for changing a map of a space in which the electronic device (100) is located, a user input for changing a threshold size for a projection area, a user input for changing a preset time for a user's head direction, and the like.

[0238] The output unit (190) includes a circuit, and the processor (160) can output various functions that the electronic device (100) can perform through the output unit (190). In addition, the output unit (190) can include at least one of a display, a speaker, and an indicator.

[0239] The display can output image data under the control of the processor (160). Specifically, the display can output an image previously stored in the memory (150) under the control of the processor (160). In particular, the display according to one or more embodiments of the present disclosure can also display a user interface stored in the memory (150). The display can be implemented as an LCD (Liquid Crystal Display Panel), an OLED (Organic Light Emitting Diodes), etc., and in some cases, the display can also be implemented as a flexible display, a transparent display, etc. However, the display according to the present disclosure is not limited to a specific type.

[0240] The speaker can output audio data under the control of the processor (160). The indicator can be turned on under the control of the processor (160). The indicator can be turned on in various colors under the control of the processor (160). For example, the indicator can be implemented as a light emitting diode (LED), a liquid crystal display panel (LCD), a vacuum fluorescent display (VFD), etc., but is not limited thereto.

[0241] In particular, in one or more embodiments, the processor (160) may output, through the output unit (190), a message to guide that a screen is being projected on a projection area, a message to provide information included in the screen or additional information about information included in the screen, a message to guide that a map for a space where the electronic device (100) is located has been updated, etc.

[0242] FIG. 16 is a flowchart illustrating a method of controlling an electronic device (100) according to one or more embodiments.

[0243] The electronic device (100) can receive status information indicating the operating status of the external device (200) (S1610). Specifically, the electronic device (100) can receive status information indicating the operating status of the external device (200) from the external device (200) based on at least one of the occurrence of a preset event and a user input. For example, the electronic device (100) can receive status information from the external device (200) every time a preset cycle arrives.

[0244] The electronic device (100) can identify at least one area including an area on the surface of the external device (200) and at least one area around the external device (200) based on an image of the external device (200) obtained through the camera (120) (S1620).

[0245] The electronic device (100) can obtain an image of an external device (200) through a camera (120). The electronic device (100) can obtain an image of the external device (200) through the camera (120) while the electronic device (100) is moving. The electronic device (100) can also obtain an image of the external device (200) by receiving image data from another external device including a camera (120) through a communication unit (110).

[0246] When an image of an external device (200) is received, the electronic device (100) can analyze the image of the external device (200) to identify at least one area including an area on the surface of the external device (200) and at least one area around the external device (200).

[0247] The electronic device (100) can identify a projection area for providing a screen related to status information among at least one area based on relevant information of the external device (200) (S1630). Once the projection area is identified, the electronic device (100) can project a screen onto the projection area (S1640).

[0248] The electronic device (100) can obtain related information about the external device (200) based on an image of the external device (200). For example, the electronic device (100) can obtain an image of the external device (200) through a camera (120), analyze the image of the external device (200), or input the image of the external device (200) into a learned neural network model, thereby obtaining information about the type of the external device (200), information about the surface of the external device (200), etc.

[0249] The electronic device (100) may obtain relevant information about the external device (200) by receiving relevant information about the external device (200) from at least one of the external device (200) and other external devices such as a server. For example, the electronic device (100) may receive information for identifying the external device (200), information about the type of the external device (200), etc. from the external device (200) while performing a communication connection with the external device (200).

[0250] The electronic device (100) may also receive, from the server, information related to the external device (200) transmitted to the server by the external device (200) through a communication connection with the server. In this case, the server may be a server for providing an IoT (Internet of Things) service, and the external device (200) may be a device registered as a device constituting an IoT network, i.e., an IoT device.

[0251] The electronic device (100) can identify a projection area based on at least one of information about the type of the external device (200), information about the surface of the external device (200), information about the main operation of the external device (200), and information about the configuration included in the external device (200).

[0252] Once the projection area is identified, the electronic device (100) can project a screen onto the projection area by controlling the projection unit (140). Furthermore, once the projection area is identified, the electronic device (100) can project a screen onto the projection area by controlling the driving unit (130) and the projection unit (140).

[0253] Meanwhile, the control method of the electronic device (100) according to the above-described embodiment may be implemented as a program and provided to the electronic device (100). In particular, the program including the control method of the electronic device (100) may be stored and provided in a non-transitory computer readable medium.

[0254] Specifically, in a non-transitory computer-readable recording medium including a program for executing a control method of an electronic device (100), the control method of the electronic device (100) may include a step of receiving status information indicating an operating status of an external device (200), a step of identifying at least one area including an area on a surface of the external device (200) and at least one area around the external device (200), based on an image of the external device (200), a step of identifying a projection area for providing a screen related to the status information among the at least one area, based on related information of the external device (200), and a step of projecting the screen onto the projection area.

[0255] In the above, a method for controlling an electronic device (100) and a computer-readable recording medium including a program for executing the method for controlling an electronic device (100) have been briefly described, but this is only to omit redundant descriptions, and it goes without saying that various embodiments of the electronic device (100) can also be applied to a method for controlling an electronic device (100) and a computer-readable recording medium including a program for executing the method for controlling an electronic device (100).

[0256] The artificial intelligence-related function according to the present disclosure is operated through the processor (160) and memory (150) of the electronic device (100).

[0257] The processor (160) may be composed of one or more processors (160). At this time, the one or more processors (160) may include at least one of a CPU (Central Processing Unit), a GPU (Graphic Processing Unit), and an NPU (Neural Processing Unit), but is not limited to the examples of the processors (160) described above.

[0258] The CPU is a general-purpose processor (160) capable of performing not only general calculations but also artificial intelligence calculations. Its multi-layer cache structure allows for the efficient execution of complex programs. The CPU is advantageous in a serial processing method, which enables organic linking of previous and subsequent calculation results through sequential calculations. The general-purpose processor (160) is not limited to the aforementioned examples, except in cases where it is specifically designated as a CPU.

[0259] A GPU is a processor (160) for large-scale operations such as floating point operations used in graphic processing, and can perform large-scale operations in parallel by integrating a large number of cores. In particular, a GPU may be advantageous compared to a CPU in parallel processing methods such as convolution operations. In addition, a GPU may be used as a co-processor (160) to supplement the functions of a CPU. The processor (160) for large-scale operations is not limited to the examples described above, except in cases where it is specifically referred to as a GPU.

[0260] An NPU is a processor (160) specialized in artificial intelligence operations using an artificial neural network, and each layer constituting the artificial neural network can be implemented with hardware (e.g., silicon). At this time, since the NPU is designed specifically according to the required specifications of the company, it has a lower degree of freedom compared to a CPU or GPU, but it can efficiently process the artificial intelligence operations requested by the company. Meanwhile, as a processor (160) specialized in artificial intelligence operations, the NPU can be implemented in various forms such as a TPU (Tensor Processing Unit), an IPU (Intelligence Processing Unit), a VPU (Vision processing unit), etc. The artificial intelligence processor (160) is not limited to the above-described examples, except in cases where it is specified as the above-described NPU.

[0261] Additionally, one or more processors (160) may be implemented as a SoC (System on Chip). In this case, the SoC may further include, in addition to one or more processors (160), a memory (150), and a network interface such as a bus for data communication between the processor (160) and the memory (150).

[0262] When a plurality of processors (160) are included in a SoC (System on Chip) included in an electronic device (100), the electronic device (100) may perform operations related to artificial intelligence (e.g., operations related to learning or inference of an artificial intelligence model) by using some of the plurality of processors (160). For example, the electronic device (100) may perform operations related to artificial intelligence by using at least one of a GPU, an NPU, a VPU, a TPU, and a hardware accelerator specialized in artificial intelligence operations such as convolution operations and matrix multiplication operations among the plurality of processors (160). However, this is merely an example, and it is of course possible to process operations related to artificial intelligence by using a CPU or a general-purpose processor (160).

[0263] In addition, the electronic device (100) can perform operations related to functions related to artificial intelligence by utilizing multiple cores (e.g., dual cores, quad cores, etc.) included in one processor (160). In particular, the electronic device (100) can perform artificial intelligence operations such as convolution operations, matrix multiplication operations, etc. in parallel by utilizing multiple cores included in the processor (160).

[0264] One or more processors (160) are controlled to process input data according to predefined operation rules or artificial intelligence models stored in the memory (150). The predefined operation rules or artificial intelligence models are characterized by being created through learning.

[0265] Here, "created through learning" means that a predefined set of behavioral rules or an AI model with desired characteristics is created by applying a learning algorithm to a large number of learning data. This learning may be performed on the device itself, where the AI ​​according to the present disclosure is implemented, or through a separate server / system.

[0266] An artificial intelligence model may be composed of multiple neural network layers. At least one layer has at least one weight value and performs its operation through the operation result of the previous layer and at least one defined operation. Examples of neural networks include a convolutional neural network (CNN), a deep neural network (DNN), a recurrent neural network (RNN), a restricted boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), deep Q-networks, and a transformer. The neural networks in the present disclosure are not limited to the above-described examples unless otherwise specified.

[0267] A learning algorithm is a method for training a target device (e.g., a robot) using a large amount of learning data, thereby enabling the target device to make decisions or predictions on its own. Examples of learning algorithms include supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning. Unless otherwise specified, the learning algorithms in this disclosure are not limited to the aforementioned examples.

[0268] A device-readable storage medium may be provided in the form of a non-transitory storage medium. Here, the term "non-transitory storage medium" simply means a tangible device that does not contain signals (e.g., electromagnetic waves). This term does not distinguish between cases where data is permanently stored in the storage medium and cases where data is temporarily stored. For example, a "non-transitory storage medium" may include a buffer in which data is temporarily stored.

[0269] According to one or more embodiments, the methods according to the various embodiments disclosed in the present document may be provided as included in a computer program product. The computer program product may be traded as a commodity between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) via an application store (e.g., Play Store™) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product (e.g., downloadable app) may be at least temporarily stored or temporarily generated in a machine-readable storage medium, such as a memory (150) of a manufacturer's server, an application store's server, or a relay server.

[0270] Each of the components (e.g., modules or programs) according to the various embodiments of the present disclosure as described above may be composed of a single or multiple entities, and some of the sub-components described above may be omitted, or other sub-components may be further included in the various embodiments. Alternatively or additionally, some components (e.g., modules or programs) may be integrated into a single entity, which may perform the same or similar functions as those performed by each of the respective components prior to integration.

[0271] According to various embodiments, operations performed by a module, program or other component may be executed sequentially, in parallel, iteratively or heuristically, or at least some operations may be executed in a different order, omitted, or other operations may be added.

[0272] Meanwhile, the terms "part" or "module" used in the present disclosure include units composed of hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A "part" or "module" may be an integrally composed component, a minimum unit performing one or more functions, or a portion thereof. For example, a module may be composed of an application-specific integrated circuit (ASIC).

[0273] Various embodiments of the present disclosure may be implemented as software including commands stored in a machine-readable storage medium that can be read by a machine (e.g., a computer). The device is a device that can call commands stored in the storage medium and operate according to the called commands, and may include an electronic device (100) (e.g., electronic device (100) (100)) according to the disclosed embodiments.

[0274] When the above command is executed by the processor (160), the processor (160) may perform a function corresponding to the command directly or by using other components under the control of the processor (160). The command may include code generated or executed by a compiler or interpreter.

[0275] Although the preferred embodiments of the present disclosure have been illustrated and described above, the present disclosure is not limited to the specific embodiments described above, and various modifications may be made by a person having ordinary skill in the art to which the present disclosure pertains without departing from the gist of the present disclosure as claimed in the claims, and such modifications should not be understood individually from the technical idea or prospect of the present disclosure.

Claims

1. In electronic devices, Department of Communications; camera; Projection section; and Receive status information indicating the operating status of an external device through the above communication unit, Obtain relevant information of the external device from memory, Based on the image of the external device acquired through the camera, at least one area among an area on the surface of the external device and an area around the external device is identified, Based on the relevant information of the external device acquired above, a projection area is identified among the at least one area on which a screen related to the received status information is to be projected, An electronic device comprising a processor for controlling the projection unit to project the screen onto the identified projection area.

2. In paragraph 1, The above processor, Based on the acquired related information and the received status information, identifying the projection area among the at least one area, An electronic device wherein the acquired related information includes at least one of information for identifying the external device, information about the type of the external device, information about the surface of the external device, information about the main operation of the external device, and information about the configuration included in the external device.

3. In paragraph 2, The above processor, Identifying a threshold size for the identified projection area based on at least one of the amount of the received state information and the importance of the received state information, An electronic device that identifies an area on the surface of the external device as the identified projection area if the size of the surface of the external device is greater than or equal to the identified threshold size.

4. In paragraph 3, The above processor, If the size of the surface of the external device is less than the identified threshold size, and if the external device includes a display, identifying an area adjacent to the display among the at least one area around the external device as the identified projection area, An electronic device that identifies an area adjacent to a configuration of the external device performing the primary operation among the at least one area around the external device as the projection area, if the size of the surface is less than the threshold size and the external device does not include the display.

5. In paragraph 2, The above processor, Based on the information about the configuration included in the external device included in the acquired related information, the configuration related to the received status information is identified among the configurations included in the external device, An electronic device that controls the projection unit to project the screen at a position corresponding to the identified configuration in the identified projection area.

6. In paragraph 2, The above processor, Based on the received status information and the acquired related information, identify whether to project the screen through the projection unit, If it is identified that the above screen is not being projected, the communication unit is controlled to transmit the received status information to the user terminal, An electronic device that acquires an image of said external device through said camera when identified as projecting said screen.

7. In paragraph 6, The above processor, An electronic device that identifies whether to project the screen through the projection unit based on the received status information, depending on whether the importance of the received status information corresponds to an emergency.

8. In paragraph 1, further comprising a driving unit; The above processor, An electronic device that controls the driving unit and the projection unit to project the screen onto the identified projection area.

9. In paragraph 8, The above processor, Obtain an image of the user through the above camera, Based on the image of the user, information about the user is obtained, Based on the acquired user information, the acquired related information and the received status information, identifying the identified projection area among the at least one area, An electronic device in which the information about the user obtained includes at least one of information about the user's location, information about the direction in which the user's head is facing, information about the distance between the user and the external device, and information about the user's lifestyle pattern.

10. In paragraph 9, The above processor, If the direction in which the user's head is looking is maintained for a preset period of time, the area corresponding to the direction in which the user's head is looking is identified as the projection area. Based on the received status information, identify whether the external device is in a state of performing a preset operation, When the external device is in a state of performing the operation, the driving unit and the projection unit are controlled to project the screen including information on the progress of the operation, An electronic device that controls the driving unit and the projection unit to project the screen including information for controlling the operation of the external device when the external device is not performing the operation.

11. In paragraph 9, The above processor, If it is identified that the received status information is to be displayed on a plurality of screens based on the received status information, the driving unit and the projection unit are controlled to project a first screen among the plurality of screens onto the identified projection area, An electronic device that controls the driving unit and the projection unit to project a second screen subsequent to the first screen among the plurality of screens when it is determined that the user has performed an action corresponding to the first screen based on the acquired image of the user while the first screen among the plurality of screens is being projected.

12. A method for controlling an electronic device including a communication unit, a camera, and a projection unit, A step of receiving status information indicating the operating status of an external device through the above communication unit; A step of obtaining relevant information of the external device from memory; A step of identifying at least one area among an area on a surface of the external device and an area around the external device based on an image of the external device acquired through the camera; A step of identifying a projection area on which to project a screen related to the received status information among at least one area based on the relevant information of the acquired external device; and A method for controlling an electronic device, comprising: projecting the screen onto the identified projection area; 13. In paragraph 13, The step of identifying the above identified projection area is: A step of identifying the identified projection area among the at least one area based on the acquired related information and the received status information; comprising; A control method of an electronic device, wherein the acquired related information includes at least one of information for identifying the external device, information about the type of the external device, information about the surface of the external device, information about the main operation of the external device, and information about the configuration included in the external device.

14. In paragraph 13, The step of identifying the above identified projection area is: A step of identifying a threshold size for the identified projection area based on at least one of the amount of the received state information and the importance of the received state information; and A control method of an electronic device further comprising: a step of identifying an area on the surface of the external device as the identified projection area if the size of the surface of the external device is greater than or equal to the identified threshold size; 15. In paragraph 14, The step of identifying the above identified projection area is: A step of identifying an area adjacent to the display among the at least one area around the external device as the identified projection area, wherein the size of the surface of the external device is less than the identified threshold size, and the external device includes a display; and A method of controlling an electronic device further comprising: identifying an area adjacent to a configuration of the external device performing the main operation among the at least one area around the external device as the identified projection area, if the size of the surface of the external device is less than the threshold size and the external device does not include the display;

Images