Electronic apparatus and control method thereof

Abstract

This electronic apparatus comprises a sensor unit that acquires sensing data, a memory, a projection unit, and at least one processor, wherein the at least one processor: acquires size information of a face region of a user identified on the basis of the sensing data; on the basis of the size information of the face region, acquires distance information of the face region corresponding to the distance between the position of the electronic apparatus and the position of the user; and controls the projection unit so that a content image is output on the basis of the distance information of the face region.

Description

Electronic device and method of controlling the same

[0001] The present disclosure relates to an electronic device and a control method thereof, and more particularly, to an electronic device and a control method thereof that adjusts the size of a projection image by using the distance between a user and the electronic device.

[0002] Various sensors (e.g., depth cameras, 3D TOF, lidar, etc.) can be used to measure distance information between a user and an electronic device. However, due to various constraints, highly accurate depth sensors may not be utilized to measure distance information.

[0003] Among various sensors, RGB cameras that capture RGB images can be difficult to use when measuring distance information. Using two RGB cameras can improve the accuracy of distance measurement. However, locating two cameras in an electronic device can present challenges due to space constraints and cost.

[0004] For devices that project images, it's necessary to calibrate the projected image based on the relative distance from the user. Failure to account for the distance between the device and the user can result in the image being too large or too small, making it difficult to provide an appropriate image for the user.

[0005] The present disclosure is designed to improve the above-described problem, and an object of the present disclosure is to provide an electronic device that outputs a projection image while taking into account the distance between the electronic device and a user, and a method for controlling the same.

[0006] An electronic device according to one embodiment includes a sensor unit for acquiring sensing data, a memory, a projection unit, and at least one processor, wherein the at least one processor acquires size information of a face area of ​​a user identified based on the sensing data, acquires distance information of the face area corresponding to a distance between a location of the electronic device and a location of the user based on the size information of the face area, and controls the projection unit to output a content image based on the distance information of the face area.

[0007] The at least one processor can obtain the size information including at least one of a horizontal length of the face region, a vertical length of the face region, and an area of ​​the face region.

[0008] The at least one processor may, when a user's face area is identified based on the sensing data, obtain coordinate information of the face area and obtain the size information based on the coordinate information.

[0009] The at least one processor may obtain projection resolution information indicating the size of an image output through the projection unit based on the distance information, and control the projection unit to output the content image based on the projection resolution information.

[0010] The distance information is first distance information, and the at least one processor can obtain second distance information corresponding to a distance between a position of the electronic device and a projection surface, and obtain the projection resolution information based on the first distance information and the second distance information.

[0011] The memory stores a projection ratio range, and the at least one processor can obtain a resolution range based on the projection ratio range and the second distance information, and obtain projection resolution information among the resolution range based on the first distance information.

[0012] The at least one processor can control the projection unit to obtain a scaling factor based on the content resolution information and the projection resolution information, scale the content image based on the scaling factor to obtain a projection image, and output the projection image.

[0013] The at least one processor may control the projection unit to obtain first size information of the first face area and second size information of the second face area when the first face area and the second face area are identified based on the sensing data, obtain a first distance value of the first face area based on the first size information of the first face area, obtain a second distance value of the second face area based on the second size information of the second face area, obtain a representative distance value based on the first distance value and the second distance value, obtain the distance information including the representative distance value, and output the content image based on the distance information.

[0014] The above memory stores an artificial intelligence model including at least one learned weight, and the at least one processor can input the size information into the artificial intelligence model to obtain the distance information as output data.

[0015] The at least one weight may be learned based on first learning data including an RGB image and second learning data including a depth image.

[0016] A control method of an electronic device for acquiring sensing data according to one embodiment includes the steps of acquiring size information of a face area of ​​a user identified based on the sensing data, the step of acquiring distance information of the face area corresponding to a distance between a location of the electronic device and a location of the user based on the size information of the face area, and the step of outputting a content image based on the distance information of the face area.

[0017] The step of obtaining the above size information may obtain the size information including at least one of the horizontal length of the face region, the vertical length of the face region, and the width of the face region.

[0018] The step of obtaining the above size information may include, when the user's face area is identified based on the sensing data, obtaining coordinate information of the face area, and obtaining the size information based on the coordinate information.

[0019] The control method further includes a step of obtaining projection resolution information indicating the size of an image output through the electronic device based on the distance information, and the step of outputting the content image can output the content image based on the projection resolution information.

[0020] The above distance information is first distance information, and the step of obtaining the projection resolution information may obtain second distance information corresponding to a distance between a position of the electronic device and a projection surface, and obtain the projection resolution information based on the first distance information and the second distance information.

[0021] The electronic device stores a projection ratio range, and the step of obtaining the projection resolution information may obtain a resolution range based on the projection ratio range and the second distance information, and may obtain the projection resolution information among the resolution range based on the first distance information.

[0022] The step of outputting the above content image may include obtaining a scaling factor based on content resolution information and the projection resolution information, scaling the content image based on the scaling factor to obtain a projection image, and outputting the projection image.

[0023] The step of obtaining the size information may include, when a first face area and a second face area are identified based on the sensing data, obtaining first size information of the first face area and second size information of the second face area, and the step of obtaining the distance information may include obtaining a first distance value of the first face area based on the first size information of the first face area, obtaining a second distance value of the second face area based on the second size information of the second face area, obtaining a representative distance value based on the first distance value and the second distance value, obtaining the distance information including the representative distance value, and the step of outputting the content image may output the content image based on the distance information.

[0024] The electronic device stores an artificial intelligence model including at least one learned weight, and the step of obtaining the distance information may input the size information into the artificial intelligence model to obtain the distance information as output data.

[0025] The at least one weight may be learned based on first learning data including an RGB image and second learning data including a depth image.

[0026] FIG. 1 is a drawing for explaining an operation of determining the size of an image output through an electronic device according to one embodiment.

[0027] FIG. 2 is a block diagram illustrating an electronic device according to one embodiment.

[0028] FIG. 3 is a block diagram illustrating a specific configuration of the electronic device of FIG. 2, according to one embodiment.

[0029] FIG. 4 is a diagram for explaining an operation of obtaining distance information of a face area according to one embodiment.

[0030] FIG. 5 is a diagram for explaining an operation of detecting a face area in an input image according to one embodiment.

[0031] FIG. 6 is a drawing for explaining an operation of displaying distance information of a face area according to one embodiment.

[0032] FIG. 7 is a diagram for explaining an operation using an artificial intelligence model according to one embodiment.

[0033] FIG. 8 is a diagram for explaining an operation of learning an artificial intelligence model according to one embodiment.

[0034] FIG. 9 is a diagram for explaining an image alignment operation according to one embodiment.

[0035] FIG. 10 is a diagram for explaining an image alignment operation according to one embodiment.

[0036] FIG. 11 is a diagram for explaining an operation of extracting a distance value related to a face area according to one embodiment.

[0037] FIG. 12 is a diagram for explaining an operation of obtaining distance information of a face area according to one embodiment.

[0038] FIG. 13 is a drawing for explaining an operation of outputting a projection image using distance information of a face area, according to one embodiment.

[0039] FIG. 14 is a drawing for explaining an operation of outputting a projection image using the distance to a projection surface according to one embodiment.

[0040] FIG. 15 is a diagram for explaining an operation of identifying multiple face regions according to one embodiment.

[0041] FIG. 16 is a diagram for explaining an operation of obtaining distance information of each of a plurality of face regions according to one embodiment.

[0042] FIG. 17 is a drawing for explaining an operation of controlling the distance between a user and an electronic device according to one embodiment.

[0043] FIG. 18 is a drawing for explaining an operation of avoiding a user according to one embodiment.

[0044] FIG. 19 is a drawing for explaining a guide screen related to a user and an electronic device according to one embodiment.

[0045] FIG. 20 is a drawing for explaining a control operation of an electronic device according to one embodiment.

[0046] Hereinafter, the present disclosure will be described in detail with reference to the attached drawings.

[0047] The terms used in the embodiments of this disclosure have been selected from widely used, current terms, taking into account the functions of this disclosure. However, these terms may vary depending on the intentions of those skilled in the art, precedents, the emergence of new technologies, etc. In certain cases, terms may be arbitrarily selected by the applicant, and in such cases, their meanings will be described in detail in the description of the relevant disclosure. Therefore, the terms used in this disclosure should not be defined simply as names of terms, but rather based on the meanings of the terms and the overall content of this disclosure.

[0048] In this specification, expressions such as “has,” “can have,” “includes,” or “may include” indicate the presence of a feature (e.g., a number, function, operation, or component such as a part), and do not exclude the presence of additional features.

[0049] The expression "at least one of A and / or B" should be understood to mean either "A" or "B" or "A and B".

[0050] As used herein, the expressions “first,” “second,” “first,” or “second,” etc., may describe various components, regardless of order and / or importance, and are only used to distinguish one component from another, but do not limit the components.

[0051] When it is said that a component (e.g., a first component) is “(operatively or communicatively) coupled with / to” or “connected to” another component (e.g., a second component), it should be understood that the component may be directly coupled to the other component, or may be connected through another component (e.g., a third component).

[0052] Singular expressions include plural expressions unless the context clearly dictates otherwise. In this application, terms such as "comprise" or "consist of" are intended to indicate the presence of a feature, number, step, operation, component, part, or combination thereof described in the specification, but should be understood not to preclude the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0053] In the present disclosure, a "module" or "part" performs at least one function or operation and may be implemented in hardware or software, or a combination of hardware and software. Multiple "modules" or multiple "parts" may be integrated into at least one module and implemented by at least one processor (not shown), excluding any "modules" or "parts" that need to be implemented in specific hardware.

[0054] In this specification, the term user may refer to a person using an electronic device or a device using an electronic device (e.g., an artificial intelligence electronic device).

[0055] An embodiment of the present disclosure will be described in more detail with reference to the attached drawings below.

[0056] FIG. 1 is a drawing for explaining an operation of determining the size of an image output through an electronic device (100) according to one embodiment.

[0057] The electronic device (100) may be a device that outputs an image (or a projected image). The electronic device (100) may output the image to a projection surface (10). The electronic device (100) may obtain (or determine or identify) the size of the projected image using the distance between the projection surface (10) and the user (20). The size of the projected image may be included in the resolution information of the projected image. The electronic device (100) may output the projected image based on the size of the projected image.

[0058] According to embodiment (1), the distance between the projection surface (10) and the user (20) may be a first distance (d1). The electronic device (100) may determine the size of the projection image as a first size (i1) based on the first distance (d1).

[0059] According to embodiment (2), the distance between the projection surface (10) and the user (20) may be a second distance (d2). The electronic device (100) may determine the size of the projection image as a second size (i2) based on the second distance (d2).

[0060] If the first distance (d1) is smaller than the second distance (d2), the first size (i1) may be smaller than the second size (i2). The farther the user (20) is from the projection surface (10), the larger the size of the projection image may be.

[0061] The first size (i1) and the second size (i2) are described as the vertical height of the projection image. Depending on the implementation example, the first size (i1) and the second size (i2) may include at least one of the vertical length, horizontal length, or diagonal length of the projection image.

[0062] The electronic device (100) may be implemented as a mobile device. The electronic device (100) may be implemented as a mobile projector or a mobile image output device.

[0063] The electronic device (100) may include a movable member. The movable member may refer to a member that moves from a first position to a second position in a space where the electronic device (100) is placed. The electronic device (100) may control the movable member so that the electronic device (100) moves using a force generated by the driving unit (120). The electronic device (100) may generate a force to be transmitted to the movable member using a motor included in the driving unit (120).

[0064] The movable member may include at least one wheel (e.g., a circular wheel). The electronic device (100) may move to a target location (or target position) through the movable member. When a user input or a control command is received, the electronic device (100) may rotate the movable member by transmitting a force generated through a motor to the movable member. The electronic device (100) may control the movable member to adjust the rotation speed, rotation direction, etc. The electronic device (100) may perform a movement operation (or movement function) by controlling the movable member based on the target location or the direction of movement, etc.

[0065] As illustrated in FIG. 1, the electronic device (100) may be positioned lower than the user's (20) face. In order for the electronic device (100) to capture the user's (20) face, it is necessary to perform a capturing operation from bottom to top. If the capturing operation is performed from bottom to top, the quality (e.g., brightness) of the captured image may be poor.

[0066] For example, when capturing a user's (20) face from below in a dark environment, the electronic device (100) must identify (or analyze) the user's (20) face using a dark image. When using a dark image, the analysis of the user's (20) face may not be accurate.

[0067] The electronic device (100) can be effectively applied to obtain the distance between a user (20) and the electronic device (100) based on an image captured from below in an upward direction or an image of poor quality (e.g., an image with an average pixel brightness below a threshold value).

[0068] FIG. 2 is a block diagram illustrating an electronic device (100) according to one embodiment.

[0069] Referring to FIG. 2, the electronic device (100) may include at least one of a processor (111), a projection unit (112), a memory (113), and a sensor unit (121).

[0070] At least one processor (111) can perform overall control operations of the electronic device (100). Specifically, at least one processor (111) functions to control the overall operations of the electronic device (100). A specific description related to at least one processor (111) is described in FIG. 3.

[0071] The projection unit (112) is a configuration that projects images (projected images, contents, etc.) to the outside. A specific description related to the projection unit (112) is described in Fig. 3.

[0072] The memory (113) can store a projection image projected through the projection unit (112). The projection image may refer to a still image as well as a continuous image (or a moving image). The projection image may refer to an image included in the content.

[0073] The sensor unit (121) can acquire sensing data. The memory (113) can store content images. The projection unit (112) can be used to output images.

[0074] At least one processor (111) can control operations performed in the electronic device (100).

[0075] At least one processor (111) can obtain size information of the face area when the face area of ​​the user (20) is detected based on the sensing data, obtain distance information of the face area corresponding to the distance between the position of the electronic device (100) that obtained the sensing data based on the size information of the face area and the position of the user (20), and output a content image based on the distance information through the projection unit (112).

[0076] Distance information of the face area may include the distance between the location of the electronic device (100) and the location of the user (20).

[0077] At least one processor (111) can obtain sensing data through the sensor unit (121). When a preset event (e.g., user input) is identified, at least one processor (111) can transmit a control command to the sensor unit (121) for obtaining sensing data. At least one processor (111) can receive sensing data through the sensor unit (121) in response to the control command.

[0078] The sensing data may include information related to the user (20). Information related to the user (20) may include

[0079] For example, the sensing data may include an RGB image. The sensor unit (121) may include an RGB camera.

[0080] For example, the sensing data may include an infrared image. The sensor unit (121) may include an infrared sensor.

[0081] For example, the sensing data may include a depth image. The sensor unit (121) may include a depth camera.

[0082] For example, the sensing data may include lidar data. The sensor unit (121) may include a lidar sensor.

[0083] At least one processor (111) can determine whether a user (20) is detected (or identified) based on the sensing data. At least one processor (111) can detect a human face in the sensing data. At least one processor (111) can detect a facial region representing a human face. At least one processor (111) can detect a facial region.

[0084] When a face region is detected in the sensing data, at least one processor (111) can obtain size information of the face region. The size information of the face region can include at least one element necessary to indicate the size of the face region in the sensing data.

[0085] At least one processor (111) can obtain size information including at least one of the horizontal length of the face region, the vertical length of the face region, and the area of ​​the face region. At least one processor (111) can calculate at least one of the horizontal length of the face region, the vertical length of the face region, and the area of ​​the face region based on the sensing data.

[0086] At least one processor (111) can obtain distance information of a face area corresponding to the distance between a user (20) and an electronic device (100) based on size information of the face area. At least one processor (111) can obtain distance information of a face area corresponding to the distance between a location of an electronic device (100) that sensed sensing data and a location where the face of the user (20) exists.

[0087] According to various embodiments, at least one processor (111) may additionally obtain rotation axis information (yaw, pitch, roll) related to the head direction of the user (20). At least one processor (111) may additionally obtain distance information of the face area by using the rotation axis information (yaw, pitch, roll) related to the head direction of the user (20).

[0088] According to various embodiments, at least one processor (111) may detect a feature object (or landmark object) related to a face area of ​​a user (20). The feature object may include at least one of a human eye, nose, and mouth. At least one processor (111) may obtain distance information of the face area using the feature object. An operation using the feature object may include an operation using the location of the feature object.

[0089] At least one processor (111) can obtain coordinate information of the face area when the face area of ​​the user (20) is detected based on sensing data, and obtain size information based on the coordinate information.

[0090] Coordinate information can be used to indicate a specific pixel or object in the sensing data. At least one processor (111) can use the coordinate information to analyze where the facial area is located.

[0091] At least one processor (111) can obtain coordinate information of a face area that indicates where the face area included in the sensing data is positioned in the sensing data.

[0092] Coordinate information may include multiple coordinate values, as the face region may not be composed of a single pixel. While the center point or reference point may be a single pixel, the face region itself may include multiple coordinate values. The coordinate information of the face region may include multiple coordinate values ​​indicating the position of the user's (20) face.

[0093] At least one processor (111) can obtain size information of a face area based on coordinate information.

[0094] For example, at least one processor (111) can obtain the leftmost x-value (first value) among the coordinate information of the face region. At least one processor (111) can obtain the rightmost x-value (second value) among the coordinate information of the face region. At least one processor (111) can obtain the difference value between the leftmost x-value (first value) and the rightmost x-value (second value) as the horizontal length of the face region.

[0095] For example, at least one processor (111) can obtain the uppermost y value (third value) among the coordinate information of the face region. At least one processor (111) can obtain the lowermost y value (fourth value) among the coordinate information of the face region. At least one processor (111) can obtain the difference value between the uppermost y value (third value) and the lowermost y value (fourth value) as the vertical length of the face region.

[0096] For example, at least one processor (111) can obtain the width (or area) of the inner direction of the face region by considering coordinate information of the face region.

[0097] At least one processor (111) can generate a face box (or face GUI) based on size information of a face region. At least one processor (111) can obtain a captured image with a face box applied to a face portion of a user (20) based on coordinate information and size information of the face region.

[0098] The memory (113) stores content resolution information, and at least one processor (111) obtains projection resolution information indicating the size of an image output through the projection unit (112) based on distance information, and can output a content image based on the projection resolution information.

[0099] The content resolution information may include the original resolution associated with the content image stored in the memory (113). The content resolution information may include information indicating the size of the content image. The content resolution information may include information indicating the number of pixels of the content image.

[0100] Projection resolution information may include a corrected resolution (or changed resolution) related to a projection image output through the projection unit (112). Projection resolution information may include information indicating the size of the projection image.

[0101] The content image may be an original image included in the content, and the projection image may be an image corrected to output the content image through the projection unit (112). At least one processor (111) may convert the content image into a projection image. If the resolution of the content and the resolution that the projection unit (112) can provide are different, the quality of the projection image may deteriorate. At least one processor (111) may obtain the projection image. The conversion operation may be described as a scaling operation.

[0102] At least one processor (111) can obtain a scaling factor based on content resolution information and projection resolution information, scale a content image based on the scaling factor to obtain a projection image, and output the projection image through a projection unit (112).

[0103] At least one processor (111) can obtain a scaling factor based on content resolution information and projection resolution information. At least one processor (111) can scale a content image based on the scaling factor. At least one processor (111) can obtain a projection image as a result of the scaling function. A description related to this is provided in FIG. 13.

[0104] The distance information is first distance information, and at least one processor (111) can obtain second distance information corresponding to the distance between the position of the electronic device and the projection surface, and can obtain projection resolution information based on the first distance information and the second distance information. A description related to this is described in FIG. 14.

[0105] The first distance information may include a distance value between the electronic device (100) and the user (20). The second distance information may include a distance value between the electronic device (100) and the projection surface (10).

[0106] The memory (113) stores a projection ratio range, and at least one processor (111) can obtain a resolution range based on the projection ratio range and second distance information, and can obtain projection resolution information among the resolution range based on the first distance information.

[0107] The projection ratio range may include a range of projection ratios that can be provided by the electronic device (100). The projection ratio range may be determined based on the hardware characteristics of the projection unit (112). The projection ratio range may vary depending on the performance of the lens included in the projection unit (112). At least one processor (111) may obtain a projection ratio range corresponding to the electronic device (100) stored in the memory (113).

[0108] At least one processor (111) may output an out-of-focus image if the projection image is output outside the projection ratio range. For example, if the projection image is output outside the projection ratio range, the user may see a blurry image. It is necessary for at least one processor (111) to output a projection image within the projection ratio range.

[0109] The projection ratio can be calculated as the projection distance divided by the size of the projected image. The size of the projected image can include the diagonal length of the projected image.

[0110] The projection distance may correspond to the second distance information. At least one processor (111) may obtain a resolution range based on the second distance information (projection distance) and the projection ratio range. The resolution range may include a size range of a projection image that can be provided from the current location of the electronic device (100). The resolution range or size range may include a diagonal length of the projection image.

[0111] At least one processor (111) can acquire projection resolution information by considering first distance information within a resolution range. Since the resolution range corresponds to the size of a projection image currently available from the electronic device (100), at least one processor (111) can acquire projection resolution information for the user (20) by using the first distance information.

[0112] At least one processor (111) can obtain a maximum value (first value) within a resolution range. At least one processor (111) can obtain a minimum value (second value) within a resolution range.

[0113] At least one processor (111) can obtain a threshold constant based on the first distance information.

[0114] If the first distance information is included in the first threshold range, at least one processor (111) can obtain the first threshold constant. If the first distance information is included in the second threshold range, at least one processor (111) can obtain the second threshold constant. There may be multiple threshold ranges and threshold constants, respectively.

[0115] At least one processor (111) can obtain a value (third value) obtained by multiplying a maximum value (first value) within a resolution range by a threshold constant. At least one processor (111) can obtain a larger value (fourth value) between a minimum value (second value) within a resolution range and the multiplied value (third value) as projection resolution information.

[0116] For example, assume that the second distance information is 2 m and the projection ratio range is 1 to 2. The resolution range (or size range) may be 1 to 2 m. If the first distance information is 0.1 m, at least one processor (111) may determine the projection resolution information as 1 m. If the first distance information is 10 m, at least one processor (111) may determine the projection resolution information as 2 m. At least one processor (111) may obtain a threshold constant (c) corresponding to the first distance information (0.1 m). At least one processor (111) may obtain a value (2m*c) obtained by multiplying the threshold constant (c) and the maximum value (2m). At least one processor (111) may determine a larger value between the multiplied value (2m*c) and the minimum value (1m) as the projection resolution information.

[0117] As the user (20) gets closer to the electronic device (100), at least one processor (111) can control the size of the projected image to be smaller within the resolution range. As the user (20) gets farther away from the electronic device (100), at least one processor (111) can control the size of the projected image to be larger within the resolution range.

[0118] At least one processor (111) can obtain first size information of the first face area and second size information of the second face area when a first face area and a second face area are detected based on sensing data. At least one processor (111) can obtain a first distance value of the first face area based on the first size information of the first face area. At least one processor (111) can obtain a second distance value of the second face area based on the second size information of the second face area. At least one processor (111) can obtain a representative distance value based on the first distance value and the second distance value. At least one processor (111) can obtain distance information including the representative distance value and output a content image based on the distance information through the projection unit (112). A description related to this is described in FIGS. 15 and 16.

[0119] The distance between a user (20) and the electronic device (100) can be accurately measured based on a captured image acquired through a sensor of the electronic device (100).

[0120] Referring to FIG. 3, the electronic device (100) may include at least one of a processor (111), a projection unit (112), a memory (113), a communication interface (114), an operation interface (115), an input / output interface (116), a speaker (117), a microphone (118), a power supply unit (119), a driving unit (120), or a sensor unit (121).

[0121] The configuration illustrated in FIG. 3 is merely an example of various embodiments, and some configurations may be omitted and new configurations may be added.

[0122] The content already explained in Fig. 2 is omitted.

[0123] At least one processor (111) may be implemented as a digital signal processor (DSP), a microprocessor, or a time controller (TCON) that processes a digital signal. However, the present invention is not limited thereto, and may include one or more of a central processing unit (CPU), a micro controller unit (MCU), a micro processing unit (MPU), a controller, an application processor (AP), a graphics-processing unit (GPU), a communication processor (CP), or an advanced reduced instruction set computer (RISC) machines (ARM) processor, or may be defined by the relevant terminology. At least one processor (111) may be implemented as a system on chip (SoC) having a built-in processing algorithm, a large scale integration (LSI), or may be implemented in the form of a field programmable gate array (FPGA). At least one processor (111) may perform various functions by executing computer executable instructions stored in a memory (113).

[0124] The projection unit (112) is a component that projects an image to the outside. According to various embodiments of the present disclosure, the projection unit (112) can be implemented in various projection methods (e.g., CRT (cathode-ray tube) method, LCD (Liquid Crystal Display) method, DLP (Digital Light Processing) method, laser method, etc.). For example, the CRT method has the same principle as a CRT monitor. The CRT method magnifies the image with a lens in front of the cathode-ray tube (CRT) and displays the image on the screen. Depending on the number of cathode-ray tubes, it is divided into a single-tube type and a three-tube type, and in the case of a three-tube type, the red, green, and blue cathode-ray tubes can be implemented separately.

[0125] Another example is the LCD method, which displays images by passing light from a light source through liquid crystals. LCD methods are divided into single-panel and three-panel types. In the case of the three-panel type, light from a light source is separated into red, green, and blue by a dichroic mirror (a mirror that reflects only a specific color of light and transmits all others). After passing through the liquid crystals, the light can be refocused into a single point.

[0126] Another example is the DLP method, which displays images using a DMD (Digital Micromirror Device) chip. The DLP projection unit may include a light source, a color wheel, a DMD chip, a projection lens, etc. The light output from the light source can be colored as it passes through the rotating color wheel. The light passing through the color wheel is input to the DMD chip. The DMD chip contains numerous micromirrors and reflects the light input to the DMD chip. The projection lens can play a role in magnifying the light reflected from the DMD chip to the image size.

[0127] Another example is a laser system that uses a Diode Pumped Solid State (DPSS) laser and a galvanometer. A multi-color laser uses three DPSS lasers, one for each RGB color, with their optical axes overlapped by a special mirror. The galvanometer includes a mirror and a high-power motor that moves the mirror at high speeds. For example, the galvanometer can rotate the mirror at up to 40 kHz / sec. The galvanometer is mounted according to the scan direction, but since projectors typically scan in a planar manner, the galvanometer can also be positioned separately along the x and y axes.

[0128] The projection unit (112) may include various types of light sources. For example, the projection unit (112) may include at least one light source among a lamp, an LED, and a laser.

[0129] The projection unit (112) can output images in a 4:3 screen ratio, a 5:4 screen ratio, or a 16:9 wide screen ratio depending on the purpose of the electronic device (100) or the user's settings, and can output images in various resolutions such as WVGA (854*480), SVGA (800*600), XGA (1024*768), WXGA (1280*720), WXGA (1280*800), SXGA (1280*1024), UXGA (1600*1200), and Full HD (1920*1080) depending on the screen ratio.

[0130] The projection unit (112) can perform various functions for adjusting the output image under the control of at least one processor (111). For example, the projection unit (112) can perform functions such as zoom, keystone, quick corner (4 corner) keystone, and lens shift.

[0131] Specifically, the projection unit (112) can enlarge or reduce the image depending on the distance from the screen (projection distance). That is, the zoom function can be performed depending on the distance from the screen. At this time, the zoom function may include a hardware method of adjusting the screen size by moving the lens and a software method of adjusting the screen size by cropping the image, etc. When the zoom function is performed, the focus of the image needs to be adjusted. For example, the method of adjusting the focus includes a manual focus method, an electric focus method, etc. The manual focus method refers to a method of focusing manually, and the electric focus method refers to a method of automatically focusing using a motor built into the projector when the zoom function is performed. When performing the zoom function, the projection unit (112) may provide a digital zoom function through software, and may provide an optical zoom function of performing the zoom function by moving the lens through the driving unit (120).

[0132] The projection unit (112) can perform a keystone correction function. If the height is not right for front projection, the screen may be distorted upwards or downwards. The keystone correction function refers to a function that corrects a distorted screen. For example, if distortion occurs in the left and right directions of the screen, it can be corrected using horizontal keystone, and if distortion occurs in the up and down directions, it can be corrected using vertical keystone. The quick corner (4 corner) keystone correction function is a function that corrects the screen when the center area of ​​the screen is normal but the corner areas are not balanced. The lens shift function is a function that moves the screen as it is when the screen is off the screen.

[0133] The projection unit (112) can automatically analyze the surrounding environment and projection environment without user input to provide zoom / keystone / focus functions. Specifically, the projection unit (112) can automatically provide zoom / keystone / focus functions based on the distance between the electronic device (100) and the screen detected by a sensor (depth camera, distance sensor, infrared sensor, light sensor, etc.), information about the space where the electronic device (100) is currently located, information about the amount of ambient light, etc.

[0134] The projection unit (112) can provide a lighting function using a light source. In particular, the projection unit (112) can provide a lighting function by outputting a light source using an LED. According to various embodiments, the projection unit (112) can include one LED, and according to other embodiments, the electronic device (100) can include a plurality of LEDs. The projection unit (112) can output a light source using a surface-emitting LED according to an implementation example. The surface-emitting LED can refer to an LED having a structure in which an optical sheet is arranged on the upper side of the LED so that the light source is evenly distributed and output. Specifically, when a light source is output through the LED, the light source can be evenly distributed through the optical sheet, and the light source distributed through the optical sheet can be incident on the display panel.

[0135] The projection unit (112) can provide the user with a dimming function for adjusting the intensity of the light source. Specifically, when a user input for adjusting the intensity of the light source is received from the user through the operation interface (115) (e.g., a touch display button or dial), the projection unit (112) can control the LED to output the intensity of the light source corresponding to the received user input.

[0136] The projection unit (112) can provide a dimming function based on content analyzed by at least one processor (111) without user input. Specifically, the projection unit (112) can control the LED to output the intensity of a light source based on information about the currently provided content (e.g., content type, content brightness, etc.).

[0137] The projection unit (112) can control the color temperature under the control of at least one processor (111). The at least one processor (111) can control the color temperature based on the content. Specifically, when the content is identified to be output, the at least one processor (111) can obtain frame-by-frame color information of the content whose output has been determined. Then, the at least one processor (111) can control the color temperature based on the obtained frame-by-frame color information. The at least one processor (111) can obtain at least one primary color of the frame based on the frame-by-frame color information. Then, the at least one processor (111) can adjust the color temperature based on the obtained at least one primary color. For example, the color temperature that the at least one processor (111) can adjust can be classified into a warm type or a cold type. It is assumed that a frame to be output (hereinafter, referred to as an output frame) includes a scene in which a fire has occurred. At least one processor (111) can identify (or obtain) that the primary color is red based on color information included in the current output frame. In addition, at least one processor (111) can identify a color temperature corresponding to the identified primary color (red). The color temperature corresponding to red may be a warm type. At least one processor (111) can use an artificial intelligence model to obtain the color information or primary color of the frame. According to various embodiments, the artificial intelligence model can be stored in the electronic device (100) (e.g., the memory (113)). According to another embodiment, the artificial intelligence model can be stored in an external server that can communicate with the electronic device (100).

[0138] The memory (113) may be implemented as an internal memory such as a ROM (e.g., an electrically erasable programmable read-only memory (EEPROM)) or RAM included in at least one processor (111), or may be implemented as a separate memory from at least one processor (111). In this case, the memory (113) may be implemented as a memory embedded in the electronic device (100) or as a memory detachable from the electronic device (100) depending on the purpose of data storage. For example, data for driving the electronic device (100) may be stored in a memory embedded in the electronic device (100), and data for expanding functions of the electronic device (100) may be stored in a memory detachable from the electronic device (100).

[0139] In the case of memory embedded in the electronic device (100), it may be implemented as at least one of volatile memory (e.g., dynamic RAM (DRAM), static RAM (SRAM), or synchronous dynamic RAM (SDRAM)), non-volatile memory (e.g., one time programmable ROM (OTPROM), programmable ROM (PROM), erasable and programmable ROM (EPROM), electrically erasable and programmable ROM (EEPROM), mask ROM, flash ROM, flash memory (e.g., NAND flash or NOR flash), etc.), hard drive, or solid state drive (SSD), and in the case of memory that can be attached or detached to the electronic device (100), it may be implemented in the form of a memory card (e.g., compact flash (CF), secure digital (SD), micro secure digital (Micro-SD), mini secure digital (Mini-SD), extreme digital (xD), multi-media card (MMC), etc.), external memory that can be connected to a USB port (e.g., USB memory), etc.

[0140] The memory (113) may store at least one command regarding the electronic device (100). In addition, the memory (113) may store an O / S (Operating System) for driving the electronic device (100). The memory (113) may also store various software programs or applications for operating the electronic device (100) according to various embodiments of the present disclosure. In addition, the memory (113) may include a semiconductor memory such as a flash memory or a magnetic storage medium such as a hard disk.

[0141] Specifically, various software modules for operating the electronic device (100) according to various embodiments of the present disclosure may be stored in the memory (113), and at least one processor (111) may control the operation of the electronic device (100) by executing various software modules stored in the memory (113). That is, the memory (113) is accessed by at least one processor (111), and data reading / recording / modifying / deleting / updating, etc. may be performed by at least one processor (111).

[0142] In the present disclosure, the term memory (113) may be used to mean a storage unit, a ROM (not shown), a RAM (not shown) in at least one processor (111), or a memory card (not shown) (e.g., a micro SD card, a memory stick) mounted on an electronic device (100).

[0143] The communication interface (114) is a configuration that performs communication with various types of external devices according to various types of communication methods. The communication interface (114) may include a wireless communication module or a wired communication module. Each communication module may be implemented in the form of at least one hardware chip.

[0144] A wireless communication module may be a module that communicates wirelessly with an external device. For example, the wireless communication module may include at least one of a Wi-Fi module, a Bluetooth module, an infrared communication module, or other communication modules.

[0145] Wi-Fi and Bluetooth modules can communicate via Wi-Fi and Bluetooth, respectively. When using a Wi-Fi or Bluetooth module, various connection information, such as the service set identifier (SSID) and session key, is first transmitted and received. This information is then used to establish a communication connection before various other information can be transmitted and received.

[0146] Infrared communication modules perform communication based on infrared communication (IrDA, infrared Data Association) technology, which transmits data wirelessly over short distances using infrared light, which is between visible light and millimeter waves.

[0147] In addition to the above-described communication method, other communication modules may include at least one communication chip that performs communication according to various wireless communication standards such as zigbee, 3G (3rd Generation), 3GPP (3rd Generation Partnership Project), LTE (Long Term Evolution), LTE-A (LTE Advanced), 4G (4th Generation), 5G (5th Generation), etc.

[0148] A wired communication module may be a module that communicates with an external device via a wire. For example, the wired communication module may include at least one of a Local Area Network (LAN) module, an Ethernet module, a paired cable, a coaxial cable, a fiber optic cable, or an Ultra Wide-Band (UWB) module.

[0149] The manipulation interface (115) may include various types of input devices. For example, the manipulation interface (115) may include a physical button. In this case, the physical button may include a function key, a directional key (e.g., a four-way key), or a dial button. According to various embodiments, the physical button may be implemented as multiple keys. According to another embodiment, the physical button may be implemented as one key. When the physical button is implemented as one key, the electronic device (100) may receive a user input in which one key is pressed for a threshold time or longer. When a user input in which one key is pressed for a threshold time or longer is received, at least one processor (111) may perform a function corresponding to the user input. For example, at least one processor (111) may provide a lighting function based on the user input.

[0150] The manipulation interface (115) can receive user input using a non-contact method. When receiving user input using a contact method, physical force must be transmitted to the electronic device (100). Therefore, a method for controlling the electronic device (100) regardless of physical force may be required. Specifically, the manipulation interface (115) can receive user gestures and perform operations corresponding to the received user gestures. The manipulation interface (115) can receive user gestures through a sensor (e.g., an image sensor or an infrared sensor).

[0151] The manipulation interface (115) can receive user input using a touch method. For example, the manipulation interface (115) can receive user input via a touch sensor. According to various embodiments, the touch method can be implemented in a non-contact manner. For example, the touch sensor can determine whether the user's body has approached within a threshold distance. The touch sensor can identify user input even when the user does not touch the touch sensor. According to another implementation example, the touch sensor can identify user input when the user touches the touch sensor.

[0152] The electronic device (100) can receive user input in various ways other than the above-described operation interface (115). In various embodiments, the electronic device (100) can receive user input through an external remote control device. The external remote control device can be a remote control device corresponding to the electronic device (100) (e.g., a dedicated control device of the electronic device (100)) or a user's portable communication device (e.g., a smartphone or wearable device). The user's portable communication device can store an application for controlling the electronic device (100). The portable communication device can obtain user input through the stored application and transmit the obtained user input to the electronic device (100). The electronic device (100) can receive user input from the portable communication device and perform an operation corresponding to the user's control command.

[0153] The electronic device (100) can receive user input using voice recognition. According to various embodiments, the electronic device (100) can receive the user's voice through a microphone included in the electronic device (100). According to other embodiments, the electronic device (100) can receive the user's voice from a microphone or an external device. Specifically, the external device can acquire the user's voice through the microphone of the external device and transmit the acquired user's voice to the electronic device (100). The user's voice transmitted from the external device can be audio data or digital data converted from audio data (e.g., audio data converted into a frequency domain, etc.). The electronic device (100) can perform an operation corresponding to the received user's voice. Specifically, the electronic device (100) can receive audio data corresponding to the user's voice through the microphone. In addition, the electronic device (100) can convert the received audio data into digital data. In addition, the electronic device (100) can convert the converted digital data into text data using the STT (Speech To Text) function. According to various embodiments, the STT (Speech To Text) function can be performed directly in the electronic device (100).

[0154] According to another embodiment, the STT (Speech To Text) function may be performed by an external server. The electronic device (100) may transmit digital data to the external server. The external server may convert the digital data into text data and obtain control command data based on the converted text data. The external server may transmit the control command data (which may also include text data) to the electronic device (100). The electronic device (100) may perform an operation corresponding to the user's voice based on the obtained control command data.

[0155] The electronic device (100) may provide a voice recognition function using a single assistant (or artificial intelligence assistant, e.g., Bixby™), but this is merely an example and the electronic device (100) may provide the voice recognition function using multiple assistants. In this case, the electronic device (100) may provide the voice recognition function by selecting one of the multiple assistants based on a trigger word corresponding to the assistant or a specific key present on the remote control.

[0156] The electronic device (100) can receive user input using screen interaction. Screen interaction may refer to a function of identifying whether a predetermined event occurs through an image projected by the electronic device (100) on the screen (or projection surface) and acquiring user input based on the predetermined event. The predetermined event may refer to an event in which a predetermined object is identified at a specific location (e.g., a location where a UI for receiving user input is projected). The predetermined object may include at least one of a part of the user's body (e.g., a finger), a pointer, or a laser point. If the predetermined object is identified at a location corresponding to the projected UI, the electronic device (100) can identify that a user input for selecting the projected UI has been received. For example, the electronic device (100) may project a guide image to display the UI on the screen. In addition, the electronic device (100) can identify whether the user selects the projected UI. Specifically, the electronic device (100) can identify that the user has selected the projected UI when a predetermined event is identified at a location of the projected UI. The projected UI can include at least one item. The electronic device (100) can perform spatial analysis to identify whether the predetermined event is at the location of the projected UI. The electronic device (100) can perform spatial analysis through a sensor (e.g., an image sensor, an infrared sensor, a depth camera, a distance sensor, etc.). By performing the spatial analysis, the electronic device (100) can identify whether the predetermined event occurs at a specific location (a location where the UI is projected). In addition, when the predetermined event is identified as occurring at a specific location (a location where the UI is projected), the electronic device (100) can identify that a user input for selecting a UI corresponding to the specific location has been received.

[0157] The input / output interface (116) is configured to input / output at least one of an audio signal and an image signal. The input / output interface (116) can receive at least one of an audio signal and an image signal from an external device, and can output a control command to the external device.

[0158] Depending on the implementation example, the input / output interface (116) may be implemented as an interface that inputs / outputs only audio signals and an interface that inputs / outputs only image signals, or may be implemented as one interface that inputs / outputs both audio signals and image signals.

[0159] In various embodiments of the present disclosure, the input / output interface (116) may be implemented as at least one wired input / output interface among HDMI (High Definition Multimedia Interface), MHL (Mobile High-Definition Link), USB (Universal Serial Bus), USB C-type, DP (Display Port), Thunderbolt, VGA (Video Graphics Array) port, RGB port, D-SUB (Dsubminiature), and DVI (Digital Visual Interface). According to various embodiments, the wired input / output interface may be implemented as an interface that inputs / outputs only audio signals and an interface that inputs / outputs only image signals, or may be implemented as one interface that inputs / outputs both audio signals and image signals.

[0160] The electronic device (100) can receive data via a wired input / output interface, but this is merely an example of various embodiments, and can also receive power via the wired input / output interface. For example, the electronic device (100) can receive power from an external battery via a USB C-type or from an outlet via a power adapter. As another example, the electronic device (100) can receive power from an external device (e.g., a laptop or monitor) via a DP.

[0161] The audio signal may be implemented to be input through a wired input / output interface, and the image signal may be implemented to be input through a wireless input / output interface (or communication interface). Alternatively, the audio signal may be implemented to be input through a wireless input / output interface (or communication interface), and the image signal may be implemented to be input through a wired input / output interface.

[0162] The speaker (117) is a component that outputs an audio signal. In particular, the speaker (117) may include an audio output mixer, an audio signal processor, and an audio output module. The audio output mixer may synthesize a plurality of audio signals to be output into at least one audio signal. For example, the audio output mixer may synthesize an analog audio signal and another analog audio signal (e.g., an analog audio signal received from the outside) into at least one analog audio signal. The audio output module may include a speaker or an output terminal. According to various embodiments, the audio output module may include a plurality of speakers, and in this case, the audio output module may be arranged inside the main body, and sound emitted by covering at least a portion of the diaphragm of the audio output module may pass through a waveguide and be transmitted to the outside of the main body. The audio output module may include a plurality of audio output units, and the plurality of audio output units may be arranged symmetrically on the exterior of the main body, thereby radiating sound in all directions, that is, in a 360-degree omnidirectional manner.

[0163] The microphone (118) is a component for receiving a user's voice or other sounds and converting them into audio data. The microphone (118) can receive the user's voice in an activated state. For example, the microphone (118) can be formed integrally on the upper side, the front side, the side side, etc. of the electronic device (100). The microphone (118) can include various components such as a microphone for collecting the user's voice in analog form, an amplifier circuit for amplifying the collected user's voice, an A / D conversion circuit for sampling the amplified user's voice and converting it into a digital signal, and a filter circuit for removing noise components from the converted digital signal.

[0164] The power supply unit (119) can receive power from an external source and supply power to various components of the electronic device (100). The power supply unit (119) according to various embodiments of the present disclosure can receive power through various methods. In various embodiments, the power supply unit (119) can receive power using a connector (130) as illustrated in FIG. 1. The power supply unit (119) can receive power using a 220 V DC power cord. However, the present invention is not limited thereto, and the electronic device (100) can receive power using a USB power cord or a wireless charging method.

[0165] The power supply unit (119) can be supplied with power using an internal battery or an external battery. The power supply unit (119) according to various embodiments of the present disclosure can be supplied with power through the internal battery. For example, the power supply unit (119) can charge the power of the internal battery using at least one of a 220V DC power cord, a USB power cord, and a USB C-Type power cord, and can be supplied with power through the charged internal battery. The power supply unit (119) according to various embodiments of the present disclosure can be supplied with power through an external battery. For example, when the electronic device (100) is connected to the external battery through various wired communication methods such as a USB power cord, a USB C-Type power cord, and a socket home, the power supply unit (119) can be supplied with power through the external battery. That is, the power supply unit (119) can be supplied with power directly from the external battery, or can charge the internal battery through the external battery and be supplied with power from the charged internal battery.

[0166] The power supply unit (119) according to the present disclosure can receive power using at least one of the multiple power supply methods described above.

[0167] With respect to power consumption, the electronic device (100) may have a power consumption value (e.g., 43 W) or lower due to socket type and other standards. In this case, the electronic device (100) may vary its power consumption to reduce power consumption when using a battery. That is, the electronic device (100) may vary its power consumption based on the power supply method, power usage, etc.

[0168] The driving unit (120) can drive at least one hardware component included in the electronic device (100). The driving unit (120) can generate a physical force and transmit it to at least one hardware component included in the electronic device (100).

[0169] The driving unit (120) can generate driving power for movement of a hardware component included in the electronic device (100) (e.g., movement of the electronic device (100)) or rotation of the component (e.g., rotation of a projection lens).

[0170] The driving unit (120) can adjust the projection direction (or projection angle) of the projection unit (122). The driving unit (120) can move the position of the electronic device (100). The driving unit (120) can control a moving member to move the electronic device (100). For example, the driving unit (120) can control the moving member using a motor.

[0171] The sensor unit (121) may include at least one sensor. Specifically, the sensor unit (121) may include at least one of a tilt sensor for sensing the tilt of the electronic device (100) and an image sensor for capturing an image. The tilt sensor may be an acceleration sensor or a gyro sensor, and the image sensor may mean a camera or a depth camera. The tilt sensor may be described as a motion sensor. The sensor unit (121) may include various sensors in addition to the tilt sensor or the image sensor. For example, the sensor unit (121) may include an illuminance sensor and a distance sensor. The distance sensor may be a ToF (Time of Flight). The sensor unit (121) may include a lidar sensor.

[0172] The electronic device (100) can control a lighting function by linking with an external device. Specifically, the electronic device (100) can receive lighting information from the external device. The lighting information can include at least one of brightness information or color temperature information set in the external device. The external device can refer to a device connected to the same network as the electronic device (100) (e.g., an IoT device included in the same home / work network) or a device that is not in the same network as the electronic device (100) but can communicate with the electronic device (100) (e.g., a remote control server). For example, assume that an external lighting device (IoT device) included in the same network as the electronic device (100) is outputting red light at a brightness of 50. The external lighting device (IoT device) can directly or indirectly transmit lighting information (e.g., information indicating that it is outputting red light at a brightness of 50) to the electronic device (100). The electronic device (100) can control the output of a light source based on the lighting information received from the external lighting device. For example, if the lighting information received from the external lighting device includes information to output red light at a brightness of 50, the electronic device (100) can output red light at a brightness of 50.

[0173] The electronic device (100) can control a lighting function based on biometric information. Specifically, at least one processor (111) can obtain the user's biometric information. The biometric information can include at least one of the user's body temperature, heart rate, blood pressure, respiration, and electrocardiogram. The biometric information can include various types of information in addition to the information described above. For example, the electronic device (100) can include a sensor for measuring biometric information. The at least one processor (111) can obtain the user's biometric information through the sensor and control the output of a light source based on the obtained biometric information. As another example, the at least one processor (111) can receive the biometric information from an external device through an input / output interface (116). The external device can refer to the user's portable communication device (e.g., a smartphone or a wearable device). The at least one processor (111) can obtain the user's biometric information from the external device and control the output of the light source based on the obtained biometric information. According to an implementation example, the electronic device (100) can identify whether the user is sleeping, and if the user is identified as sleeping (or preparing to sleep), at least one processor (111) can control the output of the light source based on the user's biometric information.

[0174] An electronic device (100) according to various embodiments of the present disclosure can provide various smart functions.

[0175] Specifically, the electronic device (100) is connected to a portable terminal device for controlling the electronic device (100), and a screen output from the electronic device (100) can be controlled through user input input from the portable terminal device. As an example, the portable terminal device can be implemented as a smartphone including a touch display, and the electronic device (100) receives screen data provided by the portable terminal device from the portable terminal device and outputs it, and a screen output from the electronic device (100) can be controlled according to user input input from the portable terminal device.

[0176] The electronic device (100) can share content or music provided by the mobile terminal device by connecting to the mobile terminal device through various communication methods such as Miracast, Airplay, wireless DEX, and Remote PC.

[0177] In addition, the mobile terminal device and the electronic device (100) can be connected in various connection methods. In various embodiments, the mobile terminal device can search for the electronic device (100) to perform a wireless connection, or the electronic device (100) can search for the mobile terminal device to perform a wireless connection. In addition, the electronic device (100) can output content provided by the mobile terminal device.

[0178] In various embodiments, when a mobile terminal device is placed near an electronic device (100) while specific content or music is being output from the mobile terminal device, and a preset gesture is detected through the display of the mobile terminal device (e.g., motion tap view), the electronic device (100) can output the content or music being output from the mobile terminal device.

[0179] In various embodiments, when the mobile terminal device is outputting specific content or music and the mobile terminal device comes closer to the electronic device (100) to a preset distance or less (e.g., non-contact tap view) or the mobile terminal device comes into contact with the electronic device (100) twice at a short interval (e.g., contact tap view), the electronic device (100) can output the content or music being output by the mobile terminal device.

[0180] In the above-described embodiment, it has been described that the same screen as the screen provided by the mobile terminal device is provided by the electronic device (100), but the present disclosure is not limited thereto. That is, when a connection is established between the mobile terminal device and the electronic device (100), the mobile terminal device may output a first screen provided by the mobile terminal device, and the electronic device (100) may output a second screen provided by the mobile terminal device that is different from the first screen. For example, the first screen may be a screen provided by a first application installed on the mobile terminal device, and the second screen may be a screen provided by a second application installed on the mobile terminal device. For example, the first screen and the second screen may be different screens provided by a single application installed on the mobile terminal device. For example, the first screen may be a screen including a remote control-type UI for controlling the second screen.

[0181] An electronic device (100) according to the present disclosure can output a standby screen. For example, if the electronic device (100) is not connected to an external device or if no input is received from the external device for a preset period of time, the electronic device (100) can output a standby screen. Conditions for the electronic device (100) to output a standby screen are not limited to the examples described above, and the standby screen can be output under various conditions.

[0182] The electronic device (100) may output a standby screen in the form of a blue screen, but the present disclosure is not limited thereto. For example, the electronic device (100) may extract only the shape of a specific object from data received from an external device, acquire an amorphous object, and output a standby screen including the acquired amorphous object.

[0183] The electronic device (100) may further include a display (not shown).

[0184] The display (not shown) can be implemented as various types of displays such as an LCD (Liquid Crystal Display), an OLED (Organic Light Emitting Diodes) display, a PDP (Plasma Display Panel), etc. The display (not shown) may also include a driving circuit, a backlight unit, etc., which can be implemented as a type of a-si TFT (amorphous silicon thin film transistor), LTPS (low temperature poly silicon) TFT, OTFT (organic TFT), etc. The display (not shown) may be implemented as a touch screen combined with a touch sensor, a flexible display, a three-dimensional display (3D display, three-dimensional display), etc. According to various embodiments of the present disclosure, the display (not shown) may include not only a display panel that outputs an image, but also a bezel that houses the display panel. In particular, according to various embodiments of the present disclosure, the bezel may include a touch sensor (not shown) for detecting user interaction.

[0185] The electronic device (100) may further include a shutter unit (not shown).

[0186] The shutter portion (not shown) may include at least one of a shutter, a fixing member, a rail, or a body.

[0187] The shutter can block light output from the projection unit (112). The fixing member can fix the position of the shutter. The rail can be a path for moving the shutter and the fixing member. The body can be configured to include the shutter and the fixing member.

[0188] FIG. 3 is a block diagram for explaining a specific configuration of the electronic device (100) of FIG. 2, according to one embodiment.

[0189] FIG. 4 is a diagram for explaining an operation of obtaining distance information of a face area according to one embodiment.

[0190] Referring to FIG. 4, the electronic device (100) may include at least one of a photographic image acquisition module (410), a face area detection module (420), a face area size information acquisition module (430), an artificial intelligence model (440), and a face area distance information provision module (450).

[0191] The image capture module (410) can capture a captured image. The captured image may be an image including a user (20). For example, the captured image may include an RGB image.

[0192] For example, the electronic device (100) can acquire a captured image using a camera included in the electronic device (100).

[0193] For example, the electronic device (100) can receive a captured image from an external device.

[0194] The captured image acquisition module (410) can store the captured image. The captured image acquisition module (410) can transmit the captured image to the face region detection module (420).

[0195] The face region detection module (420) can receive a captured image from the face region detection module (420). The face region detection module (420) can acquire (or detect or identify) a face region from the captured image. The captured image can include the face of the user (20). The electronic device (100) can acquire a face region representing the face of the user (20) included in the captured image. The face region can be described as information representing the face region, information corresponding to the face region, and information related to the face region. The face region detection module (420) can acquire coordinate information of the face region.

[0196] When the user's (20) face area is identified in the captured image, the face area detection module (420) can obtain coordinate information of the face area. The coordinate information may include coordinate values ​​indicating the location where the face area is displayed. The coordinate information may be a two-dimensional value.

[0197] The face area detection module (420) can transmit coordinate information of the face area to the face area size information acquisition module (430).

[0198] The face area size information acquisition module (430) can receive coordinate information of the face area from the face area detection module (420). The face area size information acquisition module (430) can acquire size information of the face area based on the coordinate information of the face area.

[0199] The size information of the face region may include at least one of the horizontal length of the face region, the vertical length of the face region, and the width of the face region.

[0200] The size information of the face region may further include the diagonal length of the face region.

[0201] The face area size information acquisition module (430) can transmit the face area size information to the artificial intelligence model (440).

[0202] The artificial intelligence model (440) can receive face region size information from the face region size information acquisition module (430). The artificial intelligence model (440) can be described as a machine learning model or a deep learning model.

[0203] The artificial intelligence model (440) can acquire distance information of the face region by using the size information of the face region received from the size information acquisition module (430) of the face region as input data. The distance information of the face region can be output data of the artificial intelligence model (440). The electronic device (100) can acquire distance information of the face region as output data.

[0204] The artificial intelligence model (440) can transmit distance information of the face area to the distance information providing module (450) of the face area.

[0205] The distance information providing module (450) of the face region can obtain distance information of the face region from the artificial intelligence model (440). The distance information providing module (450) of the face region can perform various functions using the size information of the face region. The various functions can include at least one of a function of displaying distance information of the face region, a function of determining the resolution of a projection image using the distance information of the face region, a function of determining the position of a user (20), a function of maintaining a certain distance from the user (20), and a function of avoiding the user (20).

[0206] FIG. 5 is a diagram for explaining an operation of detecting a face area in an input image according to one embodiment.

[0207] Referring to the embodiment (510) of FIG. 5, the electronic device (100) can acquire a captured image. It is assumed that the captured image includes a user (20).

[0208] Referring to the embodiment (520) of FIG. 5, the electronic device (100) can acquire (or detect or identify) a face area (521) of a user (20) from a captured image. The electronic device (100) can acquire coordinate information of the face area (521). The coordinate information can include at least one coordinate value corresponding to the face area (521) representing the face of the user (20). For example, the coordinate information can include a two-dimensional coordinate value.

[0209] Referring to the embodiment (530) of FIG. 5, the electronic device (100) can obtain size information of the face area (521) using coordinate information of the face area (521). The size information can include at least one of horizontal length, vertical length, and width.

[0210] The electronic device (100) can obtain at least one of the horizontal length (531) of the face area (521) and the vertical length (532) of the face area (521).

[0211] For example, the electronic device (100) can obtain the horizontal length (531) of the face area (521).

[0212] For example, the electronic device (100) can obtain the vertical length (532) of the face area (521).

[0213] For example, the electronic device (100) can obtain the horizontal length (531) of the face area (521) and the vertical length (532) of the face area (521).

[0214] The electronic device (100) can obtain the area of ​​the face area (521).

[0215] The electronic device (100) can obtain size information of a face area (521) including at least one of horizontal length, vertical length, and width.

[0216] FIG. 6 is a drawing for explaining an operation of displaying distance information of a face area according to one embodiment.

[0217] The electronic device (100) can obtain distance information (e.g., 0.8 m) of the face area based on size information of the face area.

[0218] Referring to the embodiment (610) of FIG. 6, the electronic device (100) can output distance information of a face area to a location corresponding to the user (20). The electronic device (100) can output a UI (611) indicating the distance information of the face area. The electronic device (100) can output the UI (611) to a location corresponding to the user (20). The location corresponding to the user (20) can be a location that is a preset distance away in a preset direction based on the location of the user (20). The electronic device (100) can output the UI (611) based on the preset location and preset display settings (size, color, font, etc.).

[0219] Referring to the embodiment (620) of FIG. 6, the electronic device (100) can output a UI (621) indicating distance information of a face area and a UI (622) indicating a face area of ​​the user (20) at a location corresponding to the user (20). The electronic device (100) can output the UIs (621, 622) at a location corresponding to the user (20). The location corresponding to the user (20) can be a location that is a preset distance away in a preset direction based on the location of the user (20). The electronic device (100) can output the UIs (621, 622) based on the preset location and preset display settings (size, color, font, etc.).

[0220] FIG. 7 is a diagram for explaining an operation using an artificial intelligence model according to one embodiment.

[0221] Referring to FIG. 7, the electronic device (100) may include an artificial intelligence model (700). The artificial intelligence model (700) may correspond to the artificial intelligence model (440) of FIG. 4. Duplicate descriptions are omitted.

[0222] The artificial intelligence model (700) can receive input data (710). The input data (710) can receive (or acquire) size information of a face region. The input data (710) can include size information including at least one of the horizontal length of the face region, the vertical length of the face region, and the width of the face region.

[0223] The input data (710) may additionally include at least one of the diagonal length of the face region and the shape of the face region.

[0224] The artificial intelligence model (700) can receive input data (710) and obtain output data (720). The artificial intelligence model (700) can include learned weights. The output data (720) can include distance information of the facial area.

[0225] The artificial intelligence model (700) can receive size information (width, height, area) of a facial region as input data (710) and obtain distance information of the facial region as output data (720). The artificial intelligence model (700) can obtain output data (720) corresponding to the input data (710) by using pre-learned weights.

[0226] According to various embodiments, the input data may additionally include rotation axis information (yaw, pitch, roll) related to the head direction of the user (20). The electronic device (100) may additionally obtain output data by utilizing the rotation axis information (yaw, pitch, roll) related to the head direction of the user (20).

[0227] According to various embodiments, the input data may include a feature object (or landmark object) related to the face area of ​​the user (20). The feature object may include at least one of a human eye, nose, and mouth. The electronic device (100) may obtain output data using the feature object. An operation using the feature object may include an operation using the location of the feature object.

[0228] FIG. 8 is a diagram for explaining an operation of learning an artificial intelligence model according to one embodiment.

[0229] The electronic device (100) may include at least one of a first type of first learning data acquisition module (810), a second type of second learning data acquisition module (820), a face region detection module (830), a learning data alignment module (840), a face region attribute information acquisition module (850), a learning data set generation module (860), a learning module (870), and an artificial intelligence model storage module (880).

[0230] A first type of first learning data acquisition module (810) can acquire and store first learning data. The first learning data can include a first type of image. The first learning data can include an RGB image. The first learning data can include an RGB image capturing a user (20). The electronic device (100) can acquire the first learning data using an RGB camera.

[0231] The first type of first learning data acquisition module (810) can transmit the first learning data to at least one of the face region detection module (830) and the learning data alignment module (840).

[0232] A second type of second learning data acquisition module (820) can acquire and store second learning data. The second learning data can include a second type of image. The second learning data can include a depth image. The electronic device (100) can acquire the depth image using one of a depth camera, a 3D TOF (Time of Flight), and a Lidar.

[0233] The second learning data may include a depth image captured (or sensed) of the user (20). The second learning data may include depth map information. The depth map may include depth (or distance) values ​​corresponding to each image coordinate. For example, the depth map may include a first distance value corresponding to a first pixel and a second distance value corresponding to a second pixel.

[0234] The first learning data may be described as first learning image information, or first learning image. The second learning data may be described as second learning image information, or second learning image.

[0235] The second type of second learning data acquisition module (820) can transmit the second learning data to the learning data alignment module (840).

[0236] The first learning data and the second learning data may represent images (or data) captured of the same user (20). The first learning data may be an image captured of the user (20) by a first sensor (e.g., a camera). The second learning data may be an image captured of the user (20) by a second sensor (e.g., a depth camera). The first learning data and the second learning data may be images captured (or sensed) of the user (20) at the same point in time. The second learning data may be an image corresponding to the first learning data. The first learning data and the second learning data may include coordinate values.

[0237] According to various embodiments, the plurality of training data may include data acquired in a low-light environment. When using training data sensed in a low-light environment, training can be performed insensitive to changes in illumination. The plurality of training data may include training data captured from below the user's face in a direction looking at the user. When using training data captured from below in a direction looking at the user, training can be performed insensitive to changes in camera angle.

[0238] According to various embodiments, the plurality of training data may include data generated through data augmentation. Data augmentation operations may include functions such as left-right flipping, contrast adjustment, blur processing, and cropping.

[0239] The face region detection module (830) can receive first learning data from the first type of first learning data acquisition module (810). The face region detection module (830) can acquire coordinate information of the face region based on the first learning data. The face region detection module (830) can transmit the coordinate information of the face region to the face region attribute information acquisition module (850).

[0240] The learning data alignment module (840) can receive first learning data from the first type of first learning data acquisition module (810). The learning data alignment module (840) can receive second learning data from the second type of second learning data acquisition module (820). The learning data alignment module (840) can perform an alignment function using the first learning data and the second learning data. The learning data alignment module (840) can perform an alignment function to express the first learning data and the second learning data in the same coordinates. The electronic device (100) can perform the alignment function based on common features acquired from the first learning data and the second learning data. The common feature can be at least one of a reference line related to a face region and a reference point related to a face region.

[0241] The positions of the user (20) shown in the first learning data may differ from those shown in the second learning data. This is because the placement positions of the first sensor used to acquire the first learning data differ from the placement positions of the second sensor used to acquire the second learning data. It is difficult to implement the placement positions of the first and second sensors to be perfectly identical.

[0242] The electronic device (100) can perform an alignment function so that the coordinate information of the user (20) included in the first learning data and the coordinate information of the user (20) included in the second learning data are mapped. A description related to the alignment function is described in FIGS. 9 and 10. The alignment function can be described as a mapping function.

[0243] The learning data alignment module (840) can perform an alignment function to obtain aligned second learning data. The aligned second learning data may be an image having coordinate information mapped to the coordinate information of the first learning data. The coordinate values ​​of the user (20) included in the first learning data may match the coordinate values ​​of the user (20) included in the second learning data. The learning data alignment module (840) can transmit the aligned second learning data to the face region attribute information acquisition module (850). The aligned second learning data may be described as third learning data in that the image has been changed.

[0244] The face region attribute information acquisition module (850) can receive coordinate information of the face region from the face region detection module (830). The face region attribute information acquisition module (850) can receive aligned second learning data from the learning data alignment module (840). The face region attribute information acquisition module (850) can acquire attribute information of the face region based on the coordinate information of the face region and the aligned second learning data.

[0245] The attribute information of the face region may include at least one of the size information of the face region and the distance information of the face region. The size information of the face region may include at least one of the horizontal length of the face region, the vertical length of the face region, and the area of ​​the face region. The distance information of the face region may include the distance from the capturing position (or sensing position) to the reference point of the face region. The reference point may be a point representing the face region. The reference point may be described as a representative point. The reference point may be acquired based on the median or average value of the face region.

[0246] The facial region attribute information acquisition module (850) can transmit the facial region attribute information to the learning data set generation module (860).

[0247] The learning data set generation module (860) can receive attribute information of the face region from the face region attribute information acquisition module (850). The learning data set generation module (860) can generate (or acquire) a first learning data set that maps attribute information of the face region. The electronic device (100) can extract (or acquire) attribute information (horizontal length, vertical length, width, distance between the capturing location and the user, etc.) indicated by the face region of the user (20) using the first learning data.

[0248] The learning data set generation module (860) can transmit the first learning data set to the learning module (870).

[0249] The learning module (870) can receive a first learning data set from the learning data set generation module (860). The learning module (870) can perform learning on an artificial intelligence model based on the first learning data set and previously stored learning data (e.g., a second learning data set, a third learning data set, etc.). The learning module (870) can learn to obtain output data corresponding to input data using at least one weight (or parameter). The input data can be size information of a face region. The output data can be distance information of a face region. The learning module (870) can learn at least one weight so that distance information of the face region is output based on the size information of the face region and at least one weight.

[0250] The learning module (870) can obtain the final learned weights based on repeated multiple learning operations. The learning module (870) can transmit the learned weights to the artificial intelligence model storage module (880).

[0251] The artificial intelligence model storage module (880) can receive learned weights from the learning module (870). The artificial intelligence model storage module (880) can store an artificial intelligence model including the learned weights. The artificial intelligence model storage module (880) can transmit the artificial intelligence model according to a preset event.

[0252] FIG. 9 is a diagram for explaining an image alignment operation according to one embodiment.

[0253] Example 900-1 of FIG. 9 illustrates first learning data (910) and second learning data (920) before performing an alignment function. The first learning data (910) may include an RGB image. The second learning data (920) may include a depth map.

[0254] The electronic device (100) can identify a face region from the first learning data (911). The electronic device (100) can obtain (or identify) at least one of a first baseline (901) or a second baseline (902) related to the face region.

[0255] The electronic device (100) can determine the uppermost horizontal axis of the face area in the first learning data (910) as the first reference line (901).

[0256] The electronic device (100) can determine the uppermost horizontal axis of the face area in the second learning data (920) as the second reference line (902).

[0257] The electronic device (100) can check whether the second baseline (902) obtained from the second learning data (920) matches the first baseline (901) obtained from the first learning data (910).

[0258] If the second reference line (902) does not match the first reference line (901), the electronic device (100) can perform an alignment function on the second learning data (920). The electronic device (100) can determine whether the coordinate values ​​indicated by the second reference line (902) and the coordinate values ​​indicated by the first reference line (901) match.

[0259] If the second reference line (902) does not match the first reference line (901), the electronic device (100) can obtain a difference value between the coordinate value indicated by the second reference line (902) and the coordinate value indicated by the first reference line (901). The electronic device (100) can correct the coordinate value of the second learning data (920) based on the difference value.

[0260] For example, if the difference between the coordinate values ​​indicated by the second reference line (902) and the coordinate values ​​indicated by the first reference line (901) is 3 cm, the electronic device (100) can reduce all y-axis coordinate values ​​of the second learning data (920) by 3 cm.

[0261] Example (900-2) shows first learning data (910) and sorted second learning data (925) after performing a sorting function (or correction function).

[0262] The electronic device (100) can obtain aligned second learning data (925) by correcting the second learning data (920). The aligned second learning data (925) can be data corrected by the difference between the coordinate values ​​indicated by the second reference line (902) and the coordinate values ​​indicated by the first reference line (901) in the coordinate values ​​of the second learning data (920).

[0263] In the embodiment (900-2), the electronic device (100) can determine the uppermost horizontal axis of the face area in the aligned second learning data (925) as the second reference line (902).

[0264] In the embodiment (900-2), since the alignment function has been performed, the first reference line (901) and the second reference line (902) may be aligned. The first learning data (910) and the aligned second learning data (925) may be mapped data. The location of the face area included in the first learning data (910) and the location of the face area included in the aligned second learning data (925) may be expressed with the same coordinate values.

[0265] According to various embodiments, the electronic device (100) may perform an alignment function using the lowermost horizontal axis of the face area.

[0266] The electronic device (100) can determine the lowermost horizontal axis of the face area in the first learning data (910) as the third reference line (903).

[0267] The electronic device (100) can determine the lowermost horizontal axis of the face area in the second learning data (920) as the fourth reference line (904).

[0268] The electronic device (100) can check whether the fourth baseline (904) obtained from the second learning data (920) matches the third baseline (903) obtained from the first learning data (910).

[0269] If the fourth baseline (904) does not match the third baseline (903), the electronic device (100) can perform an alignment function on the second learning data (920). The electronic device (100) can determine whether the coordinate values ​​indicated by the fourth baseline (904) and the coordinate values ​​indicated by the third baseline (903) match.

[0270] If the fourth reference line (904) does not match the third reference line (903), the electronic device (100) can obtain a difference value between the coordinate value indicated by the fourth reference line (904) and the coordinate value indicated by the third reference line (903). The electronic device (100) can correct the coordinate value of the second learning data (920) based on the difference value.

[0271] For example, if the difference between the coordinate values ​​indicated by the fourth reference line (904) and the coordinate values ​​indicated by the third reference line (903) is 3 cm, the electronic device (100) can reduce all y-axis coordinate values ​​of the second learning data (920) by 3 cm.

[0272] Example (900-2) shows first learning data (910) and sorted second learning data (925) after performing a sorting function (or correction function).

[0273] The electronic device (100) can obtain aligned second learning data (925) by correcting the second learning data (920). The aligned second learning data (925) can be data corrected by the difference between the coordinate values ​​indicated by the fourth reference line (904) and the coordinate values ​​indicated by the third reference line (903) in the coordinate values ​​of the second learning data (920).

[0274] In the embodiment (900-2), the electronic device (100) can determine the lowermost horizontal axis of the face area in the aligned second learning data (925) as the fourth reference line (904).

[0275] In the embodiment (900-2), since the alignment function has been performed, the third reference line (903) and the fourth reference line (904) may be aligned. The first learning data (910) and the aligned second learning data (925) may be mapped data. The location of the face area included in the first learning data (910) and the location of the face area included in the aligned second learning data (925) may be expressed with the same coordinate values.

[0276] According to various embodiments, the electronic device (100) may perform an alignment function using both the uppermost horizontal axis and the lowermost horizontal axis of the face area. The electronic device (100) may compare a first reference line (901), a second reference line (902), and a third reference line (903) and a fourth reference line (904).

[0277] FIG. 10 is a diagram for explaining an image alignment operation according to one embodiment.

[0278] Example 1000-1 of FIG. 10 illustrates first learning data (1010) and second learning data (1020) before performing an alignment function. The first learning data (1010) may include an RGB image. The second learning data (1020) may include a depth map.

[0279] The electronic device (100) can identify a face region from the first learning data (1011). The electronic device (100) can obtain (or identify) at least one of a first baseline (1001) or a second baseline (1002) related to the face region.

[0280] The electronic device (100) can determine the leftmost vertical axis of the face area in the first learning data (1010) as the first reference line (1001).

[0281] The electronic device (100) can determine the leftmost vertical axis of the face area in the second learning data (1020) as the second reference line (1002).

[0282] The electronic device (100) can check whether the second baseline (1002) obtained from the second learning data (1020) matches the first baseline (1001) obtained from the first learning data (1010).

[0283] If the second baseline (1002) does not match the first baseline (1001), the electronic device (100) can perform an alignment function on the second learning data (1020). The electronic device (100) can determine whether the coordinate values ​​indicated by the second baseline (1002) and the coordinate values ​​indicated by the first baseline (1001) match.

[0284] If the second reference line (1002) does not match the first reference line (1001), the electronic device (100) can obtain a difference value between the coordinate value indicated by the second reference line (1002) and the coordinate value indicated by the first reference line (1001). The electronic device (100) can correct the coordinate value of the second learning data (1020) based on the difference value.

[0285] For example, if the difference between the coordinate values ​​indicated by the second reference line (1002) and the coordinate values ​​indicated by the first reference line (1001) is 3 cm, the electronic device (100) can reduce all y-axis coordinate values ​​of the second learning data (1020) by 3 cm.

[0286] Example (1000-2) shows first learning data (1010) and sorted second learning data (1025) after performing a sorting function (or correction function).

[0287] The electronic device (100) can obtain aligned second learning data (1025) by correcting the second learning data (1020). The aligned second learning data (1025) can be data corrected by the difference between the coordinate values ​​indicated by the second reference line (1002) and the coordinate values ​​indicated by the first reference line (1001) in the coordinate values ​​of the second learning data (1020).

[0288] In the embodiment (1000-2), the electronic device (100) can determine the leftmost vertical axis of the face area in the aligned second learning data (1025) as the second reference line (1002).

[0289] In Example (1000-2), since the alignment function has been performed, the first reference line (1001) and the second reference line (1002) may be aligned. The first learning data (1010) and the aligned second learning data (1025) may be mapped data. The location of the face region included in the first learning data (1010) and the location of the face region included in the aligned second learning data (1025) may be expressed with the same coordinate value.

[0290] According to various embodiments, the electronic device (100) may perform an alignment function using the rightmost vertical axis of the face area.

[0291] The electronic device (100) can determine the rightmost vertical axis of the face area in the first learning data (1010) as the third reference line (1003).

[0292] The electronic device (100) can determine the rightmost vertical axis of the face area in the second learning data (1020) as the fourth reference line (1004).

[0293] The electronic device (100) can check whether the fourth baseline (1004) obtained from the second learning data (1020) matches the third baseline (1003) obtained from the first learning data (1010).

[0294] If the fourth baseline (1004) does not match the third baseline (1003), the electronic device (100) can perform an alignment function on the second learning data (1020). The electronic device (100) can determine whether the coordinate values ​​indicated by the fourth baseline (1004) and the coordinate values ​​indicated by the third baseline (1003) match.

[0295] If the fourth reference line (1004) does not match the third reference line (1003), the electronic device (100) can obtain a difference value between the coordinate value indicated by the fourth reference line (1004) and the coordinate value indicated by the third reference line (1003). The electronic device (100) can correct the coordinate value of the second learning data (1020) based on the difference value.

[0296] For example, if the difference between the coordinate values ​​indicated by the fourth reference line (1004) and the coordinate values ​​indicated by the third reference line (1003) is 3 cm, the electronic device (100) can reduce all y-axis coordinate values ​​of the second learning data (1020) by 3 cm.

[0297] Example (1000-2) shows first learning data (1010) and sorted second learning data (1025) after performing a sorting function (or correction function).

[0298] The electronic device (100) can obtain aligned second learning data (1025) by correcting the second learning data (1020). The aligned second learning data (1025) can be data corrected by the difference between the coordinate values ​​indicated by the fourth reference line (1004) and the coordinate values ​​indicated by the third reference line (1003) in the coordinate values ​​of the second learning data (1020).

[0299] In the embodiment (1000-2), the electronic device (100) can determine the rightmost vertical axis of the face area in the aligned second learning data (1025) as the fourth reference line (1004).

[0300] In Example (1000-2), since the alignment function has been performed, the third baseline (1003) and the fourth baseline (1004) may be aligned. The first learning data (1010) and the aligned second learning data (1025) may be mapped data. The location of the face region included in the first learning data (1010) and the location of the face region included in the aligned second learning data (1025) may be expressed with the same coordinate values.

[0301] According to various embodiments, the electronic device (100) may perform an alignment function using both the leftmost vertical axis and the rightmost vertical axis of the face area. The electronic device (100) may compare a first reference line (1001), a second reference line (1002), and a third reference line (1003) and a fourth reference line (1004).

[0302] The ordinal numbers of the reference lines in FIGS. 9 and 10 can be changed. The x-axis and y-axis shown in FIGS. 9 and 10 can represent distances.

[0303] FIG. 11 is a diagram for explaining an operation of extracting a distance value related to a face area according to one embodiment.

[0304] Referring to the embodiment (1101) of FIG. 11, the electronic device (100) can obtain first learning data (1110) and aligned second learning data (1125). The electronic device (100) can obtain (or extract) mapped coordinate values ​​using the first learning data (1110) and aligned second learning data (1125).

[0305] The electronic device (100) can obtain coordinate information of a face area (1111) from the first learning data (1110). Based on the coordinate information of the face area (1111), the electronic device (100) can obtain coordinate information of a center point (1112) of the face area (1111).

[0306] The electronic device (100) can obtain coordinate information of the center point (1127) of the face area (1126) in the second learning data (1125) aligned based on the coordinate information of the center point (1112) of the face area (1111) included in the first learning data (1110).

[0307] For example, the coordinate information of the center point (1112) of the face area (1111) and the coordinate information of the center point (1127) of the face area (1126) may be the same. The aligned second learning data (1125) may include a depth map. The electronic device (100) may obtain depth information corresponding to the center point (1127) based on the coordinate information of the center point (1127). The depth information may include a distance value corresponding to the center point (1127). The electronic device (100) may obtain the distance value of the center point (1112) from the first learning data (1110) based on the distance value corresponding to the center point (1127).

[0308] Instead of a center point, a preset point can be used. Preset points can be designated as representative points, representative points, or reference points.

[0309] FIG. 12 is a diagram for explaining an operation of obtaining distance information of a face area according to one embodiment.

[0310] Referring to FIG. 12, the electronic device (100) can acquire a captured image (S1210). The electronic device (100) can determine whether a face area is detected in the captured image (S1220).

[0311] If the face area is not detected (S1220-N), the electronic device (100) may repeatedly perform steps S1210 and S1220.

[0312] When a face region is detected (S1220-Y), the electronic device (100) can obtain coordinate information of the face region (S1220). The electronic device (100) can obtain size information of the face region based on the coordinate information of the face region (S1230). The size information of the face region can include at least one of horizontal length, vertical length, and width.

[0313] The electronic device (100) can obtain distance information of the face area based on size information of the face area (S1250). The distance information of the face area can include the distance from the capturing point (based on the captured image) to the center point of the face area.

[0314] The electronic device (100) can provide distance information of a face area (S1260). The operation of providing distance information may include an operation of performing a function that utilizes distance information of the face area for various functions.

[0315] The various functions may include at least one of a function for displaying distance information of a face area, a function for determining the resolution of a projection image using distance information of a face area, a function for determining the position of a user (20), a function for maintaining a certain distance from a user (20), and a function for avoiding a user (20).

[0316] FIG. 13 is a drawing for explaining an operation of outputting a projection image using distance information of a face area, according to one embodiment.

[0317] Steps S1310 to S1350 of FIG. 13 may correspond to steps S1210 to S1250 of FIG. 12. Duplicate description is omitted.

[0318] The electronic device (100) can obtain a user input for content output (S1305). The electronic device (100) can obtain a captured image based on the user input (S1310). The electronic device (100) can identify a facial area of ​​the user (20) included in the captured image. The electronic device (100) can obtain distance information of the facial area (S1350). The distance information of the facial area can be described as first distance information.

[0319] The electronic device (100) can obtain projection resolution information based on the first distance information (S1370). The projection resolution information may be information indicating the projection size when the electronic device (100) outputs content to the projection surface (10). Under the same projection distance, the larger the projection size, the larger the projection image output to the projection surface (10).

[0320] The electronic device (100) can acquire content information (S1375). The content information may include at least one of a content image and content resolution information. The content image may be a concept including multiple frames. The content image may be described as a content image group, content image data, etc. The content resolution information may indicate the size of the content. The larger the content size, the more pixels may be used to express the content image.

[0321] The electronic device (100) can obtain a scaling factor based on content resolution information and projection resolution information (S1380). If the content resolution information and the projection resolution information are different, a projection image of the desired quality may not be output. The electronic device (100) can output a projection image by performing a scaling function. The electronic device (100) can correct the size of the content image included in the content and output it.

[0322] The electronic device (100) can obtain a projection image by scaling a content image based on a scaling factor (S1385). The electronic device (100) can change the size of the content image based on the scaling factor. The changed image can be described as a projection image.

[0323] The electronic device (100) can output a projection image (S1390). The electronic device (100) can output the projection image to the projection surface (10) after performing a scaling function.

[0324] FIG. 14 is a drawing for explaining an operation of outputting a projection image using the distance to a projection surface according to one embodiment.

[0325] Steps S1405, S1410 to S1450, S1475, S1480, S1485, and S1490 of FIG. 14 may correspond to steps S1305, S1310 to S1350, S1375, S1380, S1385, and S1390 of FIG. 13. Duplicate explanations are omitted.

[0326] The electronic device (100) can obtain distance information to the projection surface (10) (S1471). The distance information to the projection surface (10) can be described as second distance information. The electronic device (100) can obtain projection resolution information based on the first distance information and the second distance information (S1472). The electronic device (100) can determine projection resolution information by using both the first distance information indicating the distance between the electronic device (100) and the user (20) and the second distance information indicating the distance between the electronic device (100) and the projection surface (10).

[0327] The distance between the electronic device (100) and the projection surface (10) may be described as a projection distance. The electronic device (100) may store a preset projection ratio range. The projection ratio that the electronic device (100) can provide may be limited to a preset range. The projection ratio may be calculated as the projection distance / the size of the projection image. The size of the projection image may include the diagonal length of the projection image. The electronic device (100) may determine projection resolution information by considering the preset projection ratio range. The projection resolution information may include the size of the projection image.

[0328] FIG. 15 is a diagram for explaining an operation of identifying multiple face regions according to one embodiment.

[0329] Referring to an embodiment (1510) of FIG. 15, the electronic device (100) can identify multiple users from a captured image. The electronic device (100) can detect a face area (1511, 1512) for each of the multiple users. The electronic device (100) can obtain size information (x1, y1, x1*y1) of the first face area (1511). The electronic device (100) can obtain size information (x2, y2, x2*y2) of the second face area (1512).

[0330] Referring to the embodiment (1520) of FIG. 15, the electronic device (100) can provide distance information for each face area included in the captured image. The electronic device (100) can obtain distance information (d1) of the first face area (1511). The electronic device (100) can obtain distance information (d2) of the second face area (1512).

[0331] According to various embodiments, the electronic device (100) can output a captured image including distance information.

[0332] FIG. 16 is a diagram for explaining an operation of obtaining distance information of each of a plurality of face regions according to one embodiment.

[0333] Steps S1610, S1620, S1630, S1640, and S1660 of FIG. 16 may correspond to steps S1210, S1220, S1230, S1240, and S1260 of FIG. 12. Duplicate explanations are omitted.

[0334] After detecting the face area, the electronic device (100) can obtain first coordinate information of the first face area and second coordinate information of the second face area (S1630). The electronic device (100) can obtain first size information of the first face area and second size information of the second face area (S1640).

[0335] The electronic device (100) can obtain a first distance value of a first face area and a second distance value of a second face area (S1651). The electronic device (100) can obtain a representative distance value based on the first distance value and the second distance value (S1652). For example, the representative distance value may be one of an average value, a minimum value, and a maximum value.

[0336] For example, the representative distance value may be a distance value representing the facial area of ​​the representative user. The electronic device (100) may identify the representative user among multiple users and determine a distance value corresponding to the facial area of ​​the representative user as the representative distance value.

[0337] The electronic device (100) can obtain distance information of a face area including a representative distance value (S1653). The electronic device (100) can provide distance information of the face area (S1660).

[0338] FIG. 17 is a drawing for explaining an operation of controlling the distance between a user (20) and an electronic device (100), according to one embodiment.

[0339] Referring to an embodiment (1700) of FIG. 17, the electronic device (100) can output a projection image on the projection surface (10). The electronic device (100) can be controlled to be positioned within a preset range or a threshold distance (r) based on the user (20). The electronic device (100) can identify the location of the user (20). The electronic device (100) can determine the location of the user (20) based on distance information of the face area. The electronic device (100) can move to be positioned within the threshold distance (r) based on the location of the user (20).

[0340] If the distance between the location of the user (20) and the location of the electronic device (100) is greater than or equal to a threshold distance (r), the electronic device (100) can drive (or move) so that the electronic device (100) exists within the threshold distance (r) from the location of the user (20).

[0341] FIG. 18 is a drawing for explaining an operation of avoiding a user (20) according to one embodiment.

[0342] Referring to the embodiment (1800) of FIG. 18, the electronic device (100) can obtain the position of the projection surface (10). The electronic device (100) can obtain the position of the user (20).

[0343] The electronic device (100) can move between the position of the projection surface (10) and the position of the user (20). This is to prevent the user (20) from being obstructed when outputting the projection image. If the user (20) is present between the electronic device (100) and the projection surface (10), the projection image may be obscured by the user (20). The electronic device (100) can move so as to be positioned between the projection surface (10) and the user (20).

[0344] The electronic device (100) can drive to avoid the user (20) based on the location of the user (20). The electronic device (100) can identify the location of the user (20) using distance information of the facial area.

[0345] FIG. 19 is a drawing for explaining a guide screen related to a user (20) and an electronic device (100) according to one embodiment.

[0346] Referring to FIG. 19, the electronic device (100) can output a projection image (1900). The projection image (1900) can include information related to distance information of a face area. The projection image (1900) can include at least one of a captured image (1910), a UI (1920) indicating distance information between the electronic device (100) and the user (20), a UI (1930) indicating distance information between the electronic device (100) and the projection surface (10), a UI (1940) indicating resolution information (or size information) of the projection image, and a UI (1950) for adjusting the resolution.

[0347] The captured image (1910) may include an image captured by the electronic device (100). The captured image (1910) may include at least one of a GUI representing a user's face area and a GUI representing distance information.

[0348] FIG. 20 is a drawing for explaining a control operation of an electronic device (100) according to one embodiment.

[0349] Referring to FIG. 20, a control method of an electronic device (100) for acquiring sensing data and storing a content image includes a step (S2010) of acquiring size information of a face area when a user's face area is detected based on the sensing data, a step (S2020) of acquiring distance information of a face area indicating a distance between a position of an electronic device (100) that acquired the sensing data and a position of a user based on the size information of the face area, and a step (S2030) of outputting a content image based on the distance information.

[0350] The step of obtaining size information (S2010) may obtain size information including at least one of the horizontal length of the face region, the vertical length of the face region, and the width of the face region.

[0351] The step of obtaining size information (S2010) may include obtaining coordinate information of the face area when the user's face area is detected based on sensing data, and obtaining size information based on the coordinate information.

[0352] The electronic device (100) stores content resolution information, and the control method further includes a step of obtaining projection resolution information indicating the size of an image output through the electronic device (100) based on distance information, and the step of outputting the content image (S2030) can output the content image based on the projection resolution information.

[0353] The distance information is first distance information, and the step of obtaining projection resolution information may obtain second distance information indicating the distance between the position of the electronic device (100) and the projection surface, and may obtain projection resolution information based on the first distance information and the second distance information.

[0354] The electronic device (100) stores a projection ratio range, and the step of obtaining projection resolution information may obtain a resolution range based on the projection ratio range and second distance information, and may obtain projection resolution information within the resolution range based on the first distance information.

[0355] The step of outputting a content image (S2030) may include obtaining a scaling factor based on content resolution information and projection resolution information, scaling the content image based on the scaling factor to obtain a projection image, and outputting the projection image.

[0356] The step of obtaining size information (S2010) obtains first size information of the first face area and second size information of the second face area when a first face area and a second face area are detected based on sensing data, the step of obtaining distance information (S2030) obtains a first distance value of the first face area based on the first size information of the first face area, obtains a second distance value of the second face area based on the second size information of the second face area, obtains a representative distance value based on the first distance value and the second distance value, obtains distance information including the representative distance value, and the step of outputting a content image (S2030) can output a content image based on the distance information.

[0357] The electronic device (100) stores an artificial intelligence model including at least one learned weight, and the step of obtaining distance information (S2030) can input size information into the artificial intelligence model to obtain distance information as output data.

[0358] At least one weight can be learned based on first training data including an RGB image and second training data including a depth image.

[0359] The methods according to the various embodiments of the present disclosure described above can be implemented in the form of applications that can be installed on existing electronic devices.

[0360] The methods according to the various embodiments of the present disclosure described above can be implemented only with a software upgrade or a hardware upgrade for an existing electronic device.

[0361] The various embodiments of the present disclosure described above may also be performed through an embedded server provided in an electronic device, or an external server of at least one of the electronic device and the display device.

[0362] According to an example embodiment of the present disclosure, the various embodiments described above may be implemented as software including instructions stored in a machine-readable storage medium that can be read by a machine (e.g., a computer). The device may include an electronic device according to the disclosed embodiments, which is a device that can call instructions stored in the storage medium and operate according to the called instructions. When the instructions are executed by a processor, the processor may perform a function corresponding to the instructions directly or under the control of the processor using other components. The instructions may include code generated or executed by a compiler or interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. 'Non-transitory' means that the storage medium does not contain signals and is tangible, but does not distinguish between whether data is stored semi-permanently or temporarily in the storage medium.

[0363] According to one embodiment of the present disclosure, the method according to the various embodiments described above may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)) or online through an application store (e.g., Play Store™). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated in a storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0364] Each of the components (e.g., modules or programs) according to the various embodiments described above may be composed of a single or multiple entities, and some of the sub-components described above may be omitted, or other sub-components may be further included in various embodiments. Alternatively or additionally, some components (e.g., modules or programs) may be integrated into a single entity, which may perform the same or similar functions as those performed by each of the respective components prior to integration. Operations performed by modules, programs or other components according to various embodiments may be executed sequentially, in parallel, iteratively or heuristically, or at least some operations may be executed in a different order, omitted, or other operations may be added.

[0365] Although the preferred embodiments of the present disclosure have been illustrated and described above, the present disclosure is not limited to the specific embodiments described above, and various modifications may be made by a person having ordinary skill in the art to which the present disclosure pertains without departing from the gist of the present disclosure as claimed in the claims, and such modifications should not be understood individually from the technical idea of ​​the present disclosure.

Claims

1. In electronic devices, A sensor section that acquires sensing data; memory; Projection section; and comprising at least one processor; At least one processor of the above, Obtain size information of the user's face area identified based on the above sensing data, Obtain distance information of the face area corresponding to the distance between the location of the electronic device and the location of the user based on the size information of the face area, An electronic device that controls the projection unit so that a content image is output based on distance information of the face area.

2. In paragraph 1, At least one processor of the above, An electronic device that obtains size information including at least one of a horizontal length of the face region, a vertical length of the face region, and an area of ​​the face region.

3. In paragraph 1, At least one processor of the above, When the user's face area is identified based on the above sensing data, coordinate information of the face area is obtained, An electronic device that obtains the size information based on the above coordinate information.

4. In paragraph 1, At least one processor of the above, Based on the above distance information, projection resolution information indicating the size of the image output through the projection unit is obtained, An electronic device that controls the projection unit so that the content image is output based on the projection resolution information.

5. In paragraph 4, The above distance information is the first distance information, At least one processor of the above, Obtain second distance information indicating the distance between the position of the electronic device and the projection surface, An electronic device that obtains projection resolution information based on the first distance information and the second distance information.

6. In paragraph 5, The above memory is, Save the projection ratio range, At least one processor of the above, Obtaining a resolution range based on the above projection ratio range and the second distance information, An electronic device that obtains projection resolution information within the resolution range based on the first distance information.

7. In paragraph 4, At least one processor of the above, Obtain a scaling factor based on the content resolution information and the projection resolution information, Obtaining a projection image by scaling the content image based on the above scaling factor, An electronic device that controls the projection unit to output the projection image.

8. In paragraph 1, At least one processor of the above, When the first face region and the second face region are identified based on the sensing data, first size information of the first face region and second size information of the second face region are obtained, Obtaining a first distance value of the first face area based on the first size information of the first face area, Obtaining a second distance value of the second face area based on the second size information of the second face area, Obtain a representative distance value based on the first distance value and the second distance value, Obtaining the distance information including the representative distance value, An electronic device that controls the projection unit to output the content image based on the distance information.

9. In paragraph 1, The above memory is, Save an artificial intelligence model that includes at least one learned weight, At least one processor of the above, An electronic device that inputs the above size information into the above artificial intelligence model and obtains the above distance information as output data.

10. In paragraph 9, At least one of the above weights is, An electronic device trained based on first training data including an RGB image and second training data including a depth image.

11. A method for controlling an electronic device for acquiring sensing data, A step of obtaining size information of a face area of ​​an identified user based on the above sensing data; A step of obtaining distance information of the face area corresponding to the distance between the location of the electronic device and the location of the user based on the size information of the face area; and A control method, comprising: a step of outputting a content image based on distance information of the above face area.

12. In paragraph 11, The step of obtaining the above size information is: A control method for obtaining size information including at least one of a horizontal length of the face region, a vertical length of the face region, and an area of ​​the face region.

13. In paragraph 11, The step of obtaining the above size information is: When the user's face area is identified based on the above sensing data, coordinate information of the face area is obtained, A control method for obtaining the size information based on the above coordinate information.

14. In paragraph 11, The above control method is, A step of obtaining projection resolution information indicating the size of an image output through the electronic device based on the distance information is further included; The step of outputting the above content image is: A control method for outputting the content image based on the projection resolution information.

15. In paragraph 14, The above distance information is the first distance information, The step of obtaining the above projection resolution information is: Obtain second distance information corresponding to the distance between the position of the electronic device and the projection surface, A control method for obtaining projection resolution information based on the first distance information and the second distance information.

Images