Electronic device for providing screen in posture optimized for user and control method thereof

Abstract

An electronic device includes a camera, a driving unit, a display, and at least one processor operably connected to the camera, the driving unit, and the display, controlling the camera such that an image is captured, recognizing a gesture of a user included in the captured image, and displaying the gesture of the user. And a controller configured to identify at least one parameter from a plurality of parameters related to the user based on the user's posture, control the driving unit based on the at least one parameter so as to change a position of the electronic device, and change a posture of the display.

Description

Technical Field

[0001] The present disclosure relates to an electronic device and a control method thereof, and more particularly, to an electronic device for providing a screen in a posture optimized for a user and a control method thereof. Background Art

[0002] Due to the progress of electronic technology, electronic devices providing various functions have been developed. As recent electronic devices have smaller sizes, an environment is being formed in which content can be consumed in various places in addition to the living room.

[0003] For example, a user may want to use the display in a comfortable position on a sofa, bed, etc. without holding the display with his or her hands. Figure 1a As shown, the smartphone / touchpad stand can manually adjust the display to a desired height or angle.

[0004] However, if Figure 1b As shown, in the case of using the stand, the user is required to directly move the stand for each space and adjust the angle of the stand whenever changing his or her posture, and the size of the display that can be mounted on the stand is limited. Summary of the invention

[0005] According to one aspect of the present disclosure, an electronic device includes a camera; a driver; a display; and at least one processor, wherein the at least one processor is operably connected to the camera, the driver, and the display, and is configured to: control the camera to capture an image, recognize a user posture included in the captured image, recognize at least one parameter among multiple parameters related to the user based on the user posture, and control the driver based on the at least one parameter to change the position of the electronic device and change the posture of the display.

[0006] The electronic device may also include a communication interface; and a microphone, wherein the at least one processor may also be configured to: control the drive to move the electronic device toward the user, and control the camera to capture multiple images at predetermined time intervals based on a user command received from another electronic device through the communication interface or the user command received through the microphone.

[0007] The at least one processor may be further configured to recognize the user gesture and a distance from the electronic device to the user based on the plurality of images, and stop operation of the driver based on the distance from the electronic device to the user corresponding to the user gesture.

[0008] The at least one processor may also be configured to: identify the multiple parameters related to the user based on the captured image, identify the user posture based on the multiple parameters, change the posture of the display based on at least one first parameter among the multiple parameters based on the user posture being a first posture, and change the posture of the display based on at least one second parameter among the multiple parameters based on the user posture being a second posture.

[0009] The at least one processor may be further configured to control the display to display content and change a posture of the display based on a type of content and the user posture.

[0010] The at least one processor may be further configured to, based on the user posture changing and a predetermined time elapsed, change a posture of the display based on the changed user posture.

[0011] The at least one processor may also be configured to: control the camera to capture multiple images at predetermined time intervals, obtain a predicted user posture at a predetermined time point from a current time point based on the multiple images, and change the posture of the display based on the user posture and the predicted user posture.

[0012] The electronic device may further include a memory configured to store a first neural network model, wherein the at least one processor may further be configured to recognize the user gesture by inputting the captured image into the first neural network model.

[0013] The electronic device may further include a memory configured to store a second neural network model, wherein the at least one processor may further be configured to obtain data about the posture of the display by inputting the multiple parameters into the second neural network model, and change the posture of the display based on the data about the posture of the display.

[0014] The electronic device may further include a user interface, wherein the at least one processor may further be configured to: based on a user command for changing the posture of the display received through the user interface, change the posture of the display based on the user command, and update the second neural network model based on the multiple parameters and the changed posture of the display.

[0015] The electronic device may further include a microphone, wherein the at least one processor may further be configured to: control the camera to capture multiple images at predetermined time intervals, and change the posture of the display based on user gestures recognized from the multiple images or based on user voice received through the microphone for changing the posture of the display.

[0016] The camera may be arranged at one side of the display, and wherein the at least one processor may be further configured to change a posture of the display so that the user or a predetermined user body part is included in a predetermined area in the captured image.

[0017] The electronic device may further include a display gesture driver, wherein the at least one processor may be further configured to: control the display gesture driver to change a gesture of the display.

[0018] The at least one processor may be further configured to control the driver to change a position of the electronic device based on movement of the user gesture.

[0019] According to one aspect of the present disclosure, a control method includes: capturing an image; identifying a user posture included in the captured image; identifying at least one parameter among multiple parameters related to the user based on the user posture; and changing the position of the electronic device based on the at least one parameter, and changing the posture of a display included in the electronic device.

[0020] According to one aspect of the present disclosure, an electronic device includes: a memory configured to store instructions; at least one processor configured to: obtain an image captured by a camera, recognize a user posture included in the captured image, identify at least one parameter among multiple parameters related to the user based on the user posture, and control the electronic device based on the at least one parameter to change the position of the electronic device and change the posture of the display.

[0021] The at least one processor may also be configured to: control the electronic device to move toward the user based on receiving a user command; and control capturing a plurality of images at predetermined time intervals.

[0022] The at least one processor may be further configured to recognize the user gesture and a distance from the electronic device to the user based on the plurality of images, and stop operation of the electronic device corresponding to the user gesture based on the distance from the electronic device to the user.

[0023] The at least one processor may also be configured to: identify the multiple parameters related to the user based on the captured image, identify the user posture based on the multiple parameters, change the posture of the display based on at least one first parameter of the multiple parameters based on the user posture being a first posture, and change the posture of the display based on at least one second parameter of the multiple parameters based on the user posture being a second posture.

[0024] The at least one processor may be further configured to control the display to display content and change a posture of the display based on a type of content and the user posture. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The above and other aspects, features and advantages of certain embodiments of the present disclosure will become more apparent through the following description in conjunction with the accompanying drawings, in which: Figure 1a and Figure 1b is a view showing an example bracket device; Figure 2 is a block diagram showing a configuration of an electronic device according to one or more embodiments of the present disclosure; Figure 3 is a block diagram showing a detailed configuration of an electronic device according to one or more embodiments of the present disclosure; Figure 4 is a view showing a structure of an electronic device according to one or more embodiments of the present disclosure; Figure 5 is a view illustrating the operation of an electronic device according to one or more embodiments of the present disclosure; Figure 6 is a diagram showing data used by a processor and an operation of the processor according to one or more embodiments of the present disclosure; Figure 7 is a flowchart illustrating a method of determining a posture of a display according to one or more embodiments of the present disclosure; Figure 8 is a diagram illustrating a method of determining a position of an electronic device according to a user gesture according to one or more embodiments of the present disclosure; Fig. 9 is a view showing roll angle control of a display according to one or more embodiments of the present disclosure; Fig.10 and Fig.11 is a diagram illustrating a prediction operation according to one or more embodiments of the present disclosure; Fig.12 and Fig.13 is a diagram illustrating operations based on content types according to one or more embodiments of the present disclosure; Fig.14 is a diagram showing an adaptive operation to a user according to one or more embodiments of the present disclosure; and Fig.15 is a flowchart illustrating a control method of an electronic device according to one or more embodiments of the present disclosure. DETAILED DESCRIPTION

[0026] Example embodiments are shown in the accompanying drawings and are described in detail in the detailed description. However, it should be understood that the present disclosure is not limited to specific embodiments, but includes all modifications, equivalents and substitutions without departing from the spirit and scope of the present disclosure. In addition, well-known functions or configurations are not described in detail because they would obscure the present disclosure with unnecessary details.

[0027] Provided are an electronic device for providing a screen in a posture optimized for a user's posture and a control method thereof.

[0028] Hereinafter, the present disclosure will be described in detail with reference to the accompanying drawings.

[0029] Considering the functions in the present disclosure, currently widely used general terms are selected as the terms used in the embodiments of the present disclosure, and may be changed based on the intention of technicians in the field or the emergence of judicial precedents, new technologies, etc. In addition, in certain cases, there may be terms arbitrarily selected by the applicant. The meanings of these terms are mentioned in detail in the corresponding description of the present disclosure. Therefore, the terms used in the embodiments of the present disclosure need to be defined on the basis of the meaning of the terms and the content throughout the present disclosure, rather than on the basis of the simple names of the terms.

[0030] In the present disclosure, expressions such as “having”, “may have”, “including”, “may include”, etc. indicate the existence of corresponding features (for example, values, functions, operations, or components such as parts), and do not exclude the existence of additional features.

[0031] The expression “at least one of A or B” may indicate only A, only B, or both A and B.

[0032] The expressions "first", "second", etc. used in the specification may indicate various components regardless of the order or importance of the components. These expressions are only used to distinguish one component from another component and do not limit the corresponding components.

[0033] Unless the context clearly indicates otherwise, a singular term may include its plural form. It should be understood that the terms "comprising", "consisting of", etc. used in this application specify the presence of the features, numbers, steps, operations, components, parts or combinations thereof mentioned in the specification, and do not exclude the presence or addition of one or more other features, numbers, steps, operations, components, parts or combinations thereof.

[0034] In this specification, a term such as “user” may refer to a person using an electronic device or a device (eg, an artificial intelligence electronic device) using an electronic device.

[0035] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings.

[0036] Figure 2is a block diagram illustrating a configuration of an electronic device according to one or more embodiments of the present disclosure.

[0037] The electronic device 100 may be a device that provides a screen to a user. For example, the electronic device 100 may be a device that has a display and provides a displayed screen in a posture optimized for the user. Here, the display may be a component included in the electronic device 100. However, the present disclosure is not limited thereto, and a television (TV), a monitor, a laptop computer, a smart phone, a tablet personal computer (PC), etc. may be attached to the electronic device 100 and operate as a display. The display may be detachable from the electronic device 100.

[0038] However, the present disclosure is not limited thereto, and the electronic device 100 may be a device on which a device having a display is mounted and which provides a position and direction of the display of the mounted device in a posture optimized for a user.

[0039] refer to Figure 2 , the electronic device 100 may include a camera 110 , a driver 120 , a display 130 , and a processor 140 .

[0040] The camera 110 may be a component for capturing a still image or a moving image. The camera 110 may capture a still image at a specific time point, and may also capture still images continuously.

[0041] The camera 110 may capture one direction of the electronic device 100. Specifically, the processor 140 may recognize a user position, a user gesture, etc. based on an image captured by the camera 110.

[0042] The camera 110 may include a lens, a shutter, an aperture, a solid-state imaging device, an analog front end (AFE), and a timing generator (TG). The shutter may adjust the time when light reflected from the subject enters the camera 110, and the aperture may mechanically increase or decrease the size of the opening through which the light enters, thereby controlling the amount of light incident on the lens. If the light reflected from the subject accumulates as photoelectric charges, the solid-state imaging device may output an image as an electrical signal through the photoelectric charges. The TG may output a timing signal for reading out pixel data of the solid-state imaging device, and the AFE may sample and digitize the electrical signal output from the solid-state imaging device.

[0043] The camera 110 may be arranged at one side of the display 130. However, the present disclosure is not limited thereto, and the camera 110 may be arranged at another location of the electronic device 100. For example, the camera 110 may be arranged at a portion connecting the driver 120 and the display 130 to each other. The camera 110 may be arranged at any location where the camera can capture the user.

[0044] The driver 120 may be a component that moves the position of the electronic device 100 under the control of the processor 140 and may include wheels, motors, etc. However, the present disclosure is not limited thereto, and the driver 120 may have any configuration as long as the driver can move the position of the electronic device 100.

[0045] The display 130 may be a component that displays an image, and may be implemented as various types of displays such as a liquid crystal display (LCD), an organic light emitting diode (OLED) display, and a plasma display panel (PDP). The display 130 may also include a driving circuit that may be implemented in a form such as an amorphous silicon thin film transistor (TFT), a low temperature polysilicon (LTPS) TFT, or an organic TFT (OTFT), a backlight unit, and the like. The display 130 may be implemented as a touch screen combined with a touch sensor, a flexible display, a three-dimensional (3D) display, and the like.

[0046] The processor 140 may control the overall operation of the electronic device 100. In detail, the processor 140 may be connected to each component of the electronic device 100, thereby controlling the overall operation of the electronic device 100. For example, the processor 140 may be connected to components such as the camera 110, the driver 120, the display 130, or a communication interface, thereby controlling the operation of the electronic device 100.

[0047] At least one processor 140 may include at least one of a central processing unit (CPU), a graphics processing unit (GPU), an accelerated processing unit (APU), an integrated many-core (MIC), a neural processing unit (NPU), a hardware accelerator, or a machine learning accelerator. At least one processor 140 may control one or any combination of other components of the electronic device 100 and perform operations related to communication or data processing. At least one processor 140 may execute at least one program or instruction stored in a memory. For example, at least one processor 140 may execute a method according to one or more embodiments of the present disclosure by executing at least one instruction stored in a memory.

[0048] If the method according to one or more embodiments of the present disclosure includes multiple operations, the multiple operations may be performed by one processor, or may be performed by multiple processors. For example, the first operation, the second operation, and the third operation may be performed by the method according to one or more embodiments. The first operation, the second operation, and the third operation may all be performed by the first processor. Alternatively, the first operation and the second operation may all be performed by the first processor (e.g., a general-purpose processor), and the third operation may be performed by the second processor (e.g., a pure artificial intelligence processor).

[0049] At least one processor 140 may be implemented as a single-core processor including one core, or may be implemented as at least one multi-core processor including multiple cores (homogeneous multi-cores or heterogeneous multi-cores). If at least one processor 140 is implemented as a multi-core processor, each multi-core included in the multi-core processor may include a processor internal memory, such as a cache memory or an on-chip memory, and a public cache shared by the multi-cores may be included in the multi-core processor. In addition, each of the multi-cores (or some of the multi-cores) included in the multi-core processor may independently read and execute program instructions for implementing the method according to one or more embodiments of the present disclosure. Optionally, all (or some) of the multi-cores may be linked to read and execute program instructions for implementing the method according to one or more embodiments of the present disclosure.

[0050] If the method according to one or more embodiments of the present disclosure includes multiple operations, the multiple operations may be performed by one of the multiple cores included in the multi-core processor, or may be performed by the multiple cores. For example, the first operation, the second operation, and the third operation may be performed by the method according to one or more embodiments. The first operation, the second operation, and the third operation may all be performed by the first core included in the multi-core processor. Alternatively, the first operation and the second operation may be performed by the first core included in the multi-core processor, and the third operation may be performed by the second core included in the multi-core processor.

[0051] In one or more embodiments of the present disclosure, the processor may refer to a system on chip (SoC) in which at least one processor and other electronic components are integrated, a single-core processor, a multi-core processor, or a core included in a single-core processor or a multi-core processor. Here, the core may be implemented as a CPU, a GPU, an APU, a MIC, an NPU, a hardware accelerator, or a machine learning accelerator. However, the embodiments of the present disclosure are not limited thereto. However, for the convenience of description, the operation of the electronic device 100 is described hereinafter by the expression "processor 140".

[0052] The processor 140 may capture an image through the camera 110 and recognize a user gesture included in the captured image. For example, the processor 140 may recognize the user gesture by inputting the captured image into a first neural network model. Alternatively, the processor 140 may recognize a main part of the user from the captured image and recognize the user gesture based on the recognized main part. For example, the processor 140 may recognize the eyes, shoulders, waist, knees, or ankles of the user and recognize the user gesture based on the position of the main part and the distance between the main parts.

[0053] The processor 140 may identify at least one parameter among a plurality of parameters related to the user based on the user's posture, control the driver 120 to change the position of the electronic device 100 based on the at least one parameter, and may change the posture of the display 130 .

[0054] For example, the user may have a sitting posture. In this case, the processor 140 may identify a yaw angle, a pitch angle, and a roll angle representing the user's eye height, eye position, or head / body orientation, and may control the driver 120 to change the position of the electronic device 100 to a predetermined position in front of the user based on at least one parameter, and may change the posture of the display 130. Here, the predetermined position may be a position of the electronic device 100 determined during its manufacture. For example, the predetermined position may be the position of the electronic device 100 1 meter in front of the user. The predetermined position may be changed based on a user operation. For example, the user may manually change the position of the electronic device 100. In this case, the electronic device 100 may have a predetermined position that is changed based on a manual operation of the user. The posture of the display 130 may represent the orientation and height of the display 130, and the orientation of the display 130 may be represented by a yaw angle, a pitch angle, or a roll angle. The yaw angle may be an angle that changes in the left and right directions on a horizontal plane, the pitch angle may be an angle that tilts up and down on a horizontal plane, and the roll angle may be an angle that tilts left and right on a horizontal plane.

[0055] Alternatively, the user may have a standing posture. In this case, the processor 140 may identify a yaw angle representing eye height, eye position, or body orientation, and control the driver 120 to change the position of the electronic device 100 to a predetermined position in front of the user based on the identified parameters, and may change the posture of the display 130.

[0056] Alternatively, the user may have a lying position. In this case, the processor 140 may identify a yaw angle or a roll angle representing eye height, eye position, or body orientation, and may control the driver 120 to change the position of the electronic device 100 to a predetermined position in front of the user based on the identified parameters, and may change the posture of the display 130.

[0057] That is, the processor 140 may change the parameters for changing the position of the electronic device 100 or the posture of the display 130 based on the user's posture. Here, if an image is captured by the camera 110, the processor 140 may obtain a plurality of parameters from the captured image. Alternatively, the processor 140 may obtain a plurality of parameters related to the user through a sensor. Alternatively, the processor 140 may obtain a plurality of parameters after recognizing the user.

[0058] The processor 140 may recognize a user position or a user posture from the image captured by the camera 110, may control the driver 120 to change the position of the electronic device 100 to a predetermined position in front of the user based on the user position, may finely adjust the position of the electronic device 100 based on the user posture, and may change the posture of the display 130. For example, the processor 140 may recognize the user from the captured image and move toward the user. For example, the user may be recognized on the left side of the center of the captured image. The processor 140 may control the driver 120 to move the electronic device 100 to the left front. Here, the driver 120 may include at least one wheel and at least one motor, and may have any configuration as long as the driver can move the electronic device 100. The processor 140 may then change the position of the electronic device 100 based on the user posture. That is, the processor 140 may roughly move the electronic device 100 based on the user position, and then if the user posture is recognized, recognize the final position of the electronic device 100 based on the user posture.

[0059] The electronic device 100 may further include a communication interface and a microphone, and if a user command to call the electronic device 100 is received from another electronic device through the communication interface, or a user voice to call the electronic device 100 is received through the microphone, the processor 140 may control the driver 120 to move toward the user based on the user command or the user voice, and acquire multiple images by turning on the camera 110 to capture images at predetermined time intervals. The processor 140 may recognize the user's gesture and the distance from the user in real time through the images acquired at predetermined time intervals, and if the distance from the user corresponds to the user's gesture, stop the operation of the driver 120. In addition, the processor 140 may keep the camera 110 turned on.

[0060] For example, the user may input a user command for calling the electronic device 100 through a device such as a smart phone, or call the electronic device 100 through a user voice. The processor 140 may change the position of the electronic device 100 based on the user command received from the smart phone or the user voice received through the microphone. For example, if the user inputs a user command of "come to the living room", the processor 140 may move to the living room through text recognition. Alternatively, the electronic device 100 may include a plurality of microphones, and the processor 140 may recognize the user's speaking position through the plurality of microphones and move to the user position. The processor 140 may turn on the camera 110 to capture an image when changing the position of the electronic device 100, and stop changing the position of the electronic device 100 based on the user identified in the image. For example, the processor 140 may turn on the camera 110 to capture an image when changing the position of the electronic device 100, and stop changing the position of the electronic device 100 if the distance between the electronic device 100 and the user is a predetermined distance based on the user identified in the image. The position of the electronic device 100 may be efficiently changed through this operation. In addition, since the camera 110 is turned on after the user command is input or the user voice is received, unnecessary power consumption of the camera may be reduced.

[0061] Optionally, if a user command to call the electronic device 100 is received from another electronic device through the communication interface, or a user voice to call the electronic device 100 is received through the microphone, the processor 140 may turn on the camera 110 to capture images at a predetermined time interval to obtain multiple images, and control the driver 120 to move toward a user identified from each of the multiple images.

[0062] However, the present disclosure is not limited thereto. The processor 140 may capture images through the camera 110 at predetermined time intervals, and if a predetermined user gesture is recognized, the driver is controlled to move toward the user. That is, even if there is no user command or user voice, the processor 140 may control the driver 120 to move toward the user based on a predetermined gesture. Here, the predetermined gesture may be a gesture for calling the electronic device 100, and includes at least one of a gesture stored during the manufacturing period or a gesture set by the user. Optionally, the electronic device 100 may pre-store data about a plurality of predetermined positions, and the processor 140 may change the position of the electronic device 100 to one of the plurality of predetermined positions based on a user command or user voice. For example, a user may manually move the electronic device to a specific position in a bedroom, and the specific position is mapped and stored together with the command "bedroom", and if a user voice of "bedroom" is received, the processor 140 may change the position of the electronic device 100 to the specific position in the bedroom. The processor 140 may turn on the camera 110 after the movement is completed, rather than turning on the camera when the electronic device 100 moves. In addition, the processor 140 may recognize a user gesture by capturing one image instead of capturing a plurality of images after the camera 110 is turned on.

[0063] However, the present disclosure is not limited thereto, and the above-mentioned command mapping may be performed in various ways. For example, the electronic device 100 may be moved to a specific location based on a user call, and then the user is asked whether command mapping is required. Optionally, the electronic device 100 may be moved to a specific location based on a user call, and then if the user feels comfortable operating the electronic device at a specific location, the specific location and command are mapped and stored. Optionally, the electronic device 100 may identify a plan view of the user's location when moving based on a user call, identify the type of each space in the plan view, and identify a predetermined location based on the type of each space. For example, the electronic device 100 may identify a plan view when moving based on a user call, wherein the plan view includes a first area where the electronic device 100 is movable and a second area where there are obstacles such as walls at the user's location, and may identify whether the first area is a living room, etc., and may identify a predetermined location based on each type of the first area. Here, the predetermined location may be a location set by a manufacturer. For example, the center may be pre-stored as a predetermined location of a living room type space.

[0064] When the user views content through the display 130, the camera 110 may remain turned on. For example, the camera 110 may remain turned on until the time when the user turns off the electronic device 100. However, the present disclosure is not limited thereto, and the camera 110 may always be turned on. For example, the camera 110 may be turned on even if the display 130 does not display content. The processor 140 may control the electronic device 100 by detecting a user gesture from images continuously captured by the camera 110.

[0065] The processor 140 may identify multiple parameters related to the user based on the captured image, and identify the user's posture based on the multiple parameters. For example, the processor 140 may identify the yaw angle, pitch angle or roll angle representing the user's position or head / body orientation, changes in posture (sitting, lying, standing, etc.), user gestures, user key points, etc. as multiple parameters based on the captured image. However, the present disclosure is not limited to this, and the multiple parameters related to the user may include various elements. Here, the user's key points may represent the main body parts of the user and may be predetermined by the manufacturer. For example, the user's key points may include at least one of the user's eyes, ears, neck, shoulders, elbows, hips, or knees. However, the present disclosure is not limited to this, and the user's key points may include various main body parts of the user.

[0066] If the user gesture is a first gesture, the processor 140 may change the gesture of the display 130 based on at least one first parameter among the plurality of parameters, and if the user gesture is a second gesture, the processor 140 may change the gesture of the display 130 based on at least one second parameter among the plurality of parameters. That is, the processor 140 may change the gesture of the display 130 by using at least one parameter among the plurality of parameters recognized based on the user gesture. Here, at least one first parameter used in the case of the first gesture and at least one second parameter used in the case of the second gesture may partially overlap each other.

[0067] However, the present disclosure is not limited thereto, and the processor 140 may change the posture of the display 130 by using the user posture and all of the plurality of parameters. Alternatively, the processor 140 may change the posture of the display 130 by using all of the plurality of parameters. For example, the processor 140 may determine whether to use all or some parameters based on the spare resources of the electronic device 100.

[0068] The processor 140 may control the display 130 to display content and change the posture of the display 130 based on the content type and the user posture. For example, if the content type is the second type instead of the first type, even if the user posture is the same, the processor 140 may increase the height of the display 130 or further change the direction of the display 130. For example, if the content type is a sports broadcast instead of a game broadcast, the processor 140 may increase the height of the display for multiple users to view the content.

[0069] If the user posture changes, the processor 140 may change the posture of the display 130 based on the changed user posture. For example, if the height of the user's face changes, the processor 140 may change the height of the display 130 based on the height of the user's face.

[0070] Alternatively, if the user posture changes and a predetermined time has passed, the processor 140 may change the posture of the display 130 based on the changed user posture. For example, if the height of the user's face changes and one minute has passed, the processor 140 may change the height of the display 130 based on the height of the user's face. Here, the predetermined time may be changed in various ways.

[0071] However, the present disclosure is not limited thereto, and only when one of the user's sitting posture, standing posture, and lying posture is changed to another posture, the processor 140 may change the posture of the display 130. For example, if the user posture is changed from the sitting posture to the standing posture and a predetermined time has passed, the processor 140 may change the posture of the display 130 based on the changed user posture. However, if only the height of the user's face is changed when the user is in the sitting posture, the processor 140 may not change the posture of the display 130.

[0072] In addition, if the user posture changes and a predetermined time has passed, the processor 140 may change the position of the electronic device 100 based on the changed user posture. For example, if the user posture changes from a sitting position to a standing position and a predetermined time has passed, the processor 140 may increase the distance between the user and the electronic device 100.

[0073] The processor 140 may acquire a plurality of images by capturing images at predetermined time intervals by the camera 110, acquire a predicted user posture at a predetermined time point from a current time point based on the plurality of images, and change the posture of the display 130 based on the user posture and the predicted user posture. In addition, the processor 140 may change the position of the electronic device 100 based on the user posture and the predicted user posture. Through this operation, the processor 140 may solve the inconvenience caused to the user by the time delay caused by the user's body movement or the change of the body posture.

[0074] The electronic device 100 may further include a memory storing a first neural network model, and the processor 140 may recognize a user gesture by inputting the captured image into the first neural network model. However, the present disclosure is not limited thereto, and the processor 140 may recognize a user gesture from an image through a rule-based model. Here, the first neural network model may be a model acquired by learning a sample image including a user and data on a user gesture corresponding to the sample image.

[0075] Optionally, the electronic device 100 may further include a memory storing a second neural network model, and the processor 140 may acquire data on the posture of the display 130 by inputting a plurality of parameters into the second neural network model, and change the posture of the display 130 based on the data on the posture of the display 130. For example, the processor 140 may input a plurality of parameters into the second neural network model to acquire the height, yaw angle, pitch angle, or roll angle of the display 130, and change the posture of the display 130 based on the height, the yaw angle, the pitch angle, or the roll angle. Here, the second neural network model may be a model acquired by learning a plurality of sample parameters and data on the posture of the display 130 corresponding to the plurality of sample parameters.

[0076] Here, the electronic device 100 may further include a user interface such as a button, a touchpad, a mouse, a keyboard, or a touch screen. If a user command for changing the posture of the display 130 is received through the user interface, the processor 140 may change the posture of the display 130 based on the user command, and update the second neural network model based on multiple parameters and the changed posture of the display 130. Through this operation, the processor can adapt the operation of the electronic device 100 to the user. That is, even if the user posture does not change, the posture of the display 130 can be changed based on the user's manual operation, and the processor 140 can learn this situation and provide the best posture of the display 130 to the user.

[0077] The electronic device 100 may further include a microphone, and the processor 140 may acquire a plurality of images by capturing images at predetermined time intervals using the camera 110, and if a predetermined user gesture is recognized from the plurality of images, or a user voice for changing the posture of the display 130 is received through the microphone, the posture of the display 130 is changed based on the user gesture or the user voice. Here, the predetermined time interval may be the number of frames per second (fps) of the camera 110. However, the present disclosure is not limited thereto. For example, the predetermined time interval may be lower than the fps of the camera 110, thereby reducing power consumption. Alternatively, the predetermined time interval may be changed based on the distance between the electronic device 100 and the user. For example, if the distance between the electronic device 100 and the user is a predetermined distance or more, the predetermined time interval may be a first time interval, and if the distance between the electronic device 100 and the user is less than the predetermined distance, the predetermined time interval may be a second time interval less than the first time interval.

[0078] However, the present disclosure is not limited thereto, and the processor 140 may change the posture of the display 130 based on a user command received from a smart phone or the like.

[0079] The camera 110 may be arranged at one side of the display 130, and the processor 140 may change the posture of the display 130 so that the user or a predetermined user body part is included in a predetermined area in the captured image. For example, the processor 140 may change the posture of the display 130 so that the user's face is included within a predetermined distance from the center of the captured image. That is, the processor 140 may maintain the posture of the display 130 so that the camera 110 captures the user in front of the display 130 when the posture of the display 130 is changed. Therefore, the processor 140 can prevent the problem of failing to capture the user due to a drastic change in the posture of the display 130.

[0080] The electronic device 100 may further include a display gesture driver 190 , and the processor 140 may control the display gesture driver to change the gesture of the display 130 .

[0081] The functions of the electronic device related to artificial intelligence according to the present disclosure may be operated through the processor 140 and the memory.

[0082] The processor 140 may include at least one processor. Here, the at least one processor may be a general-purpose processor such as a central processing unit (CPU), an application processor (AP), or a digital signal processor (DSP), a graphics-only processor such as a graphics processing unit (GPU) or a visual processing unit (VPU), or a pure artificial intelligence processor such as a neural processing unit (NPU).

[0083] At least one processor may control the input data to be processed based on predefined operating rules or artificial intelligence models stored in the memory. Optionally, at least one processor may be a pure artificial intelligence processor. The pure artificial intelligence processor may be designed to have a hardware structure that is specialized for processing a specific artificial intelligence model. The predefined operating rules or artificial intelligence models may be generated by learning.

[0084] Here, generating by learning may indicate that the basic artificial intelligence model performs learning based on a learning algorithm by using a number of learning data, thereby generating a predefined operating rule or artificial intelligence model that is set to perform the desired feature (or purpose). Such learning can be performed by the machine itself that performs artificial intelligence according to the present disclosure, or by a separate server or system. An example of a learning algorithm may include supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, and may not be limited thereto.

[0085] The artificial intelligence model may include multiple neural network layers. The multiple neural network layers may have multiple weight values, respectively, and perform neural network operations by using the operation results of the previous layers and operations between the multiple weight values. The multiple weight values ​​of the multiple neural network layers may be optimized by the learning results of the artificial intelligence model. For example, the multiple weight values ​​may be updated during the learning process to reduce or minimize the loss value or cost value obtained from the artificial intelligence model.

[0086] For example, the first neural network model may be a model acquired by learning a plurality of sample images and user postures included in the plurality of sample images. Here, the plurality of sample images may be captured images including a user, and the user postures included in the plurality of sample images may be manually input data. The second neural network model may be a model acquired by learning a plurality of sample parameters and data about the posture of the display 130 corresponding to the plurality of sample parameters. The second neural network model may be updated by performing additional learning based on the user operation. For example, if the posture of the display 130 is changed based on the user operation, and the user posture is not changed, the second neural network model may be updated by additionally learning a plurality of sample parameters corresponding to the user posture and the changed posture of the display 130.

[0087] The artificial neural network may include a deep neural network (DNN), which may include, for example, a convolutional neural network (CNN), a deep neural network (DNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a generative adversarial network (GAN) or a deep Q network, etc., and is not limited thereto.

[0088] Figure 3 is a block diagram illustrating a detailed configuration of an electronic device according to one or more embodiments of the present disclosure.

[0089] The electronic device 100 may include a camera 110, a driver 120, a display 130, and a processor 140. Figure 3 , the electronic device 100 may also include a communication interface 150, a microphone 160, a memory 170, a user interface 180, a sensor 185, a display gesture driver 190, and a speaker 195. This specification omits Figure 3 The components shown are Figure 2 Descriptions of components that are identical to the components shown.

[0090] The communication interface 150 may be a component that communicates with various types of external device components by using various types of communication methods. For example, the electronic device 100 may communicate with a user terminal device or a server through the communication interface 150 .

[0091] The communication interface 150 may include a Wireless Fidelity (Wi-Fi) module, a Bluetooth module, an infrared communication module, a wireless communication module, etc. Here, each communication module may be implemented in the form of at least one hardware chip.

[0092] The Wi-Fi module and the Bluetooth module can perform communication in a Wi-Fi manner and a Bluetooth manner, respectively. In the case of using the Wi-Fi module or the Bluetooth module, various connection data such as a service set identifier (SSID) or a session key may be first sent and received, communication may be connected by using this connection data, and various data may be subsequently sent and received. The infrared communication module may perform communication based on the Infrared Data Association (IrDA) technology, which wirelessly transmits data within a short distance by using infrared rays between visible waves and millimeter waves.

[0093] In addition to the above-mentioned communication methods, the wireless communication module may include at least one communication chip, which performs communication based on various wireless communication standards (such as zigbee, third generation (3G), third generation partnership project (3GPP), long term evolution technology (LTE), advanced LTE (LTE-A), fourth generation (4G) and fifth generation (5G)).

[0094] Optionally, the communication interface 150 may include a wired communication interface such as a High-Definition Multimedia Interface (HDMI), a Display Port (DP), a Thunderbolt interface, a Universal Serial Bus (USB), a Red, Green, and Blue (RGB) port, a D-Subminiature Interface (D-SUB), a Digital Video Interface (DVI), and the like.

[0095] In addition, the communication interface 150 may include at least one of a wired communication module that performs communication by using a local area network (LAN) module, an Ethernet module, a twisted pair cable, a coaxial cable, an optical fiber cable, or the like.

[0096] The microphone 160 may be a component for receiving sound and converting it into an audio signal (ie, an electrical signal). The microphone 160 may be electrically connected to the processor 140 and may receive sound under the control of the processor 140.

[0097] For example, the microphone may be arranged on the top, front or side of the electronic device 100. Alternatively, the microphone 160 may be arranged in a device separate from the electronic device 100, for example, a remote controller. The remote controller may be a separate device that can receive sound through the microphone 160 and provide an audio signal to the electronic device 100, the audio signal being an electrical signal corresponding to the received sound.

[0098] The microphone 160 may include various components such as a microphone that collects sound in an analog form, an amplifier circuit that amplifies the collected sound, an analog-to-digital (A / D) conversion circuit that samples the amplified sound and converts it into a digital signal, and a filter circuit that removes noise components from the converted digital signal.

[0099] The microphone 160 may be implemented in the form of a sound sensor and may use any method that can collect sound.

[0100] The memory 170 may refer to hardware that stores data in an electrical or magnetic form for the processor 140 etc. to access the data. To this end, the memory 170 may be implemented as a nonvolatile memory, a volatile memory, a flash memory, a hard disk drive (HDD), a solid state drive (SSD), a random access memory (RAM), a read-only memory (ROM), etc.

[0101] The memory 170 may store at least one instruction necessary for the operation of the electronic device 100 or the processor 140. Here, the instruction is a code unit that instructs the operation of the electronic device 100 or the processor 140, and may be written in a machine language, which is a language that a computer can understand. Alternatively, the memory 170 may store a plurality of instructions for executing a specific task of the electronic device 100 or the processor 140 as an instruction set.

[0102] The memory 170 may store data in units of bits or bytes, and the data may represent characters, numbers, images, etc. For example, the memory 170 may store a neural network model, etc.

[0103] The memory 170 can be accessed by the processor 140 , and the reading, recording, correction, deletion, update, etc. of an instruction, an instruction set, or data can be executed by the processor 140 .

[0104] The user interface 180 may be implemented as a button, a touch pad, a mouse or a keyboard, or may be implemented as a touch screen that can perform a display function and an operation input function. Here, the button may be any one of various types of buttons, such as a mechanical button, a touch pad and a scroll wheel, which is arranged in any area of ​​the main body appearance of the electronic device 100, such as its front surface portion, side surface portion or rear surface portion.

[0105] The sensor 185 may include a sensor with red, green, blue plus depth data (RGB-D sensor) or a time of flight (ToF) sensor. The processor 140 may acquire an RGB image in the visible light region through the RGB-D sensor, a depth image through infrared (IR), and a thermal image through IR. The ToF sensor is a three-dimensional sensor and may be a sensor that recognizes a three-dimensional effect, spatial data, and motion of an object by calculating the distance of light emitted from the object through infrared wavelengths reflected back in time.

[0106] The sensor 185 may further include an illuminance sensor. The illuminance sensor may be a sensor that detects illuminance representing the amount of light received per unit area per unit time.

[0107] The display posture driver 190 may be a component for changing the posture of the display 130. For example, the display posture driver 190 may include a first driver for changing the height of the display 130 and three second drivers for changing the three-axis angle of the display; and the posture of the display 130 may be changed by changing at least one of the height or the three-axis angle of the display 130. Here, the first driver may include a first motor connected to a gear, and a member in which one angle of the gear is engaged with the gear. The first driver may drive the first motor under the control of the processor 140, and move the member up and down when the first motor is driven. Here, the member may be connected to the display 130. Each of the three second drivers may include a second motor connected to the gear. Each of the three second drivers may drive the second motor under the control of the processor 140, and rotate the display 130 when the second motor is driven. Here, the three second drivers may be implemented to rotate the display 130 in directions perpendicular to each other. However, the present disclosure is not limited thereto, and the display posture driver 190 may be implemented in various forms.

[0108] The speaker 195 may be a component that outputs various notification sounds, audio messages, etc., as well as various audio data processed by the processor 140 .

[0109] As described above, the electronic device 100 can improve the convenience of the user by providing a screen in a posture optimized for the user. In addition, the electronic device 100 can perform adaptive operations on the user by learning the user's posture and the posture of the display.

[0110] In the following, reference Figures 4 to 13 The operation of the electronic device 100 is described in more detail. Figures 4 to 13 However, reference is made to Figures 4 to 13 The described various embodiments may be implemented in any combination thereof.

[0111] Figure 4 is a view showing a structure of an electronic device 100 according to one or more embodiments of the present disclosure.

[0112] like Figure 4 As shown, the electronic device 100 may include a body 405 , a driver 120 , a display 130 , and a display gesture driver 190 .

[0113] The body 405 may be a structure including the driver 120 disposed on its lower side and supporting the display 130 through the display posture driver 190. The body 405 may include a battery 410 and a light detection and ranging (LiDAR) sensor 420-1. The battery 410 may supply power to the driver 120, and the LiDAR sensor 420-1 may acquire scan data of the surrounding terrain.

[0114] The driver 120 may be disposed at the lower side of the body 405 and include a plurality of wheels. The driver 120 may move the position of the electronic device 100 under the control of the processor 140. For example, the driver 120 may include a plurality of wheels and a control circuit for controlling the plurality of wheels under the control of the processor 140.

[0115] The driver 120 may include an RGB-D sensor 420-2 and a time-of-flight (ToF) sensor 420-3. The RGB-D sensor 420-2 may include an infrared sensor, and estimate its distance from the subject by analyzing an infrared signal incident after the infrared rays emitted by the infrared sensor are reflected by the subject. In addition, the RGB-D sensor 420-2 can detect heat sources by using infrared rays, because heat sources can be detected in parts of the infrared region, and all objects of 0K or greater can emit thermal radiation. That is, the RGB-D sensor 420-2 can acquire RGB images in the visible light region of the user, acquire depth images through infrared rays (IR), and acquire thermal images through IR. The ToF sensor 420-3 is a three-dimensional sensor, and can be a sensor that recognizes the three-dimensional effect, spatial data, and motion of an object by calculating the distance reflected back in time from the light emitted by the object through the infrared wavelength.

[0116] The display posture driver 190 may connect the main body 405 with the display 130 and change the posture of the display 130 under the control of the processor 140. For example, the display posture driver 190 may include a first driver for changing the height of the display 130 and three second drivers for changing the three-axis angle of the display. Here, the first driver may be a component for changing the height of the display posture driver 190 based on the main body 405. Alternatively, the display posture driver 190 may include a mounting member for supporting the display 130 and a connecting member connecting the mounting member with the main body 405, the first driver may be a component for changing the height of the mounting member based on the connecting member, and the three second drivers may be components for changing the three-axis angle of the mounting member based on the connecting member.

[0117] Figure 5 is a view illustrating an operation of an electronic device according to one or more embodiments of the present disclosure.

[0118] The electronic device 100 may include Figure 5 The display 130 is shown. However, the present disclosure is not limited thereto, and the electronic device 100 can be implemented in a form that enables the electronic device to accommodate an external display. For example, the electronic device 100 may include mounting members around the upper, lower, left, right, and outer edges of the external display.

[0119] The processor 140 may change the posture of the display 130 based on at least one parameter corresponding to the user's posture. Figure 5 As shown on the left side of , the processor 140 may recognize that the user has a lying position based on the user's body orientation. Optionally, the processor 140 may recognize that the user has a lying position based on the positions of the user's eyes, waist, and knees. Optionally, the processor 140 may input an image into a first neural network model to recognize that the user has a lying position, and recognize the user's posture by using various methods. If the user has a lying position, the processor 140 may tilt the display 130 based on the height of the user's eyes.

[0120] The processor 140 may change the posture of the display 130 based on the change of the user's posture. Figure 5 The user's lying position shown on the left side of the figure can be changed to Figure 5 The processor 140 may change the position of the electronic device 100 and the posture of the display 130 based on the eye height of the user.

[0121] Optionally, the processor 140 may change the position of the electronic device 100 based on the yaw angle of the user's head / shoulder, and change the height and pitch angle of the display 130 based on the user's eye height and pitch angle. Figure 5 The yaw angle of the user's head shown on the left side is 510-1, and the user rotates a certain angle to Figure 5 The yaw angle of the user's head shown on the right side of FIG. 5 is 510-2. The processor 140 can control the driver 120 to move the position of the electronic device 100 to a position farther away from the user. In addition, if the user's eye height changes from Figure 5 The left side increases to Figure 5 , the processor 140 may lower the height of the display 130. Here, the processor 140 may identify the relative height of the user's eyes from the captured images. For example, the processor 140 may identify the user's eyes from the first image, identify the user's eyes from the second image captured after the first image, and identify that the position of the user's eyes is raised.

[0122] However, the present disclosure is not limited thereto, and the processor 140 may change at least one of the position of the electronic device 100 or the posture of the display 130 based on various parameters.

[0123] Optionally, the processor 140 may change at least one of the position of the electronic device 100 or the posture of the display 130 based on obstacles around the electronic device 100. For example, if an obstacle is found when the processor changes the position of the electronic device 100 based on the change of the user's posture, the processor 140 may change the operation of changing the position of the electronic device 100 to the operation of changing the posture of the display 130. For example, if the processor 140 is no longer close to the user due to the obstacle found when the processor changes the position of the electronic device, the processor 140 may stop the operation of changing the position of the electronic device 100 and increase the height of the display 130. Optionally, when the processor 140 changes the position of the electronic device 100 to the first position based on the change of the user's posture, an obstacle may be found. The processor 140 may change the position of the electronic device 100 from the first position to the second position, and update the posture of the display 130 based on the changed second position. For example, if the processor 140 changes the position of the electronic device 100 from the first position to the second position further to the left, the processor 140 may further rotate the posture of the display 130 to the right.

[0124] Figure 6 1 is a diagram showing data used by the processor 140 and operations of the processor based on the data according to one or more embodiments of the present disclosure. Figure 6 , the present specification describes that the processor 140 includes an image recognizer, a speech recognizer, and a display gesture determiner. Each of the image recognizer, the speech recognizer, and the display gesture determiner may be implemented as software or as a hardware component of the processor 140.

[0125] The image recognizer may recognize the orientation of the user's head / body, the user's position (distance, or direction, i.e., the position of a person (distance or angle)), the user's posture (or the person's posture), the eye position, key points, or gestures from the captured image by using the camera 110. For example, the image recognizer may gradually recognize data about the user, such as recognizing the user's head from the captured image by using the camera 110, and recognizing the eyes in the user's head. In addition, the image recognizer may recognize important points such as the user's eyes, ears, shoulders, elbows, pelvis, knees, etc. as key points from the captured image. The image recognizer may acquire data such as head orientation, body orientation, user position, user posture, eye height, etc. based on the acquired data. For example, the image recognizer may acquire the key points of the user by inputting the captured image into a neural network model. Here, the neural network model may be a model acquired by learning a plurality of sample images and the key points of the users included in the plurality of sample images. However, the present disclosure is not limited thereto, and the image recognizer may also acquire data about various users. In addition, the image recognizer may be implemented with a neural network model.

[0126] The speech recognizer may recognize the user direction based on the user speech received through the microphone 160. Specifically, the speech recognizer may recognize the user's mouth direction based on the user speech. For example, the microphone 160 may include a first microphone and a second microphone spaced d from the first microphone. The incident angle θ of the user speech received through the first microphone may be Here, ψ is a phase difference, λ is a wavelength, and the phase difference and the wavelength can be acquired by two antennas. However, the present disclosure is not limited thereto, and various methods for acquiring the user direction may be provided.

[0127] However, the voice recognizer may be an additional component. For example, the posture of the display 130 may be determined based only on data acquired from the image recognizer.

[0128] The display posture determiner may determine the posture of the display 130 based on data received by at least one of the image recognizer or the voice recognizer. For example, the display posture determiner may change at least one of the three-axis angle or height of the display 130 based on the user posture acquired from the image recognizer.

[0129] The display posture determiner may determine the posture of the display 130 based on the data received from the image recognizer and the spatial data (or semantic graph) of the user's position. For example, if the user is recognized as having a lying posture and lying on the bed, rather than if the user is recognized as having a lying posture and lying on the floor, the display posture determiner may add the posture of the display 130. That is, the display posture determiner may determine the posture of the display 130 by further considering the spatial data of the user's position as well as the user's posture. Here, the spatial data of the user's position may be acquired through a neural network model. For example, the neural network model that outputs the spatial data of the user's position may be a model acquired by learning data about a plurality of sample images and the space where the user is located in the plurality of sample images.

[0130] The display posture determiner may determine the posture of the display 130 based on the data received from the image recognizer and the content type. For example, if the user has a sitting posture and the content is a normal video, the display posture determiner may determine a longer portion of the screen of the display 130 as the posture of the display parallel to the ground; and if the user has a sitting posture and the content is a user interface (UI) for a video call, the display posture determiner may determine a shorter portion of the screen of the display 130 as the posture of the display parallel to the ground.

[0131] In the above, the present specification exemplifies various factors for determining the posture of the display 130, the present disclosure is not limited thereto, and various parameters may be used in the process of determining the posture of the display 130. For example, the display posture determiner may recognize the illuminance around the electronic device 100, and determine the posture of the display 130 by further considering the illuminance. For example, if the illuminance is low, the display posture determiner may tilt the display 130 more than in the case where the illuminance is high, thereby preventing light emitted from the display 130 from directly hitting the user's eyes.

[0132] Alternatively, if a plurality of users are identified, the display posture determiner may identify the user calling the electronic device 100, and recognize the posture of the display 130 based on the posture of the user calling the electronic device 100. For example, if a call signal is received from a user terminal device such as a smart phone, the display posture determiner may identify the user of the smart phone among the plurality of users as the user calling the electronic device 100, and recognize the posture of the display 130 based on the posture of the user of the smart phone. If a plurality of users are identified and the user calling the electronic device 100 is not identified, the processor 140 may display a UI for inquiring the user calling the electronic device 100, or output a sound for inquiring the user calling the electronic device 100. Alternatively, if a plurality of users are identified, the processor 140 may display a UI for inquiring whether the call of the electronic device 100 is for individual viewing or collective viewing, or output a sound for inquiring whether the call of the electronic device 100 is for individual viewing or collective viewing. If a user command indicating that the call of the electronic device 100 is for individual viewing is received, the display posture determiner may move the position of the electronic device 100 and change the posture of the display 130 based on the posture of the user providing the user command. If a user command indicating that the call of the electronic device 100 is for collective viewing is received, the display posture determiner may change the position of the electronic device 100 and change the posture of the display 130 based on the positions of the plurality of users.

[0133] The display posture determiner may change the posture of the display 130 by providing the display posture driver with data regarding the posture of the display 130. Here, the data regarding the posture of the display 130 may include a signal for changing at least one of the height of the display 130 or the three-axis angle of the display 130. For example, the display posture determiner may provide the display posture driver with a control signal for increasing the height of the display 130 by two stages.

[0134] Figure 7 Is a flowchart showing a method of determining the posture of the display 130 according to one or more embodiments of the present disclosure. First, the processor 140 may capture an image by using the camera 110 (S705). For example, if a user calls the electronic device 100, the processor 140 may capture an image by using the camera 110. The processor 140 may capture an image at predetermined time intervals or capture a moving image.

[0135] The processor 140 may extract a plurality of parameters from the captured image (S710). For example, the processor 140 may identify at least one of a user position, an eye height, a head orientation, a body orientation, or a key point. Alternatively, the processor 140 may extract the parameters through a sensor. For example, the camera 110 may further include a depth camera, and the processor 140 may acquire three-dimensional (3D) data of a front object through the depth camera. Each of the plurality of parameters may be represented as data in a 3D space.

[0136] The processor 140 may identify the user posture based on a plurality of parameters (S720). For example, the processor 140 may identify the user posture as one of a sitting posture, a standing posture, and a lying posture based on a plurality of parameters. For example, if the user position is a bed or a sofa, the processor 140 may identify whether the user has a sitting posture or a lying posture based on the user's eye height, head orientation, and body orientation, while excluding the standing posture in the user posture. That is, the processor 140 may identify the user posture by gradually using a plurality of parameters. However, the present disclosure is not limited thereto, and the user posture may be further subdivided.

[0137] Optionally, the processor 140 may recognize the user's gesture based on a change in eye height, a change in head orientation, a change in body orientation, or at least one of these changes in a plurality of parameters. The processor 140 may control the camera 110 to capture a plurality of images and recognize changes in a plurality of parameters related to the user from the plurality of images.

[0138] Alternatively, the processor 140 may assign a score to each of the plurality of postures based on each of the plurality of parameters, and finally recognize the posture with the highest score as the user posture. For example, if the user position is a bed or a sofa, the processor 140 may assign 1 point to each of the sitting posture and the lying posture, and if the user's eye height is 1m or more from the ground, the processor 140 may assign 1 point to each of the sitting posture and the standing posture. The processor 140 may assign 1 point to each of the plurality of postures based on each of the plurality of parameters in this manner, and then recognize the user posture.

[0139] Alternatively, the processor 140 may recognize the user gesture through a neural network model. For example, the neural network model may be a model acquired by learning the relationship between multiple parameters related to the user and the user gesture, and the processor 140 may input multiple parameters into the neural network model to recognize the user gesture.

[0140] If the user is identified as having a first posture, the processor 140 may determine the posture of the display 130 based on at least one first parameter among the plurality of parameters (S760). For example, if the user is identified as having a sitting posture, the processor 140 may determine the posture of the display 130 based on a yaw angle, a pitch angle, and a roll angle (S730) representing eye height, a user position, and a head / body orientation among the plurality of parameters (S760). For example, in the case of a sitting posture, the user body may be fixed on a chair, while the user's head may be freely rotated, and thus the processor 140 may determine the posture of the display 130 by considering the head orientation and the body orientation.

[0141] Alternatively, if the user is identified as having a second posture, the processor 140 may determine the posture of the display 130 based on at least one second parameter among the plurality of parameters (S760). For example, if the user is identified as having a standing posture, the processor 140 may determine the posture of the display 130 based on the yaw angle (S740) indicating the eye height, the user position, or the body orientation among the plurality of parameters (S760). For example, in the case of a standing posture, the user's body can move freely. Therefore, under the premise of obtaining the most natural posture, the processor 140 may determine the posture of the display 130 by considering only the yaw angle indicating the body orientation.

[0142] Alternatively, if the user is identified as having a third posture, the processor 140 may determine the posture of the display 130 based on at least one third parameter among the plurality of parameters (S760). For example, if the user is identified as having a lying posture, the processor 140 may determine the posture of the display 130 based on the yaw angle and the roll angle (S750) representing the eye height, the user position, and the body orientation among the plurality of parameters (S760). For example, in the case of a lying posture, the user's head cannot move freely. Therefore, the processor 140 may determine the posture of the display 130 by considering the yaw angle and the roll angle representing the body orientation.

[0143] That is, a parameter for recognizing a user gesture and a parameter for determining a gesture of the display 130 after recognizing the user gesture may be different from each other.

[0144] However, the present disclosure is not limited thereto, and various methods of determining the user posture and the posture of the display 130 may be provided. For example, the electronic device 100 may include a plurality of microphones, and the processor 140 may recognize the user's mouth direction through the plurality of microphones. For example, if a user voice calling the electronic device 100 is received through the plurality of microphones, the processor 140 may recognize the user's mouth direction through the plurality of microphones, recognize the user posture as one of a sitting posture, a standing posture, and a lying posture based on the user's mouth direction, change the position of the electronic device 100 based on the user position, and change the posture of the display 130 based on the recognized user posture.

[0145] Figure 8 is a view illustrating a method of determining a position of the electronic device 100 according to a user gesture according to one or more embodiments of the present disclosure.

[0146] The processor 140 may then change the position of the electronic device 100 based on the user's posture. For example, the processor 140 may change the position of the electronic device 100 to separate the electronic device 100 from the user by a reference distance. Here, the reference distance may be a constant value regardless of the user's posture. To this end, the processor 140 may correct the distance between the user's head and the electronic device 100 based on the inclination of the user's upper body. For example, Figure 8 As shown in the upper side of the figure, the upper body of the user may not be tilted. The processor 140 may change the position of the electronic device 100 based on the head of the user to keep the distance between the head of the user and the electronic device 100 at a predetermined reference distance. Figure 8 As shown in the lower side of , the upper body of the user may be tilted. The processor 140 may correct the distance between the head of the user and the electronic device 100 based on the tilt of the upper body of the user.

[0147] However, the present disclosure is not limited thereto, and the processor 140 may change the position of the electronic device 100 based on a point other than the user's head.

[0148] Alternatively, the processor 140 may change the position of the electronic device 100 based on the user's eye position, regardless of the tilt of the user's upper body. The electronic device 100 may be spaced apart from the user's eyes by a predetermined distance.

[0149] The camera 110 may be disposed on one side of the display 130, and the processor 140 may change the display 130 so that the user or a predetermined user body part is included in a predetermined area in the captured image. Figure 8 , the camera 110 may be arranged to capture the direction of light emitted from the display 139, and the portion indicated by the triangular hatching may represent the field of view (FoV) of the camera 110. The processor 140 may change the posture of the display 130 based on the user posture so that the user's head can be located within the FoV even if the posture of the display 130 changes. That is, the change of the posture of the display 130 may have a limitation that the user's head is required to be within the FoV.

[0150] Fig. 9 is a view illustrating roll angle control of the display 130 according to one or more embodiments of the present disclosure.

[0151] The processor 140 may control the roll angle of the display 130 based on the user's head orientation. For example, Fig. 9Each roll angle of the display 130 in the case where a line connecting both eyes of the user in the horizontal direction has zero degrees, 45 degrees, -45 degrees, and 90 degrees is shown.

[0152] The processor 140 may identify the head orientation of the user based on both eyes of the user. If the head orientation of the user changes, the processor 140 may change the roll angle of the display 130 based on the changed head orientation of the user. The processor 140 may capture the front of the electronic device 100 by using the camera 110 at predetermined time intervals, and identify the change in the head orientation of the user from the images captured at predetermined time intervals.

[0153] Alternatively, if the user's head orientation changes and a predetermined time has passed, the processor 140 may change the roll angle of the display 130 based on the changed user's head orientation. For example, if the user's head orientation changes and 10 seconds have passed, the processor 140 may change the roll angle of the display 130 based on the changed user's head orientation. If the roll angle of the display 130 changes continuously without delay based on the user's head orientation, the user's viewing may be disturbed, which can be solved by a predetermined time condition.

[0154] Alternatively, if the change in the user's head orientation is greater than or equal to a predetermined amount, the processor 140 may change the roll angle of the display 130 based on the changed head orientation of the user. It is impossible for the user's head not to move, and the roll angle of the display 130 may change due to a small movement that is unrelated to the user's intention. Here, the user's viewing may be disturbed, which can be solved by the condition that the change in the user's head orientation is greater than or equal to a predetermined amount.

[0155] Alternatively, if the change in the user's head orientation is greater than or equal to a predetermined amount and a predetermined time has elapsed, processor 140 may change the roll angle of display 130 based on the changed user's head orientation.

[0156] Fig.10 and Fig.11 is a diagram illustrating a prediction operation according to one or more embodiments of the present disclosure.

[0157] The processor 140 may change the posture of the display 130 based on the user's posture. Fig.10 As shown, if the user's posture changes from 1010 posture through 1020 posture to 1030 posture, the processor 140 may change the posture of the display 130 to correspond to each posture.

[0158] Here, the display posture driver 190 that changes the posture of the display 130 may include at least one motor, and a delay may occur due to its mechanical operation. In addition, a delay may also occur in the operation of the processor recognizing that the posture of the display 130 is changed based on the user's posture. As a result, a delay occurs due to the change of the user's posture, and thus the change of the posture of the display 130 in response to the change of the user's posture may be delayed.

[0159] However, even if a change in the posture of the display 130 requires a certain amount of delay to reflect the user's intention, if the change in the user's posture has a certain regularity, such a change can be reflected in advance.

[0160] To this end, the processor 140 may acquire multiple images by capturing images at predetermined time intervals by the camera 110, acquire a predicted user posture at a predetermined time point from a current time point based on the multiple images, and change the posture of the display 130 based on the user posture and the predicted user posture.

[0161] For example, Fig.11 As shown, in the case where the user posture changes from the posture 1110 to the posture 1140 through the posture 1120 and the posture 1130, the processor 140 may predict the change of the posture 1140 at a specific time point in the process of changing from the posture 1110 to the posture 1140, for example, at the time point of the posture 1130. Therefore, even at the time point of the posture 1130, the processor 140 may change the yaw angle of the display 130 based on the posture 1140.

[0162] Fig.12 and Fig.13 is a diagram illustrating operations based on content types according to one or more embodiments of the present disclosure.

[0163] The processor 140 may change the posture of the display 130 based on the user command. For example, the user may Fig.12 When viewing content in the posture of the display 130 shown on the left, a user command "full screen display" is input. The processor 140 may change the posture of the display 130, such as Fig.12 The processor 140 may store data about the content type or data about the application providing the content in the memory 170 at the time when the user command is input. Fig.12 The gesture shown on the left side of the display 130 plays the same content as the content type stored in the memory 170 or the content provided by the application stored in the memory 170, and the processor 140 can automatically change the gesture of the display 130, such as Fig.12 as shown on the right.

[0164] Optionally, if the user Fig.13When viewing content as shown on the left side of FIG. 1 , the processor 140 receives a user command such as “connect me to a video call”, and the processor 140 may change the posture of the display 130, such as Fig.13 as shown on the right.

[0165] Fig.14 is a diagram illustrating an adaptive operation to a user according to one or more embodiments of the present disclosure.

[0166] The processor 140 may change the posture of the display 130 (or may perform automatic view adjustment 1410 ) based on the user posture.

[0167] The processor 140 may then change the posture of the display 130 based on at least one of the user gesture 1420-1 or the user voice command 1420-2. The processor 140 may learn at least one of the user gesture 1420-1 or the user voice command 1420-2 and the changed posture of the display 130 (1440), and then use it for the automatic view adjustment operation.

[0168] Alternatively, if the posture of the display 130 is maintained for a certain time or longer without a change in the user posture ( 1430 ), the processor 140 may learn the user posture and the posture of the display 130 ( 1440 ) and then use them for the automatic view adjustment operation.

[0169] Fig.15 is a flowchart illustrating a control method of an electronic device according to one or more embodiments of the present disclosure.

[0170] First, the method may include capturing an image (S1510). In addition, the method may include recognizing a user gesture included in the captured image (S1520). In addition, the method may include recognizing at least one parameter among a plurality of parameters related to the user based on the user gesture (S1530). In addition, the method may include changing a position of an electronic device based on the at least one parameter, and changing a gesture of a display included in the electronic device (S1540).

[0171] In addition, the method may also include: if a user command to call the electronic device is received from another electronic device, or a user voice to call the electronic device is received through a microphone included in the electronic device, moving toward the user based on the user command or the user voice, wherein, when capturing, multiple images can be acquired by opening a camera included in the electronic device to capture images at predetermined time intervals.

[0172] Furthermore, when changing the display posture (S1540), the user posture and the distance of the display to the user may be recognized based on the plurality of images, and if the distance to the user is a distance corresponding to the user posture, the change of the position of the electronic device may be stopped.

[0173] In addition, when recognizing a user posture (S1520), multiple parameters related to the user can be recognized based on the captured image, and the user posture can be recognized based on the multiple parameters; and when changing the display posture (S1540), if the user posture is a first posture, the posture of the display can be changed based on at least one first parameter of the multiple parameters, and if the user posture is a second posture, the posture of the display can be changed based on at least one second parameter of the multiple parameters.

[0174] In addition, the method may further include displaying the content, and when changing the display posture ( S1540 ), the posture of the display may be changed based on the content type and the user posture.

[0175] Furthermore, in changing the display posture (S1540), if the user posture is changed and a predetermined time has passed, the posture of the display may be changed based on the changed user posture.

[0176] In addition, when capturing images (S1510), multiple images can be obtained by capturing images at predetermined time intervals, and when recognizing a user posture (S1520), a predicted user posture at a predetermined time point can be obtained from a current time point based on the multiple images, and when changing a display posture (S1540), the posture of the display can be changed based on the user posture and the predicted user posture.

[0177] In addition, when recognizing a user gesture (S1520), the user gesture may be recognized by inputting the captured image into a first neural network model.

[0178] Furthermore, when changing the display posture (S1540), data on the display posture may be acquired by inputting the plurality of parameters into the second neural network model, and the posture of the display may be changed based on the data on the display posture.

[0179] In addition, the method may further include: if a user command for changing the posture of the display is received, changing the posture of the display based on the user command, and updating the second neural network model based on the multiple parameters and the changed posture of the display.

[0180] In addition, when capturing an image (S1510), a plurality of images may be acquired by capturing images at predetermined time intervals, and when changing the display posture (S1540), if a predetermined user gesture is recognized from the plurality of images or a user voice for changing the display posture is received, the posture of the display may be changed based on the user gesture or the user voice.

[0181] Furthermore, a camera included in the electronic device may be disposed at one side of the display, and when changing the display posture (S1540), the posture of the display may be changed so that the user or a predetermined user body is included in a predetermined area in the captured image.

[0182] According to various embodiments of the present disclosure as described above, the electronic device can improve the convenience of the user by providing a screen with a posture optimized for the user.

[0183] Furthermore, the electronic device may perform adaptive operations on the user by learning the user's gestures and the display's gestures.

[0184] According to one or more embodiments of the present disclosure, the various embodiments described above may be implemented in software including instructions stored in a machine-readable storage medium (e.g., a computer-readable storage medium). A machine may be a device that calls stored instructions from a storage medium and operates based on the called instructions, and may include an electronic device (e.g., electronic device A) in the disclosed embodiments. If the instructions are executed by a processor, the processor may directly execute the functions corresponding to the instructions or other components may execute the functions corresponding to the instructions under the control of the processor. The instructions may include code provided or executed by a compiler or interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, the term "non-transitory" indicates that the storage medium is tangible, does not include signals, and does not distinguish whether the data is stored semi-permanently or temporarily in the storage medium.

[0185] In addition, according to one or more embodiments of the present disclosure, the above methods according to various embodiments may be provided by being included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., a compact disc read-only memory (CD-ROM)), or distributed online through an application store (e.g., PlayStoreTM). In the case of online distribution, at least a portion of the computer program product may be at least temporarily stored or temporarily provided in a storage medium, such as a memory of a manufacturer's server, a server of an application store, or a relay server.

[0186] In addition, according to one or more embodiments of the present disclosure, the various embodiments described above can be implemented in a computer or a computer-readable recording medium using software, hardware, or a combination of software and hardware. In some cases, one or more embodiments described in the present disclosure can be implemented by the processor itself. According to software implementation, one or more embodiments such as the processes and functions described in the present disclosure can be implemented by separate software modules. Each software module can perform one or more functions and operations described in the present disclosure.

[0187] Computer instructions for executing the processing operations of the machine according to the various embodiments described above may be stored in a non-transitory computer-readable medium. If the instructions are executed by a processor of a specific machine, the computer instructions stored in the non-transitory computer-readable medium allow the specific machine to execute the processing operations in the machine according to the various embodiments described above. A non-transitory computer-readable medium is not a medium in which data is stored for a period of time, such as a register, a cache, or a memory, but indicates a medium in which data is semi-permanently stored and machine-readable. Specific examples of non-transitory computer-readable media may include a compact disc (CD), a digital versatile disc (DVD), a hard disk, a Blu-ray disc, a universal serial bus (USB), a memory card, a read-only memory (ROM), and the like.

[0188] In addition, each component (e.g., module or program) according to the various embodiments described above may include a single entity or multiple entities, and some of the corresponding subcomponents described above may be omitted, or other subcomponents may be included in various embodiments. Alternatively or additionally, some components (e.g., modules or programs) may be integrated into one entity, and the functions performed by each corresponding component before being integrated may be performed in the same or similar manner. The operations performed by the modules, programs or other components according to various embodiments may be performed in a sequential manner, in a parallel manner, in an iterative manner or in a heuristic manner, and at least some operations may be performed in a different order or omitted, or other operations may be added.

[0189] In addition, although the embodiments are shown and described in the present disclosure as described above, the present disclosure is not limited to the above embodiments, and a person skilled in the art to which the present disclosure belongs may make various modifications without departing from the gist of the present disclosure disclosed in the appended claims. These modifications should also be understood to be within the scope and spirit of the present disclosure.

Claims

1. An electronic device comprising: camera; Driver; monitor; as well as at least one processor operably connected to the camera, the driver, and the display and configured to: controlling the camera to capture an image, recognizing a user gesture included in the captured image, identifying at least one parameter among a plurality of parameters associated with the user based on the user gesture, and The driver is controlled based on the at least one parameter to change the position of the electronic device and change the posture of the display.

2. The electronic device according to claim 1, further comprising: Communication interface; as well as microphone, Wherein, the at least one processor is further configured to: controlling the driver to move the electronic device toward the user based on a user command received from another electronic device through the communication interface or a user command received through the microphone, and The camera is controlled to capture a plurality of images at predetermined time intervals.

3. The electronic device according to claim 2, wherein: The at least one processor is further configured to: recognizing the user gesture and a distance from the electronic device to the user based on the plurality of images, and Based on the distance from the electronic device to the user corresponding to the user gesture, the operation of the driver is stopped.

4. The electronic device according to claim 1, wherein: The at least one processor is further configured to: identifying the plurality of parameters associated with the user based on the captured image, recognizing the user gesture based on the plurality of parameters, Based on the user gesture being a first gesture, changing a gesture of the display based on at least one first parameter of the plurality of parameters, and Based on the user gesture being a second gesture, changing a gesture of the display based on at least one second parameter of the plurality of parameters.

5. The electronic device according to claim 1, wherein: The at least one processor is further configured to: controlling the display to display content, and The gesture of the display is changed based on the content type and the user gesture.

6. The electronic device according to claim 1, wherein: The at least one processor is further configured to, based on the user posture changing and a predetermined time elapsed, change a posture of the display based on the changed user posture.

7. The electronic device according to claim 1, wherein: The at least one processor is further configured to: controlling the camera to capture a plurality of images at predetermined time intervals, obtaining a predicted user gesture at a predetermined time point from a current time point based on the plurality of images, and A posture of the display is changed based on the user posture and the predicted user posture.

8. The electronic device according to claim 1, further comprising: a memory configured to store a first neural network model, Wherein, the at least one processor is further configured to: recognize the user gesture by inputting the captured image into the first neural network model.

9. The electronic device according to claim 1, further comprising: a memory configured to store a second neural network model, Wherein, the at least one processor is further configured to: acquiring data about the posture of the display by inputting the plurality of parameters into the second neural network model, and The posture of the display is changed based on data regarding the posture of the display.

10. The electronic device according to claim 9, further comprising: User interface, Wherein, the at least one processor is further configured to: based on receiving a user command for changing the posture of the display through the user interface, changing the posture of the display based on the user command, and The second neural network model is updated based on the multiple parameters and the changed posture of the display.

11. The electronic device according to claim 1, further comprising: microphone, Wherein, the at least one processor is further configured to: controlling the camera to capture a plurality of images at predetermined time intervals, and The posture of the display is changed based on a user gesture recognized from the plurality of images, or based on a user voice received through the microphone for changing the posture of the display.

12. The electronic device according to claim 1, wherein: The camera is disposed on one side of the display, and The at least one processor is further configured to: change the posture of the display so that the user or a predetermined user body part is included in a predetermined area in the captured image.

13. The electronic device according to claim 1, further comprising: Display gesture driver, Wherein, the at least one processor is further configured to: control the display posture driver to change the posture of the display.

14. The electronic device according to claim 1, wherein: The at least one processor is further configured to control the driver to change the position of the electronic device based on the movement of the user gesture.

15. A control method for an electronic device, the control method comprising: Capture images; recognizing a user gesture included in the captured image; identifying at least one parameter among a plurality of parameters associated with the user based on the user gesture; as well as The position of the electronic device is changed based on the at least one parameter, and a posture of a display included in the electronic device is changed.

Images