Electronic device including projection unit for projecting video and providing video call function and control method thereof

Abstract

An electronic device for providing a video call function and a control method thereof are provided. The electronic device of the present disclosure comprises: a projection unit; a camera; a sensor for acquiring information on a distance to an external object; a memory to store at least one instruction; and at least one processor connected to the projection unit, the camera, the sensor, and the memory, and for controlling the electronic device. The at least one processor executes at least one instruction such that, if the video call function is performed, a user is captured by the camera and thus a first user video is acquired. The at least one processor calculates a horizontal correction angle and a vertical correction angle based on the first user video, a rotation angle of the camera, a projection angle of the projection unit, and a sensing value acquired by the sensor. The at least one processor acquires a front video of the user based on the first user video, the horizontal correction angle, and the vertical correction angle. The at least one processor controls the projection unit such that a video call screen including the acquired front video of the user is projected.

Description

Technical Field

[0001] The present disclosure relates to an electronic device including a projection unit for projecting a video and a control method thereof, and more particularly, to an electronic device for providing a video call function through the projection unit and a control method thereof. Background Art

[0002] In recent years, with the development of communication technology, a video call function that can perform a video call with an external terminal device has become common. In particular, the video call function is provided not only by user terminals (such as smartphones) but also by various electronic devices (such as TVs in homes).

[0003] Based on performing a video call with an external terminal device, the electronic device may use a camera to capture a user video and use the captured user video to perform a video call. However, in the case of an electronic device such as a projector, the camera is not located in the direction the user is looking at, but is located to the side or behind the user. Therefore, when performing a video call using an electronic device such as a projector, the user's face is captured from the side rather than from the front, so there is a limitation that the other party may not be able to perform a video call when looking at the user's face. Summary of the invention

[0004] According to one aspect of the present disclosure, an electronic device for providing a video call function includes: a projection unit; a camera; a sensor configured to obtain information about the distance to an external object; a memory configured to store at least one instruction; and at least one processor configured to be connected to the projection unit, the camera, the sensor, and the memory, and to control the electronic device. At least one processor is configured to run at least one instruction to obtain a first user video by capturing the user through the camera based on the video call function being executed. At least one processor calculates a horizontal correction angle and a vertical correction angle based on the first user video, the rotation angle of the camera, the projection angle of the projection unit, and the sensed value obtained by the sensor. At least one processor obtains a frontal video of the user based on the first user video, the horizontal correction angle, and the vertical correction angle. At least one processor controls the projection unit so that a video call screen including the obtained frontal video of the user is projected.

[0005] According to another aspect of the present disclosure, a control method for an electronic device for providing a video call function, the electronic device including a projection unit for projecting a video, the control method including: based on the video call function being executed, capturing the user by a camera to obtain a first user video; calculating a horizontal correction angle and a vertical correction angle based on the first user video, a rotation angle of the camera, a projection angle of the projection unit, and a sensing value obtained by a sensor of the electronic device; obtaining a front video of the user based on the first user video, the horizontal correction angle, and the vertical correction angle; and projecting a video call screen including the obtained front video of the user by the projection unit.

[0006] According to another aspect of the present disclosure, a computer-readable medium stores a program for executing a control method for an electronic device for providing a video call function, the electronic device including a projection unit for projecting a video, wherein the control method includes: based on the video call function being executed, capturing the user by a camera to obtain a first user video; calculating a horizontal correction angle and a vertical correction angle based on the first user video, a rotation angle of the camera, a projection angle of the projection unit, and a sensing value obtained by a sensor of the electronic device; obtaining a front video of the user based on the first user video, the horizontal correction angle, and the vertical correction angle; and projecting a video call screen including the obtained front video of the user by the projection unit. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1 is a diagram for describing an electronic device for providing a video call function according to an embodiment of the present disclosure.

[0008] Figure 2a is a block diagram illustrating a configuration of an electronic device according to an embodiment of the present disclosure.

[0009] Figure 2b is a diagram illustrating an appearance of an electronic device according to an embodiment of the present disclosure.

[0010] Figure 3 is a diagram for describing a method for correcting a user video captured by an electronic device while performing a video call according to an embodiment of the present disclosure.

[0011] Figure 4 is a block diagram illustrating a configuration for correcting a user video while performing a video call according to an embodiment of the present disclosure.

[0012] Figure 5a is a diagram for describing a method for calculating a first height from a floor to a center of a projection surface according to an embodiment of the present disclosure.

[0013] Figure 5bis a diagram for describing a method for calculating a second height from a floor to a center of a user's face according to an embodiment of the present disclosure.

[0014] Figure 6 is a diagram for describing a method for calculating a horizontal correction angle according to an embodiment of the present disclosure.

[0015] Figure 7 is a diagram for describing a method for calculating a vertical correction angle according to an embodiment of the present disclosure.

[0016] Figure 8 is a diagram for describing a method for obtaining a frontal video using a horizontal correction angle and a vertical correction angle according to an embodiment of the present disclosure.

[0017] Fig. 9 is a diagram for describing a method for providing an avatar video according to an embodiment of the present disclosure.

[0018] Fig.10a and Fig.10b is a diagram for describing a method for moving an electronic device by identifying whether the electronic device is located in a space where a video call can be performed according to an embodiment of the present disclosure.

[0019] Fig.11 is a flowchart for describing a control method of an electronic device for providing a video call function according to an embodiment of the present disclosure.

[0020] Fig.12 is a block diagram illustrating in detail the configuration of an electronic device according to an embodiment of the present disclosure.

[0021] Fig.13 is a perspective view showing the appearance of an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0022] In the following, various embodiments of the present disclosure are described. However, this is not intended to limit the technology of the present disclosure to specific embodiments, and various modified embodiments of the embodiments of the present disclosure may be variously transformed and may have multiple embodiments, and specific embodiments are shown in the drawings and described in detail in the detailed description. However, it should be understood that the present disclosure is not limited to specific embodiments, but includes various modifications, equivalent forms and / or alternative forms according to the embodiments of the present disclosure. Throughout the drawings, similar components will be represented by similar reference numerals.

[0023] In describing the present disclosure, when it is determined that a detailed description of a known function or configuration related to the present disclosure may unnecessarily obscure the gist of the present disclosure, the detailed description thereof will be omitted.

[0024] In addition, the following embodiments may be modified in a variety of different forms, and the scope and spirit of the present disclosure are not limited to the following embodiments. On the contrary, these embodiments make the present disclosure comprehensive and complete, and are provided to fully convey the technical spirit of the present disclosure to those skilled in the art.

[0025] The terms used in the present disclosure are only used to describe specific embodiments rather than to limit the scope of the present disclosure. Unless the context clearly indicates otherwise, the singular includes the plural.

[0026] In the present specification, expressions such as “having”, “may have”, “including”, “may include”, etc. indicate the existence of corresponding features (for example, numerical values, functions, operations, components (such as parts), etc.), and do not exclude the existence of additional features.

[0027] In the present disclosure, the expression "A or B", "at least one of A and / or B", or "one or more of A and / or B" may include all possible combinations of the items listed together. For example, "A or B", "at least one of A and B", or "at least one of A or B" may indicate all of the following: 1) a case where at least one A is included, 2) a case where at least one B is included, or 3) a case where both at least one A and at least one B are included.

[0028] The expressions “first” or “second” used in the present disclosure may indicate various components regardless of the order and / or importance of the components, and these expressions will only be used to distinguish one component from other components, and will not limit the corresponding components.

[0029] When it is mentioned that any component (e.g., a first component) is coupled (operably or communicatively) with / coupled to another component (e.g., a second component) or connected to another component (e.g., a second component), it will be understood that any component is directly coupled to another component or may be coupled to another component through other components (e.g., a third component).

[0030] On the other hand, when it is mentioned that any component (eg, a first component) is “directly coupled to” or “directly connected to” another component (eg, a second component), it will be understood that there is no other component (eg, a third component) between the any component and the other component.

[0031] The expression "configured (or set) to" used in the present disclosure may be replaced by "suitable for", "capable of", "designed to", "adapted to", "manufactured to" or "capable of" according to circumstances. The term "configured (or set) to" may not necessarily mean "specially designed to" in hardware.

[0032] On the contrary, in some cases, the expression "a device configured to..." may mean that the device may be "capable of..." together with other devices or components. For example, "a processor configured (or set) to perform A, B, and C" may mean a dedicated processor (e.g., an embedded processor) for performing the corresponding operations or a general-purpose processor (e.g., a central processing unit (CPU) or an application processor) that can perform the corresponding operations by running one or more software programs stored in a storage device.

[0033] In the embodiments, a "module" or "device" may perform at least one function or operation and may be implemented by hardware or software, or by a combination of hardware and software. In addition, multiple "modules" or multiple "devices" may be integrated into at least one module and implemented by at least one processor, except for "modules" or "devices" that need to be implemented by specific hardware.

[0034] In addition, various elements and regions in the drawings are schematically shown. Therefore, the spirit of the present disclosure is not limited by the relative sizes or intervals shown in the drawings.

[0035] Figure 1 is a diagram for describing an electronic device for providing a video call function according to an embodiment of the present disclosure. Figure 1 The electronic device 100 shown in FIG. 1 may be a video projection device, such as a projector that projects a video onto a wall or a screen.

[0036] According to an embodiment of the present disclosure, based on the video call function being executed, the electronic device 100 can capture the user 10 through a 360-degree rotatable camera to obtain a first user video. In this case, the first user video can be a video captured from the side of the user 10 based on the user 10 looking at the projection surface (especially, the other party) 50.

[0037] In addition, the electronic device 100 can use the first user video to obtain a frontal video of the user 10 looking to the front. Specifically, the electronic device 100 can calculate the horizontal correction angle and the vertical correction angle based on the rotation angle of the camera, the projection angle of the projection unit, and the sensing value obtained by the sensor. In this case, the horizontal correction angle is an angle for correcting the first user video by rotating the first user video in the horizontal direction, and the vertical correction angle is an angle for correcting the second user video by rotating the second user video in the vertical direction. The electronic device 100 can obtain a frontal video of the user 10 based on the first user video, the horizontal correction angle, and the vertical correction angle.

[0038] like Figure 1As shown, the electronic device 100 may project a video call screen including the obtained front video of the user 10 onto the projection surface 50. Therefore, even if the electronic device 100 captures the user 10 from the side direction of the user 10, the user 10 may provide a video call using the front video, thereby providing a more natural video call screen.

[0039] In addition, the electronic device 100 may capture a second user video while performing a video call. In addition, the electronic device 100 may analyze the second user video to correct the frontal video so as to reflect the natural movement of the user 10 (e.g., head rotation, lip movement, etc.) in the frontal video. Specifically, the electronic device 100 may analyze feature points (e.g., eyes, nose, lips, etc.) of the second user video to obtain information about the rotation angle of the user's face or the movement of the feature points, thereby correcting the frontal video.

[0040] Therefore, the electronic device 100 may provide a natural user video by correcting the front video to reflect the movement of the user while performing a video call.

[0041] Figure 2a is a block diagram showing a configuration of an electronic device according to an embodiment of the present disclosure. Figure 2a As shown, the electronic device 100 may include a projection unit 112, a memory 113, a sensor unit 121, a camera 122, and at least one processor 111. Figure 2a The configuration of the electronic device 100 shown in FIG. 1 is merely an embodiment, and other configurations may be added or some configurations may be omitted.

[0042] The projection unit 112 is a configuration for projecting light to the outside to express a video. The projection unit 112 may include various detailed components such as a light source, a projection lens, and a reflector. The operation method and detailed configuration of the projection unit 112 will be described in detail again in the following section. In particular, when the electronic device 100 makes a video call with an external device, the projection unit 112 can project the video call screen onto the projection surface.

[0043] The memory 113 may store at least one instruction for controlling the electronic device 100. Specifically, the memory 120 may store data required by a module for correcting a user video while performing a video call to perform various operations. Figure 4 As shown, the module for correcting the user video while performing a video call may include a distance information acquisition module 410, a height information acquisition module 420, a correction angle calculation module 430, a front video acquisition module 440, a video analysis module 450 and a video correction module 460.

[0044] In addition, the memory 113 may include a nonvolatile memory that can retain stored information even if power supply is interrupted and a volatile memory that requires continuous power supply to retain stored information. A module for correcting a user video while performing a video call may be stored in the nonvolatile memory.

[0045] The sensor unit 121 is configured to obtain state information of the electronic device 100 or information about the surrounding environment of the electronic device 100. Specifically, the sensor unit 121 may include at least one distance sensor for obtaining information about the distance between the electronic device 100 and an object. In this case, the at least one distance sensor may be located around the camera 122 or around the projection unit 112, but is not limited thereto.

[0046] At least one distance sensor may obtain a sensing value for obtaining information about a distance between the electronic device 100 and the center of the projection surface or a sensing value for obtaining information about a distance between the electronic device 100 and the center of the face of the user 10 .

[0047] The camera 122 may be disposed on one side surface of the electronic device 100 to capture still video or moving video. Figure 2b As shown, the camera 122 may be disposed on a side surface of the cylindrical electronic device 100. In addition, the camera 122 may also rotate 360 ​​degrees. Therefore, the camera 122 may capture a user located around the electronic device 100.

[0048] At least one processor 111 may be electrically connected to the memory 113 to control the overall functions and operations of the electronic device 100. Based on the detection of an event of performing a video call or an event of correcting a user video while performing a video call, the at least one processor 111 may load data stored in a non-volatile memory for performing various operations for a module for correcting a user image while performing a video call into a volatile memory to perform various operations. The at least one processor 111 may perform various operations by using various modules based on the data loaded into the volatile memory. Here, loading means an operation of loading and storing data stored in a non-volatile memory into a volatile memory so that the at least one processor 111 can access it.

[0049] In particular, at least one processor 111 executes at least one instruction stored in the memory 113, and therefore, based on the video call function being executed, the at least one processor 111 captures the user through the camera 122 to obtain a first user video. The at least one processor 111 calculates a horizontal correction angle and a vertical correction angle based on the first user video, the rotation angle of the camera 122, the projection angle of the projection unit 112, and the sensed value obtained by the sensor unit 121. The at least one processor 111 obtains a frontal video of the user based on the first user video, the horizontal correction angle, and the vertical correction angle. The at least one processor 111 controls the projection unit 112 to project a video call screen including the obtained frontal video of the user.

[0050] In particular, the at least one processor 111 may obtain information about a first distance between the electronic device 100 and the center of the projection surface based on a first sensing value obtained by the sensor unit 121. In particular, the at least one processor 111 may obtain information about a second distance between the electronic device 100 and the user's face based on a second sensing value obtained by the sensor unit 121. In addition, the at least one processor 111 may calculate a horizontal correction angle based on the information about the first distance, the information about the second distance, and the first rotation angle of the camera 122. In this case, the first rotation angle of the camera 122 is an angle at which the camera 122 rotates left and right relative to the center of the projection surface.

[0051] In addition, the at least one processor 111 may obtain information about a first height from the floor to the center of the projection surface based on the projection angle of the projection unit 112 and the second sensing value obtained by the sensor unit 121. The at least one processor 111 may obtain information about a second height from the floor to the user's face based on the second rotation angle of the camera 122 and the second sensing value obtained by the sensor unit 121. In this case, the second rotation angle of the camera 122 is an angle at which the camera 122 rotates up and down relative to the floor. The at least one processor 111 may obtain information about a third distance between the user's face and the center of the projection surface based on the information about the first distance, the information about the second distance, and the second rotation angle of the camera. The at least one processor 111 may calculate a vertical correction angle based on the information about the first height, the information about the second height, and the information about the third distance.

[0052] In addition, the at least one processor 111 may rotate the first user video in a first direction based on the horizontal correction angle, and rotate the first user video in a second direction based on the vertical correction angle, thereby obtaining a frontal video of the user.

[0053] In addition, the at least one processor 111 may obtain a frontal video of the user by inputting the first user video, the horizontal correction angle, and the vertical correction angle into the trained neural network model.

[0054] In addition, the at least one processor 111 may generate an avatar video based on the front video of the user, and control the projection unit 112 to project the generated avatar video onto the video call screen.

[0055] In addition, the at least one processor 111 may identify whether the electronic device 100 is located in a space where a video call can be made based on at least one of the first user video, the rotation angle of the camera, the projection angle of the projection unit, and the sensing value obtained by the sensor unit 121. In addition, based on the electronic device 100 not being located in the space where a video call can be made, the at least one processor 111 may control the driving unit 120 to move to the space where a video call can be made.

[0056] In addition, the at least one processor 111 may obtain a second user video using the camera 122 while performing a video call. In addition, the at least one processor 111 may calculate a rotation angle of the user's face based on the second user video. The at least one processor 111 may rotate the user's frontal video based on the calculated rotation angle of the user's face.

[0057] Furthermore, based on the inclusion of a plurality of users in the first user video, the at least one processor 111 may identify a speaker among the plurality of users and calculate a horizontal correction angle and a vertical correction angle based on the identified speaker.

[0058] In addition, the electronic device 100 may further include Figure 2b The driving unit 120 is shown. The driving unit 120 is a component for moving the electronic device 100 and may include a motor, wheels, etc. In addition, the electronic device 100 can adjust the projection angle of the projection unit 112 by rotating the body of the electronic device 100, and the rotation angle of the camera 122 arranged on the body of the electronic device 100 can be adjusted 360 degrees.

[0059] Figure 3 is a diagram for describing a method for correcting a user video captured by an electronic device while performing a video call according to an embodiment of the present disclosure.

[0060] First, the electronic device 100 may execute a video call function (S310). Specifically, based on receiving a video call request from the outside or inputting a user command to run a video call application, the electronic device 100 may execute the video call function.

[0061] The electronic device 100 may capture the user through the camera 122 to obtain a first user video (S320). In this case, the first user video is a video captured based on the user looking at the projection screen (the other party of the video call screen). For example, the electronic device 100 may capture the user based on a guide message (e.g., please look forward) to look at the projection surface being output before starting the video call, thereby obtaining the first user video. For example, the electronic device 100 may detect a greeting message (e.g., hello) from the other party of the video call, and capture the user based on the greeting message being executed, thereby obtaining the first user video.

[0062] The electronic device 100 may obtain a frontal video of the user (S330). In detail, the electronic device 100 may calculate a horizontal correction angle and a vertical correction angle based on the first user video, the rotation angle of the camera 122, the projection angle of the projection unit 112, and the sensed value obtained by the sensor unit 121. The electronic device 100 may obtain a frontal video of the user based on the first user video, the horizontal correction angle, and the vertical correction angle. This will be described below with reference to Figures 4 to 8 The electronic device 100 may control the projection unit 112 to project a video call screen including the obtained front video of the user.

[0063] The electronic device 100 may obtain a second user video while performing the video call (S340).

[0064] The electronic device 100 may analyze the second user video (S350). Specifically, the electronic device 100 may obtain information about the user's facial rotation, lip movement, and eye movement by analyzing the movement of the user's feature points (eg, eyes, nose, lips, etc.) included in the second user video.

[0065] The electronic device 100 may correct the front video of the user based on the analysis result (S360). Specifically, the electronic device 100 may rotate the front video of the user based on the information about the user's facial rotation, and correct the front video of the user according to the change of the user's face based on the information about the lip movement or the eye movement.

[0066] As described above, through the video correction process, the electronic device 100 can perform a video call with an external user through a front video even if the user is not captured from the front.

[0067] In the following, reference will be made to Figures 4 to 8 A method for correcting a user video while performing a video call is described in more detail. Figure 4 is a block diagram showing a configuration for correcting a user's video while performing a video call according to an embodiment of the present disclosure. Figure 4As shown, the electronic device 100 may include a distance information acquisition module 410 , a height information acquisition module 420 , a correction angle calculation module 430 , a front video acquisition module 440 , a video analysis module 450 and a video correction module 460 .

[0068] In addition, the electronic device 100 may obtain a first user video. In this case, the first user video may be a video captured from the side surface of the user 10 based on the user 10 looking at the projection surface (particularly, the other party) 50. In particular, based on the electronic device 100 capturing the first user video, the camera 122 may rotate so that the face of the user 10 is located at the center of the first user video.

[0069] The distance information acquisition module 410 may obtain distance information between the electronic device 100 and the object based on a sensing value obtained from at least one distance sensor included in the sensor unit 121. Specifically, the distance information acquisition module 410 may obtain information about a first distance a between the electronic device 100 and the center of the projection surface based on a first sensing value obtained by the at least one distance sensor. 1 In addition, the distance information acquisition module 410 may obtain information about the second distance a between the electronic device 100 and the user's face based on the second sensing value obtained by at least one distance sensor. 2 information.

[0070] The height information acquisition module 420 may calculate a first height h from the floor to the center of the projection surface. 1 and a second height h from the floor to the user's face 3 Specifically, the height information acquisition module 420 may obtain the projection angle α of the projection unit 112 by the rotation angle of the main body of the electronic device 100 relative to the ground of the electronic device 100. In this case, the projection angle α of the projection unit 112 may be the line segment b having the shortest distance between the projection unit 112 and the projection surface 50. 1 Angle a between the line segment connecting the projection unit 112 and the center of the projection surface 1 The second sensing value obtained by at least one distance sensor may be used to obtain the first distance a 1 In addition, the height information acquisition module 420 may be based on the projection angle α of the projection unit 112 and the first distance a 1 The first height h from the floor to the center of the projection surface 50 is obtained by using the information 1 Information. Figure 5a , the height information acquisition module 420 can calculate c by the following equation 1 1 .

[0071] Equation 1

[0072] In addition, the height h from the floor to the camera 122 of the electronic device may be stored in advance. 2 .

[0073] Therefore, the height information acquisition module 420 can calculate the first height h from the floor to the center of the projection surface 50 by the following equation 2: 1 .

[0074] Equation 2

[0075] In addition, the height information acquisition module 420 may obtain a second rotation angle β of the camera rotating upward relative to the floor, and may obtain a second distance a based on a second sensing value obtained by at least one distance sensor. 2 In addition, the height information acquisition module 420 can be based on the second rotation angle β and the second distance a of the camera. 2 The second height h from the floor to the face of the user 10 is obtained by using the information 3 Information. Figure 5b , the height information acquisition module 420 can calculate c by the following equation 3 2 .

[0076] Equation 3

[0077] In addition, the height h from the floor to the camera 122 of the electronic device may be stored in advance. 2 .

[0078] Therefore, the height information acquisition module 420 can calculate the first height h from the floor to the face of the user 10 by the following equation 4: 1 .

[0079] Equation 4

[0080] The correction angle calculation module 430 may obtain a horizontal correction angle and a vertical correction angle based on the distance information obtained by the distance information acquisition module 410 and the height information obtained by the height information acquisition module 420 .

[0081] Specifically, the correction angle calculation module 430 may obtain the first distance a obtained by the distance information acquisition module 410. 1 and the second distance a 2 And a first rotation angle γ of the camera. In this case, the first rotation angle γ of the camera may represent an angle at which the camera 122 is rotated based on a direction in which the camera 122 faces the center of the projection surface.

[0082] Reference Figure 6 , the correction angle calculation device 430 can calculate the horizontal correction angle A by the following equations 5 and 6. In this case, a 3 The third distance may be between the face of the user 10 and the center of the projection surface 50 .

[0083] Equation 5

[0084] Equation 6

[0085] In addition, the correction angle calculation module 430 can calculate the first height h obtained by the height information acquisition module 420 by using the first height h 1 and the second height h 3 and the third distance a 3 To calculate the vertical correction angle B.

[0086] Reference Figure 7 , the correction angle calculation device 430 can calculate the vertical correction angle B through the following equations 7 and 8.

[0087] Equation 7

[0088] Equation 8

[0089] The front video acquisition module 440 may obtain the front video of the user by using the first user video, the horizontal correction angle A, and the vertical correction angle B. In this case, the front video may be a video captured by a camera located at the center of the projection surface (or the area where the other party is displayed) based on the user looking toward the center of the projection surface (or the other party).

[0090] Specifically, the front video acquisition module 440 may obtain the front video of the user by rotating the first user video in the first direction based on the horizontal correction angle A and rotating the first user video in the second direction based on the vertical correction angle B. As an example, the front video acquisition module 440 may perform 3D modeling on the first user video to obtain a 3D user video. In addition, the front video acquisition module 440 may obtain a 3D front video by rotating the 3D user video in the first direction based on the horizontal correction angle A and rotating the 3D user video in the second direction based on the vertical correction angle B. In addition, the front video acquisition module 440 may obtain a two-dimensional (2D) front video of the user based on the 3D front video. As another example, the front video acquisition module 440 may obtain the front video of the user by rotating the first user video in the first direction based on the horizontal correction angle A and rotating the first user video in the second direction based on the vertical correction angle B. In this case, the front video acquisition module 440 may use a pre-stored user video to generate an uncaptured portion of the rotated front video.

[0091] In addition, the front video acquisition module 440 may rotate the first user video (or 3D user video) based on the direction in which the electronic device 100 captures the user. In particular, the front video acquisition module 440 may rotate the first user video (or 3D user video) in a direction opposite to the direction in which the electronic device 100 captures the user. For example, Figure 8 As shown on the left side of FIG. 1 , based on the electronic device 100 capturing the user in the lower right direction, the front video acquisition module 440 may rotate the first user video 810 to the left by a horizontal correction angle A, and rotate the first user video 810 downward by a vertical correction angle B. Therefore, as Figure 8 As shown on the right side of , the front video acquisition module 440 can obtain the front video 820.

[0092] In addition, the front video acquisition module 440 can obtain the front video of the user by inputting the first user video, the horizontal correction angle A, and the vertical correction angle B into the trained neural network model. In this case, the neural network model can be an artificial intelligence model trained to input the user video, the horizontal correction angle, and the vertical correction angle and output the front video of the user.

[0093] In addition, the front video acquisition module 440 may compare the pre-stored front video of the user with the obtained front video to determine whether additional correction is required. For example, based on the difference in the feature point positions (e.g., eyes, nose, lips) of the pre-stored front video of the user and the feature point positions (e.g., eyes, nose, lips) of the obtained front video, the front video acquisition module 440 may determine that additional correction is necessary. In this case, the front video acquisition module 440 may recapture the first user video and obtain the front video of the user again.

[0094] In addition, the front video acquisition module 440 may project the video call screen including the acquired front video onto the projection surface through the projection unit 112, and transmit the acquired front video to the other terminal of the video call. In this case, the front video acquisition module 440 may change the acquired front video into an avatar video according to the user input. For example, Fig. 9 As shown, the front video acquisition module 440 can change the front video 820 into an avatar video 920, project the video call screen including the changed avatar video 920 onto the projection surface through the projection unit 112, and send the obtained front video to the other terminal of the video call.

[0095] The video analysis module 450 may analyze the second user video captured while performing a video call. In this case, the video analysis module 450 may obtain information about the rotation of the user's face, the movement of lips, eyes, etc. by analyzing the movement of feature points (e.g., eyes, nose, lips) included in the second user video. For example, based on all the multiple feature points included in the second user video moving in one direction, the video analysis module 450 may detect the rotation of the user's face and calculate the rotation angle of the user's face according to the moving distance of the feature points. In addition, based on the movement of only some feature points (e.g., feature points related to lips) among the multiple feature points, the video analysis module 450 may obtain information about the movement of the user's lips.

[0096] The video correction module 460 may correct the frontal video of the user based on the analysis result of the video analysis module 450. Specifically, the video correction module 460 may correct the frontal video of the user based on the information about the rotation of the user's face or the movement of the lips, eyes, etc. obtained by the video analysis module 450. For example, based on the analysis that the user rotates his / her face 20 degrees to the left, the video correction module 460 may correct the frontal video of the user to rotate 20 degrees to the left. For example, based on the analysis that there is a lip movement of the user, the video correction module 460 may correct the frontal video of the user to reflect the lip movement of the user.

[0097] In other words, the video correction module 460 may provide a more natural video calling experience to the user by correcting the front video to reflect the actual movement of the user while performing the video call.

[0098] In addition, the above-mentioned embodiment describes that the electronic device 100 can calculate the horizontal correction angle and the vertical correction angle based on the first user video, the rotation angle of the camera 122, the projection angle of the projection unit 112, and the sensing value obtained by the sensor unit 121, and obtain the front video of the user based on the first user video, the horizontal correction angle, and the vertical correction angle. However, this is only an embodiment, and the electronic device 100 can pre-register the front video of the user or output a message guiding the user to look at the camera 122 of the electronic device 100 before starting a video call (or before accepting a video call), and then capture the front video of the user to provide the front video of the user.

[0099] In addition, the electronic device 100 may identify whether the electronic device 100 is located in a space where a video call can be made based on at least one of the first user video, the rotation angle of the camera, the projection angle of the projection unit, and the sensing value obtained by the sensor. In addition, the electronic device 100 may control the driving unit to move to the space where a video call can be made based on the electronic device 100 not being located in the space where a video call can be made.

[0100] Fig.10a and Fig.10b is a diagram for describing a method for moving an electronic device by identifying whether the electronic device is located in a space where a video call can be performed according to an embodiment of the present disclosure.

[0101] Specifically, based on the electronic device 100 executing the video call function, the electronic device 100 may identify whether the electronic device 100 is located in a space where a video call can be made based on at least one of a first user video obtained at a current position, a rotation angle of a camera, a projection angle of a projection unit, and a sensing value obtained by a sensor. More specifically, the electronic device 100 may determine whether the first user video can be corrected to a frontal video based on at least one of the first user video, the rotation angle of a camera, the projection angle of a projection unit, and a sensing value obtained by a sensor, and may determine whether the current first area is a space where a video call can be made. In particular, based on the user face included in the first user video being less than or equal to a critical ratio (e.g., 40%) compared to the entire user face, or the rotation angle of a camera or the projection angle of a projection unit exceeding a critical range, or the distance between the electronic device 100 and the projection surface 50 or the distance between the electronic device 100 and the user 10 obtained by a sensor being greater than or equal to a critical value, the electronic device 100 may determine that the current electronic device 100 is not located in a space where a video call can be made.

[0102] In addition, based on the electronic device 100 not being located in a space where a video call can be made, the electronic device 100 may control the drive unit 120 to move to a space where a video call can be made. In this case, the electronic device 100 may move to a space where a video call can be made based on at least one of the first user video, the rotation angle of the camera, the projection angle of the projection unit, and the sensing value obtained by the sensor. For example, based on the face of the user included in the first user video being in the left rear area, the electronic device 100 may control the drive unit 120 to move to the right front. In addition, the electronic device 100 may control the drive unit 120 so that the rotation angle or projection angle of the camera is within a critical range. In addition, the electronic device 100 may control the drive unit 120 so that the projection surface 50 and the user are located within a critical distance from the electronic device 100.

[0103] For example, Fig.10a As shown, based on determining that the electronic device 100 is outside the space 1000 where the video call can be made, the electronic device 100 can determine the space 1000 where the video call can be made based on at least one of the first user video, the rotation angle of the camera, the projection angle of the projection unit, and the sensed value obtained by the sensor. Fig.10b As shown, the electronic device 100 can control the driving unit 120 so that the electronic device 100 is located in a space 1000 where a video call can be performed.

[0104] Furthermore, in the above-described embodiment, it is described that one user is included in the first user video captured by the camera 122 , but this is merely an embodiment, and a plurality of users may be included in the first user video.

[0105] In particular, based on including multiple users in the user video, the electronic device 100 may identify a speaker who speaks among the multiple users. Then, the electronic device 100 may calculate a horizontal correction angle and a vertical correction angle based on the identified speaker. Specifically, the electronic device 100 may identify the speaker based on lip movements in the faces of the multiple users included in the first video. Then, the electronic device 100 may rotate the camera 122 so that the identified speaker is located in the center area of ​​the video. The electronic device 100 may obtain the first user video based on the speaker, and calculate the first distance a based on the speaker. 1 , the second distance a 2 , the rotation angle of the camera 122, the projection angle of the projection unit 112 and the first height h 1 To the third height h 3 , to calculate the horizontal correction angle and the vertical correction angle.

[0106] Fig.11 is a flowchart for describing a control method of an electronic device for providing a video call function according to an embodiment of the present disclosure.

[0107] First, the electronic device 100 executes a video call function (S1110).

[0108] The electronic device 100 captures the user through the camera 122 to obtain a first user video (S1120). In this case, the first user video may be a video captured from a side surface of the user 10 based on the user 10 looking toward the projection surface (particularly, the other party) 50.

[0109] The electronic device 100 calculates a horizontal correction angle and a vertical correction angle based on the first user video, the rotation angle of the camera 122 , the projection angle of the projection unit 112 , and a sensing value obtained by a sensor of the electronic device 100 ( S1130 ).

[0110] Specifically, the electronic device 100 may obtain information about a first distance between the electronic device 100 and the center of the projection surface based on a first sensing value obtained by at least one distance sensor. In addition, the electronic device 100 may obtain information about a second distance between the electronic device 100 and the user's face based on a second sensing value obtained by at least one distance sensor. The electronic device 100 may calculate a horizontal correction angle based on the information about the first distance, the information about the second distance, and the first rotation angle of the camera 122.

[0111] In addition, the electronic device 100 may obtain information about a first height from the floor to the center of the projection surface based on the projection angle of the projection unit 112 and a second sensing value obtained by at least one distance sensor. The electronic device 100 may obtain information about a second height from the floor to the user's face based on a second rotation angle of the camera 122 and a second sensing value obtained by at least one sensor. The electronic device 100 may obtain information about a third distance between the user's face and the center of the projection surface based on the information about the first distance, the information about the second distance, and the second rotation angle of the camera 122. The electronic device 100 may calculate a vertical correction angle based on the information about the first height, the information about the second height, and the information about the third distance.

[0112] The electronic device 100 obtains a frontal video of the user based on the first user video, the horizontal correction angle, and the vertical correction angle (S1140). In detail, the electronic device 100 may rotate the first user video in a first direction based on the horizontal correction angle, and rotate the first user video in a second direction based on the vertical correction angle, thereby obtaining a frontal video of the user. Alternatively, the electronic device 100 may obtain a frontal video of the user by inputting the first user video, the horizontal correction angle, and the vertical correction angle into a learned neural network model.

[0113] The electronic device 100 projects a video call screen including the obtained front video of the user through the projection unit 112 (S1150). In this case, the electronic device 100 may generate an avatar video based on the front video of the user, and may project the video call screen including the generated avatar video.

[0114] In addition, the electronic device 100 may obtain a second user video using the camera 122 while performing a video call. The electronic device 100 may calculate the rotation angle of the user's face based on the second user video. The electronic device 100 may rotate the user's frontal video based on the calculated rotation angle of the user's face.

[0115] In addition, the electronic device 100 may identify whether the electronic device 100 is located in a space where a video call can be made based on at least one of the first user video, the rotation angle of the camera, the projection angle of the projection unit 112, and the sensing value obtained by the sensor unit 121. In addition, based on the electronic device 100 not being located in the space where a video call can be made, the electronic device 100 may control the driving unit 120 to move to the space where a video call can be made.

[0116] In addition, based on the inclusion of a plurality of users in the first user video, the electronic device 100 may identify a speaker among the plurality of users. The electronic device 100 may calculate a horizontal correction angle and a vertical correction angle based on the identified speaker.

[0117] Fig.12 Detailed description of the configuration of an electronic device according to an embodiment of the present disclosure. Fig.12 , the electronic device 100 may include at least one of a processor 111, a projection unit 112, a memory 113, a communication interface 114, a manipulation interface 115, an input / output interface 116, a speaker 117, a microphone 118, a power supply unit 119, a driving unit 120, or a sensor unit 121. In addition, Fig.12 The components shown in FIG. 2 are merely examples of various embodiments, and some components may be omitted and new components may be added. In addition, the contents already described in FIG. 2 are omitted.

[0118] The processor 111 may be implemented as a digital signal processor (DSP), a microprocessor, or a time controller (TCON) for processing digital signals. However, the processor 111 is not limited thereto, and may include one or more of a central processing unit (CPU), a microcontroller unit (MCU), a microprocessing unit (MPU), a controller, an application processor (AP), a graphics processing unit (GPU), or a communication processor (CP) or an advanced reduced instruction set computer (RISC) machine (ARM) processor, or may be defined by these terms. In addition, the processor 111 may be implemented by a system on chip (SoC) or a large-scale integrated circuit (LSI) embedded with a processing algorithm, or may be implemented in the form of a field programmable gate array (FPGA). In addition, the processor 111 may perform various functions by running computer executable instructions stored in the memory 113.

[0119] The projection unit 112 may be a component that projects an image to the outside. The projection unit 112 according to various embodiments of the present disclosure may be implemented in various projection types (e.g., a cathode ray tube (CRT) type, a liquid crystal display (LCD) type, a digital light processing (DLP) type, or a laser type). As various examples, the CRT type has a principle that is substantially the same as that of a CRT display. The CRT type may display an image on a screen by magnifying the image using a lens in front of a cathode ray tube (CRT). The CRT type may be divided into a single tube type and a triple tube type based on the number of cathode ray tubes, and in the triple tube type, cathode ray tubes of red, green, and blue are separated from each other.

[0120] For various examples, the LCD type can display an image by allowing light emitted from a light source to pass through liquid crystal. The LCD type can be divided into a single-panel type and a three-panel type. Based on the three-panel type, the light emitted from the light source can be separated into red, green, and blue in a dichroic mirror (which is a mirror that reflects only light of a specific color and allows the rest of the light to pass through), and then can pass through the liquid crystal, and then can be collected in one place again.

[0121] For other various examples, the DLP type can display an image by using a digital micromirror device (DMD) chip. The DLP type projection unit may include a light source, a color wheel, a DMD chip, a projection lens, etc. The light emitted from the light source may be colored when passing through a rotating color wheel. The light passing through the color wheel may be input into the DMD chip. The DMD chip may include many micromirrors and reflect the light input to the DMD chip. The projection lens may expand the light reflected from the DMD chip to an image size.

[0122] For various other examples, the laser type may include a diode pumped solid-state (DPSS) laser and a galvanometer. A laser type that outputs various colors may use a laser in which three DPSS lasers are installed for red, green, and blue (RGB), respectively, and their optical axes overlap each other by using special mirrors. The galvanometer may include a mirror and a high-power motor, and move the mirror at a high speed. For example, the galvanometer can rotate the mirror at a speed of up to 40 kHz / sec. The galvanometer can be installed along the scanning direction, and typically, the projector performs a plane scan, so the galvanometer can also be set by dividing into an x-axis and a y-axis.

[0123] In addition, the projection unit 112 may include various types of light sources. For example, the projection unit 112 may include at least one light source of a lamp, a light emitting diode (LED), and a laser.

[0124] The projection unit 112 can output images with an aspect ratio of 4:3, an aspect ratio of 5:4, and a wide aspect ratio of 16:9 based on the purpose of the electronic device 100, the user's settings, etc., and can output images with various resolutions based on the aspect ratio, such as wide video graphics array WVGA (854×480 pixels), super video graphics array SVGA (800×600 pixels), extended graphics array XGA (1024×768 pixels), wide extended graphics array WXGA (1280×720 pixels), WXGA (1280×800 pixels), super extended graphics array SXGA (1280×1024 pixels), ultra extended graphics array UXGA (1600×1200 pixels) and full high definition HD (1920×1080 pixels).

[0125] The projection unit 112 may perform various functions for adjusting an output image under the control of the processor 111. For example, the projection unit 112 may perform a zoom function, a keystone function, a fast-angle (or four-corner) keystone function, a lens shift function, and the like.

[0126] In addition, the projection unit 112 may enlarge or reduce the image based on its distance from the screen (i.e., the projection distance). That is, the projection unit 112 may perform a zoom function based on its distance from the screen. The zoom function may include a hardware method for adjusting the size of the picture by moving the lens, a software method for adjusting the size of the picture by cropping the image, and the like. In addition, in the case of performing the zoom function, it is necessary to adjust the focus of the image. For example, the method for adjusting the focus may include a manual focus method, an electric focus method, and the like. The manual focus method may indicate a method for manually adjusting the focus, and the electric focus method may indicate a method for automatically adjusting the focus of the projector based on performing a zoom function by using a motor built therein. Based on performing the zoom function, the projection unit 112 may provide a digital zoom function through software, and may provide an optical zoom function for performing a zoom function by moving the lens using the drive unit 120.

[0127] In addition, the projection unit 112 may perform a keystone correction function. The picture may be distorted upward or downward based on the height mismatch with the front projection. The keystone correction function may be a function of correcting a distorted picture. For example, in the case where distortion occurs in the horizontal direction on the picture, horizontal keystone may be used to correct the distortion, and in the case where distortion occurs in the vertical direction on the picture, vertical keystone may be used to correct the distortion. The fast corner (or four-corner) keystone correction function may be a function of correcting the picture when the balance between the corner areas of the picture is not appropriate and the center area of ​​the picture is normal. The lens shift function may be a function of moving the picture as it is when the picture is outside the screen area.

[0128] In addition, the projection unit 112 can provide a zoom / keystone / focus function by automatically analyzing the surrounding environment and the projection environment without user input. In addition, the projection unit 112 can automatically provide a zoom / keystone / focus function based on the distance between the electronic device 100 and the screen detected by a sensor (e.g., a depth camera, a distance sensor, an infrared sensor, or an illumination sensor), information about the space where the electronic device 100 is currently located, information about the amount of ambient light, etc.

[0129] In addition, the projection unit 112 may provide a lighting function by using a light source. In particular, the projection unit 112 may provide a lighting function by outputting a light source using an LED. In various embodiments, the projection unit 112 may include one LED, and in another embodiment, the electronic device 100 may include a plurality of LEDs. In addition, in an implementation example, the projection unit 112 may output a light source by using a surface-emitting LED. Here, the surface-emitting LED may be an LED in which an optical sheet is disposed on the upper side of the LED so that the light source is evenly dispersed and output. In addition, based on output through the LED, the light source may be evenly dispersed through the optical sheet, and the light source dispersed through the optical sheet may be introduced into the display panel.

[0130] In addition, the projection unit 112 may provide a dimming function for adjusting the intensity of the light source to the user. In addition, the projection unit 112 may control the LED to output the intensity of the light source corresponding to the received user input based on receiving a user input for adjusting the intensity of the light source from the user through the manipulation interface 115 (e.g., touching a display button or a dial).

[0131] In addition, the projection unit 112 may provide a dimming function without user input based on the content analyzed by the processor 111. In addition, the projection unit 112 may control the intensity of the LED output light source based on information about the currently provided content (eg, content type or content brightness).

[0132] In addition, the projection unit 112 may control the color temperature by the control of the processor 111. Here, the processor 111 may control the color temperature based on the content. In the case of identifying that the content will be output, the processor 111 may obtain the color information of each frame of the content determined to be output. In addition, the processor 111 may then control the color temperature based on the color information of each frame obtained. The processor 111 may obtain at least one main color of the frame based on the color information of each frame. Then, the processor 111 may adjust the color temperature based on the at least one main color obtained. For example, the color temperature adjustable by the processor 111 may be divided into a warm color type or a cold color type. The frame to be output (hereinafter, the output frame) may include a fire scene. The processor 111 may identify (or obtain) that the main color is red based on the color information included in the current output frame. Then, the processor 111 may identify the color temperature corresponding to the identified main color (red). The color temperature corresponding to red may be a warm color type. In addition, the processor 111 may use an artificial intelligence model to obtain the color information or main color of the frame. In various examples, the artificial intelligence model may be stored in the electronic device 100 (e.g., the memory 113). In another example, the artificial intelligence model may be stored in an external server that can communicate with the electronic device 100.

[0133] The memory 113 may be implemented as an internal memory included in the processor 111, such as a read-only memory (ROM) (e.g., an electrically erasable programmable read-only memory (EEPROM)), a random access memory (RAM), etc., or may be implemented as a memory separate from the processor 111. In this case, the memory 113 may be implemented in the form of a memory embedded in the electronic device 100 or in the form of a memory attachable to and detachable from the electronic device 100, depending on the data storage purpose. For example, data for driving the electronic device 100 may be stored in the memory embedded in the electronic device 100, and data for extended functions of the electronic device 100 may be stored in the memory attachable to and detachable from the electronic device 100.

[0134] In addition, the memory embedded in the electronic device 100 may be implemented by at least one of a volatile memory (e.g., dynamic RAM (DRAM), static RAM (SRAM), synchronous dynamic RAM (SDRAM), etc.) or a nonvolatile memory (e.g., one-time programmable ROM (OTPROM), programmable ROM (PROM), erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), mask ROM, flash ROM, flash memory (e.g., NAND flash memory, NOR flash memory, etc.), hard disk drive, or solid state drive (SSD)), and the memory that can be attached to and detached from the electronic device 100 may be implemented in the form of a memory card (e.g., compact flash (CF), secure digital (SD), micro SD, mini SD, extreme digital (xD), multimedia card (MMC), etc.), an external memory connectable to a USB port (e.g., a universal serial bus (USB) memory), etc.

[0135] The memory 113 may store at least one instruction on the electronic device 100. In addition, the memory 113 may store an operating system (O / S) for driving the electronic device 100. In various examples of the present disclosure, the memory 113 may also store various software programs or applications for operating the electronic device 100. In addition, the memory 113 may include a semiconductor memory such as a flash memory or a magnetic storage medium such as a hard disk.

[0136] In addition, in various examples of the present disclosure, the memory 113 may store various software modules for operating the electronic device 100, and the processor 111 may control the operation of the electronic device 100 by running the various software modules stored in the memory 113. That is, the memory 113 may be accessed by the processor 111, and the processor 111 may perform reading, recording, correction, deletion, updating, etc. of data in the memory 113.

[0137] In addition, in the present disclosure, the term “memory 113 ” may include a memory in the processor 111 , a read only memory (ROM) or a random access memory (RAM), or a memory card (e.g., a micro secure digital (SD) card or a memory stick) installed in the electronic device 100 .

[0138] The communication interface 114 is a component that performs communication with various types of external devices according to various types of communication methods. The communication interface 114 may include a wireless communication module or a wired communication module. Here, each communication module may be implemented in the form of at least one hardware chip.

[0139] The wireless communication module may be a module for wirelessly communicating with an external device. For example, the wireless communication module may include at least one of a Wi-Fi module, a Bluetooth module, an infrared communication module, or other communication modules.

[0140] The Wi-Fi module and the Bluetooth module perform communication in the Wi-Fi method and the Bluetooth method, respectively. Based on the use of the Wi-Fi module or the Bluetooth module, various connection information such as a service set identifier (SSID), a session key, etc. are first sent and received, the communication is connected using the connection information, and then various information can be sent and received.

[0141] The infrared communication module performs communication according to an Infrared Data Association (IrDA) technology that wirelessly transmits data over a short distance using infrared rays between visible light and millimeter waves.

[0142] In addition to the above-mentioned communication methods, other wireless communication modules may include at least one communication chip that performs communication according to various wireless communication standards (such as zigbee, third generation (3G), third generation partnership project (3GPP), long term evolution (LTE), advanced LTE (LTE-A), fourth generation (4G), fifth generation (5G), etc.).

[0143] The wired communication module may be a module for communicating with an external device in a wired manner. For example, the wired communication module may include at least one of a local area network (LAN) module, an Ethernet module, a twisted pair cable, a coaxial cable, an optical fiber cable, or an ultra-wideband (UWB) module.

[0144] The manipulation interface 115 may include various types of input devices. For example, the manipulation interface 115 may include a physical button. The physical button may include a function key, a direction key (e.g., a four-way key), or a dial button. In various examples, the physical button may be implemented as a plurality of keys. In another example, the physical button may be implemented as one key. In the case where the physical button is implemented as one key, the electronic device 100 may receive a user input in which one of the keys is pressed for a critical time or longer. Based on receiving a user input in which one of the keys is pressed for a critical time or longer, the processor 111 may execute a function corresponding to the user input. For example, the processor 111 may provide a lighting function based on the user input.

[0145] In addition, the manipulation interface 115 may receive user input by using a non-contact method. In the case of receiving user input by using a contact method, it may be necessary to transmit a physical force to the electronic device 100. Therefore, a method of controlling the electronic device 100 that is independent of physical force may be required. The manipulation interface 115 may receive a user gesture and may perform an operation corresponding to the received user gesture. Here, the manipulation interface 115 may receive a user gesture through a sensor (e.g., an image sensor or an infrared sensor).

[0146] In addition, the manipulation interface 115 can receive user input by using a touch method. For example, the manipulation interface 115 can receive user input through a touch sensor. In various examples, the touch method can be implemented as a non-contact method. For example, the touch sensor can determine whether the user's body is close to a critical distance. Here, the touch sensor can recognize the user input even if the user does not touch the touch sensor. In addition, in various examples, the touch sensor can recognize the user input of the user touching the touch sensor.

[0147] In addition, the electronic device 100 may receive user input in various ways other than the above-mentioned manipulation interface 115. In various examples, the electronic device 100 may receive user input from an external remote control device. Here, the external remote control device may be a remote control device corresponding to the electronic device 100 (e.g., a control device dedicated to the electronic device 100) or a portable communication device of the user (e.g., a smart phone or a wearable device). Here, the user's portable communication device may store an application for controlling the electronic device 100. The portable communication device may obtain user input from the application stored therein and send the obtained user input to the electronic device 100. The electronic device 100 may receive user input from the portable communication device and perform an operation corresponding to the user's control command.

[0148] In addition, the electronic device 100 may receive user input by using voice recognition. In various examples, the electronic device 100 may receive user voice through a microphone included in the electronic device 100. In various examples, the electronic device 100 may receive user voice from a microphone or an external device. In addition, the external device may obtain the user voice through the microphone of the external device and send the obtained user voice to the electronic device 100. The user voice sent from the external device may be audio data or digital data converted from the audio data (e.g., audio data converted to the frequency domain). Here, the electronic device 100 may perform an operation corresponding to the received user voice. In addition, the electronic device 100 may receive audio data corresponding to the user voice through a microphone. Then, the electronic device 100 may convert the received audio data into digital data. Then, the electronic device 100 may convert the converted digital data into text data by using a speech-to-text (STT) function. In various examples, the speech-to-text (STT) function may be directly performed by the electronic device 100.

[0149] In another example, a speech-to-text (STT) function may be performed by an external server. The electronic device 100 may send digital data to the external server. The external server may convert the digital data into text data and obtain control command data based on the converted text data. The external server may send the control command data (which may also include text data) to the electronic device 100. The electronic device 100 may perform an operation corresponding to the user's voice based on the obtained control command data.

[0150] In addition, the electronic device 100 can provide the voice recognition function by using one auxiliary tool (or an artificial intelligence agent such as Bixby TM), which is just various examples, and the electronic device 100 can provide the voice recognition function by using multiple auxiliary tools. The electronic device 100 can provide the voice recognition function by selecting an auxiliary tool based on a trigger word corresponding to one of the multiple auxiliary tools or a specific key included in the remote controller.

[0151] In addition, the electronic device 100 may receive user input by using screen interaction. The screen interaction may indicate a function of the electronic device 100 identifying whether a predetermined event is generated by an image projected to a screen (or projection plane) and obtaining user input based on the predetermined event. Here, the predetermined event may be an event in which a predetermined object is recognized at a specific location (e.g., a location where a UI for receiving user input is projected). Here, the predetermined object may include at least one of a user's body part (e.g., a finger), a pointer, or a laser pen. In the case of recognizing that a predetermined object exists at a location corresponding to the projected UI, the electronic device 100 may recognize that the electronic device 100 receives a user input for selecting the projected UI. For example, the electronic device 100 may project a guide image showing a UI on the screen. Then, the electronic device 100 may recognize whether the user selects the projected UI. In the case of recognizing a predetermined event at the location of the projected UI, the electronic device 100 may recognize that the user has selected the projected UI. The projected UI may include at least one item. The electronic device 100 may perform spatial analysis to identify whether a predetermined event exists at the location of the projected UI. The electronic device 100 may perform spatial analysis through a sensor (e.g., an image sensor, an infrared sensor, a depth camera, or a distance sensor). The electronic device 100 may identify whether a predetermined event is generated at a specific location (i.e., the location where the UI is projected) by performing spatial analysis. In addition, when it is identified that a predetermined event is generated at a specific location (i.e., the location where the UI is projected), the electronic device 100 may identify that the electronic device 100 receives a user input for selecting the UI corresponding to the specific location.

[0152] The input / output interface 116 is a component for inputting or outputting at least one of an audio signal or an image signal. The input / output interface 116 may receive at least one of an audio signal or an image signal from an external device and output a control command to the external device.

[0153] According to an implementation example, the input / output interface 116 may be implemented as an interface that inputs / outputs only audio signals and an interface that inputs / outputs only image signals, or may be implemented as one interface that inputs / outputs both audio signals and image signals.

[0154] In addition, the input / output interface 116 according to various embodiments of the present disclosure may be implemented as a wired input / output interface of at least one of a High-Definition Multimedia Interface (HDMI), a Mobile High-Definition Link (MHL), a Universal Serial Bus (USB), a USB Type-C, a DisplayPort (DP), a Thunderbolt interface, a Video Graphics Array (VGA) port, a Red, Green, and Blue (RGB) port, a D-Sub (D-SUB), or a Digital Video Interface (DVI). In various examples, the wired input / output interface may be implemented as an interface that inputs or outputs only an audio signal and an interface that inputs or outputs only an image signal, or as one interface that inputs or outputs both an audio signal and an image signal.

[0155] In addition, the electronic device 100 may receive data through the wired input / output interface, which is just various examples, and the electronic device 100 may receive power through the wired input / output interface. For example, the electronic device 100 may receive power from an external battery through a USB Type-C, or receive power from an outlet through a power adapter. For various examples, the electronic device may receive power from an external device (e.g., a laptop or monitor) through a display port (DP).

[0156] In addition, the audio signal may be implemented as received through a wired input / output interface, and the image signal may be implemented as received through a wireless input / output interface (or communication interface). Alternatively, the audio signal may be implemented as received through a wireless input / output interface (or communication interface), and the image signal may be implemented as received through a wired input / output interface.

[0157] The speaker 117 is a component that outputs an audio signal. Specifically, the speaker 117 may include an audio output mixer, an audio signal processor, and an audio output module. The audio output mixer may mix multiple audio signals to output as at least one audio signal. For example, the audio output mixer may mix an analog audio signal and another analog audio signal (e.g., an analog audio signal received from the outside) into at least one analog audio signal. The audio output module may include a speaker or an output terminal. According to various embodiments, the audio output module may include a plurality of speakers. In this case, the audio output module may be arranged in the main body, and the audio emitted when covering at least a portion of the diaphragm of the audio output module may pass through a waveguide to be transmitted to the outside of the main body. The audio output module may include a plurality of audio output units, and the plurality of audio output units may be symmetrically arranged outside the main body, so that the audio may be emitted in all directions, i.e., all directions of 360 degrees.

[0158] The microphone 118 is a component for receiving the user's voice or other sounds and converting the user's voice or other sounds into audio data. The microphone 118 can receive the user's voice in an activated state. For example, the microphone 118 can be formed integrally with the electronic device 100 in directions such as above, in front, and on the side. The microphone 118 may include various components, such as a microphone for collecting the user's voice in analog form, an amplifier circuit for amplifying the collected user's voice, an A / D conversion circuit for sampling the amplified user's voice to convert the amplified user's voice into a digital signal, a filter circuit for removing noise components from the converted digital signal, etc.

[0159] The power supply unit 119 may receive power from the outside and supply power to various components of the electronic device 100. The power supply unit 119 according to various embodiments of the present disclosure may receive power in various ways. In various examples, the power supply unit 119 may receive power by using, for example, Fig.12 The power supply unit 119 may receive power by using a USB power line or by using a wireless charging method.

[0160] In addition, the power supply unit 119 can receive power by using an internal battery or an external battery. The power supply unit 119 according to various embodiments of the present disclosure can receive power through the internal battery. For example, the power supply unit 119 can charge the power of the internal battery by using at least one of a 220V DC power line, a USB power line, or a USB C-type power line, and can receive power through the charged internal battery. In addition, the power supply unit 119 according to various embodiments of the present disclosure can receive power through an external battery. For example, in the case where the electronic device and the external battery are connected by various wired communication methods (such as a USB power line, a USB C-type power line, or a socket slot), the power supply unit 119 can receive power through an external battery. That is, the power supply unit 119 can receive power directly from the external battery, or charge the internal battery through the external battery and receive power from the charged internal battery.

[0161] The power supply unit 119 according to the present disclosure may receive power by using at least one power supply method among the above-mentioned various power supply methods.

[0162] In addition, regarding power consumption, the electronic device 100 may have a power consumption of a predetermined value (e.g., 43W) or less due to a socket type, another standard, etc. The electronic device 100 may change power consumption based on using a battery to reduce power consumption. That is, the electronic device 100 may change power consumption based on a power supply method, power usage, etc.

[0163] The driving unit 120 may drive at least one hardware component included in the electronic device 100. The driving unit 120 may generate a physical force and transmit the generated physical force to at least one hardware component included in the electronic device 100.

[0164] Here, the driving unit 120 may generate a driving force for moving a hardware component included in the electronic device 100 (eg, moving the electronic device 100 ) or rotating the component (eg, rotating a projection lens).

[0165] The driving unit 120 may adjust the projection direction (or projection angle) of the projection unit 122. In addition, the driving unit 120 may move the position of the electronic device 100. Here, the driving unit 120 may control the moving member 109 to move the electronic device 100. For example, the driving unit 120 may control the moving member 109 using a motor.

[0166] The sensor unit 121 may include at least one sensor. Specifically, the sensor unit 121 may include at least one of a tilt sensor for sensing the tilt of the electronic device 100 and an image sensor for capturing an image. Here, the tilt sensor may be an acceleration sensor or a gyroscope sensor, and the image sensor may represent a camera or a depth camera. In addition, the tilt sensor may be described as a motion sensor. In addition, the sensor unit 121 may include various sensors other than the tilt sensor or the image sensor. For example, the sensor unit 121 may include an illumination sensor and a distance sensor. The distance sensor may be a time of flight (ToF). In addition, the sensor unit 121 may include a lidar sensor.

[0167] In addition, the electronic device 100 can control the lighting function by linking with an external device. Specifically, the electronic device 100 can receive lighting information from the external device. Here, the lighting information may include at least one of brightness information or color temperature information set in the external device. Here, the external device may represent a device connected to the same network as the electronic device 100 (for example, an IoT device included in the same home / work network) or a device that is not connected to the same network as the electronic device 100 but can communicate with the electronic device 100 (for example, a remote control server). For example, assume that an external lighting device (IoT device) included in the same network as the electronic device 100 is outputting red light at a brightness of 50. The external lighting device (IoT device) may directly or indirectly send lighting information (for example, information indicating that red light is being output at a brightness of 50) to the electronic device 100. Here, the electronic device 100 may control the output of the light source based on the lighting information received from the external lighting device. For example, based on the lighting information received from the external lighting device including information indicating that red light is output at a brightness of 50, the electronic device 100 may output red light at a brightness of 50.

[0168] In addition, the electronic device 100 may control the lighting function based on biometric information. Specifically, the processor 111 may obtain the biometric information of the user. Here, the biometric information may include at least one of the user's body temperature, heart rate, blood pressure, respiration, and electrocardiogram. Here, the biometric information may include various types of information in addition to the above information. As an example, the electronic device 100 may include a sensor for measuring biometric information. The processor 111 may obtain the biometric information of the user through the sensor, and control the output of the light source based on the obtained biometric information. As another example, the processor 111 may receive biometric information from an external device through the input / output interface 116. Here, the external device may represent a portable communication device of the user (e.g., a smartphone or a wearable device). The processor 111 may obtain the biometric information of the user from the external device, and control the output of the light source based on the obtained biometric information. In addition, according to an implementation example, the electronic device 100 may identify whether the user is sleeping, and based on determining that the user is sleeping (or preparing to sleep), the processor 111 may control the output of the light source based on the biometric information of the user.

[0169] In addition, the electronic device 100 according to various embodiments of the present disclosure may provide various smart functions.

[0170] In addition, the electronic device 100 may be connected to a portable terminal device that controls the electronic device 100, and the screen output from the electronic device 100 may be controlled by a user input input from the portable terminal device. For example, the portable terminal device may be implemented as a smart phone including a touch display, and the electronic device 100 may receive screen data provided by the portable terminal device from the portable terminal device and output the data, and the screen output by the electronic device 100 may be controlled based on the user input input from the portable terminal device.

[0171] The electronic device 100 may be connected to a portable terminal device by using various communication methods such as miracast, airplay, wireless dalvik executable (DEX), and a remote personal computer (PC) method, and may share content or music provided by the portable terminal device.

[0172] In addition, the portable terminal device and the electronic device 100 can be connected to each other through various connection methods. In various examples, the portable terminal device can search for the electronic device 100 and perform a wireless connection between them, or the electronic device 100 can search for the portable terminal device and perform a wireless connection between them. Then, the electronic device 100 can output the content provided from the portable terminal device.

[0173] In various examples, when specific content or music is being output from a portable terminal device, when the portable terminal device is located around the electronic device 100 and then a predetermined gesture (e.g., a motion tap view) is detected through the display of the portable terminal device, the electronic device 100 may output the content or music being output from the portable terminal device.

[0174] In various examples, while specific content or music is being output from a portable terminal device, when it is detected that the portable terminal device is located at a predetermined distance or less around the electronic device 100 (e.g., a non-contact tap view) or the portable terminal device touches the electronic device 100 twice at a short interval (e.g., a contact tap view), the electronic device 100 may output the content or music being output from the portable terminal device.

[0175] In the above example, the screen provided by the portable terminal device is the same as the screen provided by the electronic device 100. However, the present disclosure is not limited to this. That is, when the portable terminal device and the electronic device 100 are connected to each other, the portable terminal device may output a first screen provided by the portable terminal device, and the electronic device 100 may output a second screen provided by the portable terminal device that is different from the first screen. For example, the first screen may be a screen provided by a first application installed in the portable terminal device, and the second screen may be a screen provided by a second application installed in the portable terminal device. For example, the first screen and the second screen may be different screens provided by an application installed in the portable terminal device. In addition, for example, the first screen may be a screen including a UI in the form of a remote control for controlling the second screen.

[0176] The electronic device 100 according to the present disclosure may output a standby screen. For example, when the electronic device 100 and an external device are not connected to each other or when no input is received from an external device within a predetermined time, the electronic device 100 may output a standby screen. The conditions under which the electronic device 100 outputs the standby screen are not limited to the above examples, and the standby screen may be output based on various conditions.

[0177] The electronic device 100 may output the standby screen in the form of a blue screen, and the present disclosure is not limited thereto. For example, the electronic device 100 may extract only the shape of a specific object from data received from an external device to obtain an atypical object, and output the standby screen including the obtained atypical object.

[0178] In addition, the electronic device 100 may further include a display (not shown).

[0179] The display (not shown) may be implemented as various types of displays, such as a liquid crystal display (LCD), an organic light emitting diode (OLED) display, a plasma display panel (PDP), etc. A driving circuit, a backlight unit, etc., which may be implemented in a form such as an amorphous silicon thin film transistor (a-siTFT), a low temperature polycrystalline silicon (LTPS) TFT, an organic TFT (OTFT), etc., may be included in the display (not shown). In addition, the display (not shown) may be implemented as a touch screen combined with a touch sensor, a flexible display, a three-dimensional (3D) display, etc. In addition, the display (not shown) according to various embodiments of the present disclosure may include not only a display panel that outputs an image, but also a frame that accommodates the display panel. In particular, the frame according to various embodiments of the present disclosure may include a touch sensor (not shown) for sensing user interaction.

[0180] In addition, the electronic device 100 may further include a shutter unit (not shown).

[0181] The shutter unit (not shown) may include at least one of a shutter, a fixing member, a rail, or a body.

[0182] Here, the shutter may block light output from the projection unit 112. Here, the fixing member may fix the position of the shutter. Here, the track may be a path for moving the shutter and the fixing member. Here, the body may be an assembly including the shutter and the fixing member.

[0183] Fig.13 is a perspective view showing the appearance of an electronic device according to an embodiment of the present disclosure. Fig.13 In the embodiment 1310, the electronic device 100 may include a moving member 109. The moving member 109 may refer to a member for moving from a first position to a second position in a space where the electronic device 100 is arranged. The electronic device 100 may control the moving member 109 to move the electronic device 100 by using a force generated from the driving unit 120. In this case, the moving member 109 may include a motor or a wheel.

[0184] Fig.13 Embodiment 1320 is a diagram of the electronic device 100 of embodiment 1310 observed from a different direction.

[0185] In addition, the methods according to various embodiments of the present disclosure may be included and provided in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., a compact disc read-only memory (CD-ROM)), or may be downloaded through an application store (e.g., PlayStore). TM ) or may be distributed online directly between two user devices (e.g., smart phones) (e.g., downloaded or uploaded). In the case of online distribution, at least some of the computer program products (e.g., downloadable apps) may be at least temporarily stored in a machine-readable storage medium (such as a memory of a manufacturer's server, a memory of an application store's server, or a memory of a relay server), or may be temporarily created.

[0186] The methods according to various embodiments of the present disclosure may be implemented by software including instructions stored in a machine-readable storage medium (e.g., a computer-readable storage medium). A machine is a device that can call stored instructions from a storage medium and operate according to the called instructions, and may include an electronic device (e.g., a TV) of the disclosed embodiments.

[0187] In addition, the machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, "non-transitory storage medium" means that the storage medium is a tangible device and does not include signals (e.g., electromagnetic waves), and the term does not distinguish between a case where data is semi-permanently stored on the storage medium and a case where data is temporarily stored on the storage medium. For example, the "non-transitory storage medium" may include a buffer that temporarily stores data.

[0188] In the case where the command is executed by the processor, the processor may directly execute the function corresponding to the command, or other components may execute the function corresponding to the command under the control of the processor. The command may include a code created or executed by a compiler or an interpreter.

[0189] Hereinafter, although exemplary embodiments of the present disclosure have been shown and described, the present disclosure is not limited to the above-mentioned specific exemplary embodiments, but various modifications may be made by those skilled in the art to which the present disclosure belongs without departing from the gist of the present disclosure disclosed in the appended claims. These modifications should also be understood to fall within the scope and spirit of the present disclosure.

Claims

1. An electronic device for providing a video call function, comprising: Projection unit; camera; a sensor configured to obtain information about a distance to an external object; a memory configured to store at least one instruction; as well as At least one processor is configured to be connected to the projection unit, the camera, the sensor, and the memory, and to control the electronic device, wherein the at least one processor is configured to execute the at least one instruction to perform the following operations: Based on executing the video call function, the user is captured by the camera to obtain a first user video, calculating a horizontal correction angle and a vertical correction angle based on the first user video, a rotation angle of the camera, a projection angle of the projection unit, and a sensing value obtained by the sensor, obtaining a frontal video of the user based on the first user video, the horizontal correction angle, and the vertical correction angle, and The projection unit is controlled so that a video call screen including the obtained front video of the user is projected.

2. The electronic device according to claim 1, wherein: The at least one processor is configured to: obtaining information about a first distance between the electronic device and a center of a projection surface based on a first sensing value obtained by the sensor, obtaining information about a second distance between the electronic device and the user's face based on a second sensing value obtained by the sensor, and The horizontal correction angle is calculated based on information about the first distance, information about the second distance, and a first rotation angle of the camera.

3. The electronic device as claimed in claim 2, wherein: The at least one processor is configured to: obtaining information about a first height from a floor to a center of the projection surface based on a projection angle of the projection unit and the second sensing value obtained by the sensor, obtaining information about a second height from the floor to the user's face based on a second rotation angle of the camera and the second sensing value obtained by the sensor, obtaining information about a third distance between the face of the user and the center of the projection surface based on the information about the first distance, the information about the second distance, and a second rotation angle of the camera, and The vertical correction angle is calculated based on the information about the first height, the information about the second height, and the information about the third distance.

4. The electronic device according to claim 1, wherein: The at least one processor is configured to rotate the first user video in a first direction based on the horizontal correction angle and to rotate the first user video in a second direction based on the vertical correction angle to obtain a frontal video of the user.

5. The electronic device as claimed in claim 1, wherein: The at least one processor is configured to input the first user video, the horizontal correction angle, and the vertical correction angle into a trained neural network model to obtain a frontal video of the user.

6. The electronic device as claimed in claim 1, wherein: The at least one processor is configured to: generating an avatar video based on the frontal video of the user, The projection unit is controlled so that a video call screen including the generated avatar video is projected.

7. The electronic device as claimed in claim 1, further comprising: a driving unit configured to move the electronic device, Wherein, the at least one processor is configured to: identifying whether the electronic device is located in a space where a video call can be made based on at least one of the first user video, the rotation angle of the camera, the projection angle of the projection unit, and a sensing value obtained by the sensor, and Based on the electronic device not being located in the space where the video call can be made, the driving unit is controlled to move to the space where the video call can be made.

8. The electronic device as claimed in claim 1, wherein: The at least one processor is configured to: Using the camera to obtain a second user video while performing a video call, calculating a rotation angle of the user's face based on the second user video, and The frontal video of the user is rotated based on the calculated rotation angle of the user's face.

9. The electronic device as claimed in claim 1, wherein: The at least one processor is configured to: identifying a speaker among the plurality of users based on that the first user video includes the plurality of users, The horizontal correction angle and the vertical correction angle are calculated based on the recognized speaker.

10. A control method for an electronic device for providing a video call function, the electronic device comprising a projection unit for projecting a video, the control method comprising: Based on the video call function being executed, capturing the user by a camera to obtain a first user video; calculating a horizontal correction angle and a vertical correction angle based on the first user video, a rotation angle of the camera, a projection angle of the projection unit, and a sensing value obtained by a sensor of the electronic device; obtaining a frontal video of the user based on the first user video, the horizontal correction angle, and the vertical correction angle; and A video call screen including the obtained frontal video of the user is projected through the projection unit.

11. The control method according to claim 10, wherein: The calculation steps include: obtaining information about a first distance between the electronic device and a center of a projection surface based on a first sensing value obtained by the sensor; obtaining information about a second distance between the electronic device and the user's face based on a second sensing value obtained by the sensor; and The horizontal correction angle is calculated based on information about the first distance, information about the second distance, and a first rotation angle of the camera.

12. The control method according to claim 11, wherein: The calculation steps include: obtaining information about a first height from a floor to a center of the projection surface based on a projection angle of the projection unit and the second sensing value obtained by the sensor; obtaining information about a second height from the floor to the user's face based on a second rotation angle of the camera and the second sensing value obtained by the sensor; obtaining information about a third distance between the user's face and the center of the projection surface based on the information about the first distance, the information about the second distance, and a second rotation angle of the camera; and The vertical correction angle is calculated based on the information about the first height, the information about the second height, and the information about the third distance.

13. The control method according to claim 10, wherein: When obtaining the frontal video, the first user video is rotated in a first direction based on the horizontal correction angle and in a second direction based on the vertical correction angle to obtain a frontal video of the user.

14. The control method according to claim 10, wherein: When obtaining the frontal video, the first user video, the horizontal correction angle and the vertical correction angle are input into the trained neural network model to obtain the frontal video of the user.

15. The control method according to claim 10, further comprising: generating an avatar video based on the frontal video of the user, During the projection, a video call screen including the generated avatar video is projected.

Images