Electronic apparatus and controlling method thereof

Abstract

An electronic device is disclosed. The electronic device includes a camera and a processor that tracks an object region including a user object in an image captured through the camera and identifies a display region in the image captured based on the tracked object region. The processor identifies a display region of the first image captured based on an object region identified in the first image captured, identifies a display region of the second image captured based on an object region identified in the second image captured, and identifies a display region of the third image captured based on the display region of the first image captured and the display region of the second image captured.

Description

Technology Field

[0001] The present disclosure relates to an electronic device and a method for controlling the same, and more specifically, to an electronic device for displaying an object included in an image and a method for controlling the same. Background Technology

[0002] Users who follow exercise postures (e.g., yoga, stretching) while watching content provided on a TV cannot see their own posture. Therefore, by using a camera to capture the user and displaying the captured image simultaneously with the content, the user can compare the correct posture in the content with the captured posture.

[0003] However, while the content is edited by experts and the screen movement is smooth, the problem is that the image of the user captured through the camera is not edited and the screen movement is not smooth.

[0004] In addition, when performing auto zoom or auto tracking through the camera, there was the inconvenience of the user having to operate it manually due to the fixed field of view of the camera.

[0005] In addition, in the case of object recognition technology using cameras, since it cannot predict user movements in advance when tracking simple motions, there is a problem where some areas are not displayed when the user suddenly changes their pose. The problem to be solved

[0006] The present disclosure is designed to improve upon the above-mentioned problems, and the purpose of the present disclosure is to provide an electronic device and a method for controlling the same that identify a display area of ​​an image based on the display area of ​​each of a plurality of captured images. means of solving the problem

[0007] The electronic device according to the present embodiment for achieving the above-mentioned purpose is a camera and

[0008] The apparatus includes a processor that tracks an object region containing a user object in an image captured through the camera and identifies a display region in the image captured based on the tracked object region, wherein the processor identifies a display region of the first image captured based on an object region identified in the first image captured, identifies a display region of the second image captured based on an object region identified in the second image captured, and identifies a display region of the third image captured based on the display region of the first image captured and the display region of the second image captured.

[0009] Meanwhile, if the size of the display area of ​​the second captured image is larger than the size of the display area of ​​the first captured image, the processor can identify the display area of ​​the third captured image based on the display area of ​​the second captured image.

[0010] Meanwhile, the processor can identify the object area based on the height of the user object in the captured image and identify the display area based on the height of the identified object area.

[0011] Meanwhile, the processor can identify a display area of ​​the third image when a display area larger than the size of the display area of ​​the first image is identified in a plurality of second images taken after the first image, which is a number greater than a threshold number. The third image may be an image taken after the plurality of second images.

[0012] Meanwhile, if the processor identifies a display area larger than the size of the display area of ​​the first image in a second image that is continuously captured at a threshold number or more, it can identify the display area of ​​the third image.

[0013] Meanwhile, the electronic device further includes a display, and the processor can control the display to display a screen in which a content image received from an external server is included in a first area and the identified display area is included in a second area.

[0014] Meanwhile, the processor can track a guide area including a guide object in the content image, and can identify a display area of ​​the third image based on the size information of the tracked guide area, the size information of the object area of ​​the first image, and the size information of the object area of ​​the second image.

[0015] Meanwhile, the processor can obtain first ratio information based on the size of a guide area identified in a first content image and the size of a guide area identified in a second content image, obtain second ratio information based on the size of an object area of ​​the first captured image and the size of an object area of ​​the second captured image, and identify a display area of ​​the third captured image based on the first ratio information and the second ratio information.

[0016] Meanwhile, the processor can identify the display area of ​​the third captured image based on the relatively larger ratio information among the first ratio information and the second ratio information.

[0017] Meanwhile, the processor can identify a display area to be displayed through the display in the captured image based on the resolution information of the display and the tracked object area.

[0018] Meanwhile, the method includes the steps of: tracking an object area containing a user object in an image and identifying a display area in the image based on the tracked object area; identifying a display area of ​​the first image based on the object area identified in the first image; identifying a display area of ​​the second image based on the object area identified in the second image; and identifying a display area of ​​the third image based on the display area of ​​the first image and the display area of ​​the second image.

[0019] Meanwhile, the step of identifying the display area of ​​the third captured image can identify the display area of ​​the third captured image based on the display area of ​​the second captured image when the size of the display area of ​​the second captured image is larger than the size of the display area of ​​the first captured image.

[0020] Meanwhile, the step of identifying a display area of ​​the first captured image and the step of identifying a display area of ​​the second captured image may identify the object area based on the height of the user object in the captured image, and identify the display area based on the height of the identified object area.

[0021] Meanwhile, the step of identifying the display area of ​​the third captured image can identify the display area of ​​the third captured image if a display area larger than the size of the display area of ​​the first captured image is identified in a plurality of second captured images taken after the first captured image, and the third captured image may be an image taken after the plurality of second captured images.

[0022] Meanwhile, the step of identifying the display area of ​​the third captured image can identify the display area of ​​the third captured image if a display area larger than the size of the display area of ​​the first captured image is identified in the second captured image, which is captured continuously at a threshold number or more.

[0023] Meanwhile, the control method may further include the step of displaying a screen in which a content image received from an external server is included in a first area and the identified display area is included in a second area.

[0024] Meanwhile, the control method further includes the step of tracking a guide area including a guide object in the content image, and the step of identifying a display area of ​​the third captured image can identify the display area of ​​the third captured image based on the size information of the tracked guide area, the size information of the object area of ​​the first captured image, and the size information of the object area of ​​the second captured image.

[0025] Meanwhile, the control method may further include the step of obtaining first ratio information based on the size of a guide area identified in a first content image and the size of a guide area identified in a second content image, and the step of obtaining second ratio information based on the size of an object area of ​​the first captured image and the size of an object area of ​​the second captured image, and the step of identifying a display area of ​​the third captured image may identify a display area of ​​the third captured image based on the first ratio information and the second ratio information.

[0026] Meanwhile, the step of identifying the display area of ​​the third captured image can identify the display area of ​​the third captured image based on the relatively larger ratio information among the first ratio information and the second ratio information.

[0027] Meanwhile, the steps of identifying the display area of ​​the first captured image, identifying the display area of ​​the second captured image, and identifying the display area of ​​the third captured image can identify the display area to be displayed through the display in the captured image based on the resolution information of the display and the tracked object area. Brief explanation of the drawing

[0028] Figure 1 is a diagram illustrating an electronic device for capturing images of a user. FIG. 2 is a block diagram illustrating an electronic device according to one embodiment of the present disclosure. Figure 3 is a block diagram illustrating the specific configuration of the electronic device of Figure 2. Figure 4 is a flowchart illustrating the action of tracking a user in a captured image. FIG. 5 is a diagram illustrating the operation of displaying an image of a user taking a first pose. FIG. 6 is a diagram illustrating the operation of displaying an image of a user taking a second pose. FIG. 7 is a flowchart for explaining the operation of identifying the size of a display area by comparing images of a user taking a first pose and a second pose. FIG. 8 is a diagram illustrating a display area identification operation according to one embodiment. FIG. 9 is a drawing for explaining a display area identification operation according to another embodiment. FIG. 10 is a diagram illustrating the operation of displaying a third captured image based on an identified display area. FIG. 11 is a flowchart illustrating the operation of identifying a display area based on a threshold time. FIG. 12 is a diagram illustrating the change in the display area over time by embodying the operation of FIG. 11. FIG. 13 is a flowchart illustrating the operation of identifying a display area based on a threshold number. FIG. 14 is a drawing for explaining the change in the display area over time by specifying the operation of FIG. 13. FIG. 15 is a flowchart illustrating the operation of identifying a display area based on content received from an external server. FIG. 16 is a flowchart illustrating the operation of considering ratio information to identify the size of the display area. FIG. 17 is a diagram illustrating the change in size between an image taking a first pose and an image taking a second pose. FIG. 18 is a diagram for explaining ratio information between images disclosed in FIG. 17. FIG. 19 is a diagram illustrating the operation of displaying an image by applying ratio information. FIG. 20 is a flowchart illustrating a method for controlling an electronic device according to one embodiment of the present disclosure. Specific details for implementing the invention

[0029] The present disclosure will be described in detail below with reference to the attached drawings.

[0030] The terms used in the embodiments of this disclosure have been selected to be as widely used as possible, taking into account their functions within this disclosure; however, these terms may vary depending on the intent of those skilled in the art, case law, the emergence of new technologies, etc. Additionally, in specific cases, terms have been arbitrarily selected by the applicant, and in such cases, their meanings will be described in detail in the relevant explanatory section of this disclosure. Therefore, terms used in this disclosure should be defined not merely by their names, but based on their meanings and the overall content of this disclosure.

[0031] In this specification, expressions such as “have,” “may have,” “include,” or “may include” indicate the presence of such features (e.g., numerical values, functions, operations, or components such as parts) and do not exclude the presence of additional features.

[0032] The expression "at least one of A or / and B" should be understood as representing either "A" or "B" or "A and B".

[0033] Expressions such as "first," "second," "first," or "second" used in this specification may modify various components regardless of order and / or importance, and are used only to distinguish one component from another and do not limit said components.

[0034] Where it is stated that a component (e.g., Component 1) is "(operatively or communicatively) coupled with / to" or "connected to" another component (e.g., Component 2), it should be understood that the component may be directly connected to the other component or connected through the other component (e.g., Component 3).

[0035] The singular expression includes the plural expression unless the context clearly indicates otherwise. In this application, terms such as "comprising" or "consisting of" are intended to specify the existence of the features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0036] In the present disclosure, a "module" or "part" performs at least one function or operation and may be implemented in hardware or software, or a combination of hardware and software. Additionally, a plurality of "modules" or a plurality of "parts" may be integrated into at least one module and implemented by at least one processor (not shown), except for a "module" or "part" that needs to be implemented in specific hardware.

[0037] In this specification, the term "user" may refer to a person using an electronic device or a device using an electronic device (e.g., an artificial intelligence electronic device).

[0038] An embodiment of the present disclosure will be described in more detail below with reference to the attached drawings.

[0039] Figure 1 is a diagram illustrating an electronic device for capturing images of a user.

[0040] Referring to FIG. 1, the electronic device (100) may include a camera (110). The electronic device (100) may capture an image of a user (1000) located in front of the electronic device (100) through the camera (110). The electronic device (100) may additionally include a display (140) and may display the captured image on the display (140). Here, the captured image may include a user object (image form) corresponding to the user (1000).

[0041] The electronic device (100) may not display all areas of the captured image on the display (140). This is because the resolution information captured by the camera (110) and the resolution information of the display (140) may differ. Therefore, the electronic device (100) may crop a portion of the acquired captured image. Additionally, the electronic device (100) may display only the cropped portion of the entire area of ​​the captured image on the display (140). Here, the cropped portion may be the display area.

[0042] When the user (1000) changes their pose, the user object included in the captured image may also change. Accordingly, the electronic device (100) may change the display area of ​​the captured image displayed on the display (140) based on the size of the changed user object. However, it may be difficult to change the display area in real time for an image captured in real time. Therefore, as shown in FIG. 6, in some situations, a problem may occur in which a part of the user object is not displayed on the display (140).

[0043] FIG. 2 is a block diagram illustrating an electronic device according to one embodiment of the present disclosure.

[0044] Referring to FIG. 2, the electronic device (100) may be composed of a camera (110) and a processor (120).

[0045] An electronic device (100) according to various embodiments of the present specification may include, for example, at least one of a smartphone, tablet PC, mobile phone, video phone, e-book reader, desktop PC, laptop PC, netbook computer, workstation, server, PDA, PMP (portable multimedia player), MP3 player, medical device, camera, or wearable device. The wearable device may include at least one of an accessory type (e.g., a watch, ring, bracelet, anklet, necklace, glasses, contact lenses, or head-mounted device (HMD)), a fabric or clothing integrated type (e.g., electronic clothing), a body-attached type (e.g., skin pad or tattoo), or a bio-implantable circuit. In some embodiments, the electronic device (100) may include, for example, at least one of a television, a DVD (digital video disk) player, audio, a refrigerator, an air conditioner, a vacuum cleaner, an oven, a microwave, a washing machine, an air purifier, a set-top box, a home automation control panel, a security control panel, a media box (e.g., Samsung HomeSync™, Apple TV™, or Google TV™), a game console (e.g., Xbox™, PlayStation™), an electronic dictionary, an electronic key, a camcorder, or a digital photo frame.

[0046] Meanwhile, if the electronic device (100) corresponds to a display device, the electronic device (100) may include various devices including a display. The electronic device (100) may be an electronic whiteboard, TV, desktop PC, laptop, smartphone, tablet PC, server, etc. Meanwhile, the above-described examples are merely illustrative examples for explaining electronic devices and are not necessarily limited to the devices described above.

[0047] The camera (110) is configured to capture an object and generate an image, wherein the image includes both video and still images. The camera (110) can acquire an image of at least one external device and can be implemented as a camera, lens, infrared sensor, etc.

[0048] The camera (110) may include a lens and an image sensor. The types of lenses include general-purpose lenses, wide-angle lenses, zoom lenses, etc., and may be determined according to the type, characteristics, and usage environment of the electronic device (100). As an image sensor, a Complementary Metal Oxide Semiconductor (CMOS) and a Charge Coupled Device (CCD) may be used.

[0049] The camera (110) outputs incident light as an image signal. Specifically, the camera (110) may be equipped with a lens, a pixel, and an AD converter. The lens collects light from a subject and forms an optical image in the imaging area, and the pixel can output the light collected through the lens as an analog image signal. The AD converter can convert the analog image signal into a digital image signal and output it. In particular, the camera (110) is positioned to capture the front direction of the electronic device (100), and can capture a user present in front of the electronic device (100) to generate an image.

[0050] Meanwhile, in describing the electronic device (100) according to one embodiment of the present disclosure, it is described as having one camera (110), but in actual implementation, multiple cameras may be arranged. The electronic device (100) may be equipped with multiple cameras, and the images received through the multiple cameras can be combined to identify the user's head posture. Using multiple cameras rather than one camera allows for more precise analysis of three-dimensional movements, which can be effective in identifying the user's head posture.

[0051] The processor (120) can perform overall control operations of the electronic device. Specifically, the processor (120) functions to control the overall operation of the electronic device.

[0052] The processor (120) may be implemented as a digital signal processor (DSP) that processes digital signals, a microprocessor, or a time controller (TCON). However, it is not limited thereto, and may include or be defined by one or more of a central processing unit (CPU), a micro controller unit (MCU), a micro processing unit (MPU), a controller, an application processor (AP), a graphics-processing unit (GPU), a communication processor (CP), or an ARM processor. Additionally, the processor (120) may be implemented as a System on Chip (SoC) or a large-scale integration (LSI) with a built-in processing algorithm, or may be implemented in the form of a Field Programmable Gate Array (FPGA). Furthermore, the processor (120) can perform various functions by executing computer executable instructions stored in memory.

[0053] The processor (120) can track an object region containing a user object in an image captured through a camera (110) and identify a display region in the image captured based on the tracked object region, and the processor (120) can identify a display region of the first image captured based on the object region identified in the first image captured, identify a display region of the second image captured based on the object region identified in the second image captured, and identify a display region of the third image captured based on the display region of the first image captured and the display region of the second image captured.

[0054] Here, the processor (120) can receive a captured image of the front of the electronic device (100) through the camera (110). The processor (120) can identify whether the received captured image contains a user object corresponding to the user (1000). If the received captured image contains a user object, the processor (120) can identify an object area containing the user object in the received captured image. Here, an object area may refer to an area among all areas of the captured image where the user object is displayed. The processor (120) can identify the object area based on the location information of the user object in the captured image and can track the changing object area in real time. If the user (1000) changes their pose or moves their position, the object area may also change. And, the processor (120) can track the changed object area in real time.

[0055] Here, the processor (120) can identify a display area based on the identified object area. A display area may refer to an area of ​​the entire area of ​​the captured image to be displayed on a display. Here, the display may be a display (140) of an electronic device (100), and may be a display of an external device according to an implementation example.

[0056] As described in FIG. 1, since the resolution of the camera (110) and the resolution of the display (140) may differ, the processor (120) may change the size of the captured image or crop it to obtain an image to display. An image acquisition operation according to one embodiment of the present disclosure is described based on an image obtained through a cropping operation. The processor (120) may remove a part of the captured image to display only the remaining area, or specify a part of the captured image to display only a part of the area.

[0057] Meanwhile, the processor (120) can identify a user object in the first captured image and identify an object area based on the identified user object. And, the processor (120) can identify a display area corresponding to the first captured image based on the identified object area.

[0058] In addition, likewise, the processor (120) can identify a user object in the second captured image and identify an object region based on the identified user object. And, the processor (120) can identify a display region corresponding to the second captured image based on the identified object region.

[0059] Here, the first captured image and the second captured image may be images received as consecutive frames. That is, the second captured image may be received consecutively after the first captured image is received. However, for convenience of explanation, in the present disclosure, the first captured image refers to an image of a user (1000) taking a first pose, and the second captured image may refer to an image of a user (1000) taking a second pose after taking the first pose.

[0060] In order to distinguish between the object area and the display area of ​​the first captured image and the second captured image, the object area of ​​the first captured image may be described as the first object area and the object area of ​​the second captured image may be described as the second object area. Additionally, the display area of ​​the first captured image may be described as the first display area and the display area of ​​the second captured image may be described as the second display area.

[0061] Here, the object area and display area identification operation corresponding to the first captured image and the second captured image is described later in FIGS. 4 to 6.

[0062] Here, the processor (120) can identify a display area of ​​a third captured image based on a display area of ​​a first captured image (first display area) and a display area of ​​a second captured image (second display area). Here, the third captured image may refer to an image received after the first captured image and the second captured image have been received.

[0063] If the size of the first display area and the size of the second display area are different, the processor (120) can determine which display area to set the display area of ​​the third captured image based on the size of the display area.

[0064] Here, the processor (120) can also determine the display area of ​​the third captured image itself. However, the display area of ​​the third captured image can be pre-set based on the display areas obtained from the first captured image and the second captured image. This is because if the display area is changed in real time, the screen transition may not be smooth and may cause inconvenience to the user. Therefore, the processor (120) can determine the display area of ​​the current captured image based on the display area of ​​the past captured image. However, even here, the display area for the current captured image is not determined anew, and the processor (120) can first set the display area determined based on the past captured image to display the image and then identify the display area of ​​the subsequently received captured image anew.

[0065] To clearly distinguish this, the display area may be divided into an expected display area and a set display area. The expected display area may refer to a display area identified based on user objects included in the received captured image. The set display area may refer to an area to be displayed on the display (140). Therefore, the expected display area and the set display area may differ even in captured images at the same point in time.

[0066] For example, in the embodiment of FIG. 5, the expected display area and the set display area may be the same. In this case, the image displayed on the display (140) may include all user objects. In the first captured image, it is assumed that the expected display area is the set display area.

[0067] However, in the embodiment of FIG. 6, the expected display area (623) and the set display area (523) may be different. This is because the expected display area (623) is determined based on the current object area (622), while the set display area (523) is obtained based on a previously received captured image (520).

[0068] As explained earlier, the reason for differentiating the expected display area from the set display area is the necessity of maintaining the existing display area to ensure smooth screen transitions. Additionally, it is necessary to prevent screen lag caused by processing time when changing the display area in real-time.

[0069] Accordingly, the processor (120) can identify a set display area of ​​a third image based on the expected display area of ​​the first image and the expected display area of ​​the second image.

[0070] Meanwhile, in order to change the setting display area, an expected display area larger than the size of the previously set setting display area may need to be identified. As described later in FIG. 7, since the processor (120) identifies the setting display area based on the expected display area larger, the setting display area will not decrease but will always increase when multiple captured images are continuously received.

[0071] To prevent such inconsistent behavior, the processor (120) may change from the set display area to the expected display area (the expected display area identified in the currently received captured image) after the threshold time has elapsed. For example, in the captured image acquired at 11 seconds in FIG. 12, the set display area may be displayed smaller even though the expected display area becomes smaller.

[0072] The display area is divided into the expected display area and the set display area, but for the convenience of explanation, it may be referred to as the display area.

[0073] Meanwhile, if the size of the (expected) display area of ​​the second captured image is larger than the size of the (expected) display area of ​​the first captured image, the processor (120) can identify the (set) display area of ​​the third captured image based on the (expected) display area of ​​the second captured image.

[0074] The purpose of the electronic device (100) of the present disclosure is to solve the problem where a part of the user's (1000) body is not displayed on the display (140) when the user (1000) suddenly changes pose. Accordingly, if the display area of ​​the second captured image is smaller than the size of the display area of ​​the first captured image, the processor (120) may not need to change the currently set display area. This is because if the display area is maintained in a large state, the screen is not interrupted from the user's perspective, and all parts of the user's (1000) body can also be displayed. Accordingly, the processor (120) may change the existing set display area only when the size of the (expected) display area of ​​the second captured image is larger than the size of the (expected) display area of ​​the first captured image. A detailed explanation related to this is described later in FIGS. 7 to 10.

[0075] Meanwhile, the processor (120) can identify an object area based on the height of the user object in the captured image and identify a (predicted) display area based on the height of the identified object area.

[0076] The object area may contain horizontal and vertical information. However, the most important factor in identifying the display area may be the vertical information. Here, the vertical information may be height information. Accordingly, the processor (120) can determine the height of the object area based on the height of the user object, and determine the height of the display area based on the determined height of the object area.

[0077] Here, the processor (120) can determine the width of the display area based on horizontal information. It is assumed that the user assumes a pose of moving repeatedly left and right. Here, the processor (120) can identify the range of the user's left and right movements and can identify the display area based on the identified range.

[0078] Meanwhile, the processor (120) can identify a (set) display area of ​​a third image when a (predicted) display area larger than the size of the (predicted) display area of ​​the first image is identified in a plurality of second images taken after a threshold number or more. The third image may be an image taken after the plurality of second images.

[0079] Here, a plurality of second imaging images exceeding a threshold number may include not only continuous cases but also discontinuous cases.

[0080] Here, the second captured image may be multiple images. Specifically, the processor (120) can identify the (set) display area of ​​the third captured image by comparing the (expected) display area of ​​the first captured image with the (expected) display area of ​​multiple second captured images. The reason the processor (120) considers multiple second captured images is to identify whether a predetermined event has occurred more than a threshold number. If the display area is identified based on all actions of the user (1000), the amount of data processing increases and screen switching slows down, which may cause a delay. Therefore, the processor (120) can change the (set) display area only when a predetermined event occurs.

[0081] A preset event according to one embodiment may mean that the size of the (expected) display area of ​​the second captured image is larger than the size of the (expected) display area of ​​the first captured image. A specific explanation related to this is described later in FIGS. 7 to 10.

[0082] According to another embodiment, a pre-set event may mean a case where the size of the (expected) display area of ​​the second captured image is larger than the size of the (expected) display area of ​​the first captured image and is acquired continuously for a threshold time or longer. Meanwhile, if the processor (120) identifies a (expected) display area larger than the size of the (expected) display area of ​​the first captured image in the second captured images that are captured continuously for a threshold number or more, the processor (120) may identify the display area of ​​the third captured image. A detailed explanation related to this will be described later in FIGS. 11 and 12.

[0083] According to another embodiment, a preset event may mean a case where the size of the (expected) display area of ​​the second captured image is larger than the size of the (expected) display area of ​​the first captured image and is identified by more than a threshold number. Here, the operation of being identified by more than a threshold number does not need to be continuous. A detailed explanation will be provided later in FIGS. 13 and 14.

[0084] Meanwhile, the electronic device (100) further includes a display (140), and the processor (120) can control the display (140) to display a screen in which a content image received from an external server is included in a first area and an identified (setting) display area is included in a second area.

[0085] Here, the content may be implemented in a form that is received in real time or in a form that is stored in advance in the memory (150) of the electronic device (100). An operation to additionally display the content is described later in FIGS. 5, 6, and 10.

[0086] Meanwhile, the processor (120) can track a guide area including a guide object in a content image, and can identify a (set) display area of ​​a third image based on the size information of the tracked guide area, the size information of the object area of ​​the first image, and the size information of the object area of ​​the second image. Here, the guide object may refer to an object that takes a specific pose in the content so that the user (1000) can see and follow it. The guide object may be an image of a real person, or, depending on the implementation example, a virtual 3D character. The guide area may refer to the area where the guide object is located within the entire area of ​​the content image. The guide area may be the guide area (512) and guide area (612) of FIG. 17.

[0087] Meanwhile, the processor (120) can obtain first ratio information based on the size of the guide area identified in the first content image and the size of the guide area identified in the second content image, obtain second ratio information based on the size of the object area of ​​the first captured image and the size of the object area of ​​the second captured image, and identify the display area of ​​the third captured image based on the first ratio information and the second ratio information.

[0088] Meanwhile, the processor (120) can identify the display area of ​​the third captured image based on the relatively larger ratio information among the first ratio information and the second ratio information.

[0089] Meanwhile, a detailed description regarding the operation of obtaining ratio information based on an object area is described later in FIGS. 16 to 19. Depending on the implementation example, it may be implemented in a form of obtaining ratio information based on a display area, and a detailed description is described later in FIG. 15.

[0090] Meanwhile, the processor (120) can identify a display area (set) to be displayed through the display (140) in the captured image based on the resolution information of the display (140) and the tracked object area.

[0091] The resolution of the display (140) and the resolution of the captured image may differ. Therefore, there may be cases where the processor (120) cannot display the captured image on the display (140) as is. Here, the processor (120) can crop the captured image to match the resolution information of the display (140). Specifically, the processor (120) can identify a (setting) display area to be displayed on the display (140) based on the resolution information of the display (140) and the size information of the object area, and control the display (140) to display image information corresponding to the identified (setting) display area.

[0092] Meanwhile, when a (setting) display area corresponding to the third captured image is identified, the processor (120) can control the camera (110) based on the identified (setting) display area. When the entire outline of the user object is included in the captured image obtained from the camera (110), the processor (120) can determine whether to zoom in or zoom out based on the identified (setting) display area. The zoom in or zoom out operation can be determined based on ratio information.

[0093] For example, if the changed object area (1922) described in FIG. 19 corresponds to a size smaller than a certain level in the received captured image (1920), the processor (120) can control the camera (110) to zoom in.

[0094] For example, if the modified object area (1922) described in FIG. 19 is not entirely included in the received captured image (1920), the processor (120) can control the camera (110) to zoom out.

[0095] Here, zooming for zoom in and zoom out has been described, but panning or tilting actions can be applied in the same way.

[0096] Meanwhile, the electronic device (100) initiates an operation to distinguish between an object area and a display area based on a user object, analyze a past (expected) display area, and determine a future (set) display area. By doing so, the electronic device (100) does not need to process the (set) display area in real time, thereby providing a smooth screen transition service to the user and solving the problem of parts of the user's body not being displayed on the screen.

[0097] Additionally, since the electronic device (100) can automatically control the camera (110) based on the changing (setting) display area, the user does not have to manually adjust the angle and magnification of the camera.

[0098] Meanwhile, although only a simple configuration constituting the electronic device (100) has been illustrated and described above, various additional configurations may be provided during implementation. This will be explained below with reference to FIG. 3.

[0099] Figure 3 is a block diagram illustrating the specific configuration of the electronic device of Figure 2.

[0100] Referring to FIG. 3, the electronic device (100) may be composed of a camera (110), a processor (120), a communication interface (130), a display (140), a memory (150), a user interface (160), an input / output interface (170), a microphone (180), and a speaker (190).

[0101] Meanwhile, regarding the operations of the camera (110) and processor (120) that are identical to those previously described, a redundant description is omitted.

[0102] The communication interface (130) is a configuration that performs communication with various types of external devices according to various types of communication methods. The communication interface (130) includes a Wi-Fi module, a Bluetooth module, an infrared communication module, and a wireless communication module, etc. Here, each communication module can be implemented in the form of at least one hardware chip.

[0103] Wi-Fi modules and Bluetooth modules perform communication using the Wi-Fi and Bluetooth methods, respectively. When using a Wi-Fi or Bluetooth module, various connection information, such as SSID and session key, is transmitted and received first; after establishing a communication connection using this information, various types of information can be transmitted and received.

[0104] The infrared communication module performs communication according to infrared communication (IrDA, Infrared Data Association) technology, which uses infrared rays located between visible light and millimeter waves to wirelessly transmit data over short distances.

[0105] In addition to the communication method described above, the wireless communication module may include at least one communication chip that performs communication according to various wireless communication standards such as Zigbee, 3G (3rd Generation), 3GPP (3rd Generation Partnership Project), LTE (Long Term Evolution), LTE-A (LTE Advanced), 4G (4th Generation), and 5G (5th Generation).

[0106] In addition, the communication interface (130) may include at least one of a wired communication module that performs communication using a LAN (Local Area Network) module, an Ethernet module, a pair cable, a coaxial cable, a fiber optic cable, or a UWB (Ultra Wide-Band) module.

[0107] According to one example, the communication interface (130) may use the same communication module (e.g., Wi-Fi module) to communicate with external devices such as a remote control and an external server.

[0108] According to other examples, the communication interface (130) may use different communication modules (e.g., Wi-Fi modules) to communicate with external devices such as a remote control and external servers. For example, the communication interface (130) may use at least one of an Ethernet module or a WiFi module to communicate with an external server, and may use a BT module to communicate with an external device such as a remote control. However, this is merely one embodiment, and the communication interface (130) may use at least one of various communication modules when communicating with multiple external devices or external servers.

[0109] The display (140) can be implemented as various types of displays such as an LCD (Liquid Crystal Display), an OLED (Organic Light Emitting Diodes) display, and a PDP (Plasma Display Panel). The display (140) may also include a driving circuit, a backlight unit, etc., which can be implemented in forms such as an a-si TFT, an LTPS (low temperature poly silicon) TFT, and an OTFT (organic TFT). Meanwhile, the display (140) can be implemented as a touch screen combined with a touch sensor, a flexible display, a 3D display, etc.

[0110] In addition, according to one embodiment of the present disclosure, the display (140) may include not only a display panel that outputs an image, but also a bezel that houses the display panel. In particular, according to one embodiment of the present disclosure, the bezel may include a touch sensor (not shown) for detecting user interaction.

[0111] The memory (150) may be implemented as internal memory such as ROM (e.g., EEPROM (electrically erasable programmable read-only memory)) or RAM included in the processor (120), or as memory separate from the processor (120). In this case, the memory (150) may be implemented in the form of memory embedded in the electronic device (100) or in the form of memory that can be attached to the electronic device (100) depending on the purpose of data storage. For example, data for operating the electronic device (100) may be stored in memory embedded in the electronic device (100), and data for the expansion function of the electronic device (100) may be stored in memory that can be attached to the electronic device (100).

[0112] Meanwhile, the memory embedded in the electronic device (100) may be implemented as at least one of volatile memory (e.g., DRAM (dynamic RAM), SRAM (static RAM), or SDRAM (synchronous dynamic RAM), non-volatile memory (e.g., OTPROM (one time programmable ROM), PROM (programmable ROM), EPROM (erasable and programmable ROM), EEPROM (electrically erasable and programmable ROM), mask ROM, flash ROM, flash memory (e.g., NAND flash or NOR flash), hard drive, or solid state drive (SSD), and the memory that is detachable from the electronic device (100) may be implemented in the form of a memory card (e.g., CF (compact flash), SD (secure digital), Micro-SD (micro secure digital), Mini-SD (mini secure digital), xD (extreme digital), MMC (multi-media card), etc.), or external memory that can be connected to a USB port (e.g., USB memory).

[0113] The user interface (160) may be implemented as a device such as a button, touchpad, mouse, and keyboard, or as a touch screen capable of performing the aforementioned display function and operation input function. Here, the button may be a various type of button, such as a mechanical button, touchpad, or wheel, formed in any area such as the front, side, or back of the main body exterior of the electronic device (100).

[0114] The input / output interface (170) may be any one of the following interfaces: HDMI (High Definition Multimedia Interface), MHL (Mobile High-Definition Link), USB (Universal Serial Bus), DP (Display Port), Thunderbolt, VGA (Video Graphics Array) port, RGB port, D-SUB (D-subminiature), DVI (Digital Visual Interface).

[0115] The input / output interface (170) can input and output at least one of audio and video signals.

[0116] Depending on the implementation example, the input / output interface (170) may include separate ports for inputting and outputting only audio signals and for inputting and outputting only video signals, or it may be implemented as a single port for inputting and outputting both audio and video signals.

[0117] The electronic device (100) may further include a microphone (180). The microphone is configured to receive user voice or other sounds and convert them into audio data.

[0118] The microphone (180) can receive the user's voice when active. For example, the microphone (180) may be formed integrally on the upper side, front side, or side side of the electronic device (100). The microphone (180) may include various configurations such as a microphone for collecting analog user voice, an amplifier circuit for amplifying the collected user voice, an A / D conversion circuit for sampling the amplified user voice and converting it into a digital signal, and a filter circuit for removing noise components from the converted digital signal.

[0119] The electronic device (100) may include a speaker (190). The speaker (190) may be a component that outputs various audio data processed at an input / output interface, as well as various notification sounds or voice messages.

[0120] Figure 4 is a flowchart illustrating the action of tracking a user in a captured image.

[0121] Referring to FIG. 4, the electronic device (100) can acquire a captured image from the camera (110) (S405). Here, the captured image may be an image of the front of the electronic device (100). Generally, as shown in FIG. 1, the user (1000) may be positioned in front of the electronic device (100), so the captured image may be an image for capturing the user.

[0122] And, the electronic device (100) can identify a user object in the acquired captured image (S410). The user object may mean a person object, and the electronic device (100) can identify whether the captured image contains a user object.

[0123] And, when the electronic device (100) identifies a user object in a captured image, the electronic device (100) can identify an object region containing the identified user object (S415).

[0124] And, the electronic device (100) can track the identified object area (S420). Here, tracking may mean tracking the object area to check for changes in the object area.

[0125] And, the electronic device (100) can identify a display area based on the object area tracked in the captured image (S425). The display area may refer to an area displayed on the display (140) of the electronic device (100). The entire captured image is not displayed on the display (140) of the electronic device (100), and only the portion of the entire captured image corresponding to the display area may be displayed on the display (140).

[0126] FIG. 5 is a diagram illustrating the operation of displaying an image of a user taking a first pose.

[0127] Referring to FIG. 5, the electronic device (100) can capture a user (1000) through a camera (110) and obtain a captured image (520) in which the user is in a first pose. Then, the electronic device (100) can identify a user object (521) in the captured image (520). Then, the electronic device (100) can identify an object area (522) based on the identified user object (521). Then, the electronic device (100) can identify a display area (523) based on the identified object area (522).

[0128] And, the electronic device (100) can display the identified display area (523) on the display (140) of the electronic device (100). The electronic device (100) can divide the entire area of ​​the display (140) into a first area (141) and a second area (142).

[0129] And, the electronic device (100) can display content received from an external server in a first area (141). The content may include a guide object (511). And, the electronic device (100) can display an identified display area (523) among captured images (520) in a second area (142). The image displayed in the second area (142) may include a user object (521).

[0130] That is, the electronic device (100) displays the guide object (511) in the first area (141) and the user object (521) in the second area (142) at the same time, so that the user (1000) can easily follow the pose of the guide object (511) while looking directly at the display (140).

[0131] FIG. 6 is a diagram illustrating the operation of displaying an image of a user taking a second pose.

[0132] Referring to FIG. 6, the electronic device (100) can capture a user (1000) through a camera (110) and obtain a captured image (620) of the user taking a second pose. It is assumed that the user takes a first pose and then takes a second pose.

[0133] The electronic device (100) can identify a user object (621) in a captured image (620). And, the electronic device (100) can identify an object area (622) based on the identified user object (621).

[0134] Here, the display area (523) may correspond to the display area of ​​FIG. 5. Since it may take processing time to identify the display area (523), if the user (1000) suddenly transitions from a first pose to a second pose, the display area (523) may remain unchanged. It is assumed that the user taking the second pose is captured in a larger area than the first pose. The object area (622) corresponding to the second pose may be larger than the display area (523). Here, some area (624) may extend beyond the display area (523). Some area (624) extending beyond the display area (523) may not be displayed on the display (140) of the electronic device (100).

[0135] And, the electronic device (100) can display the identified display area (523) on the display (140) of the electronic device (100). The content may include a guide object (611). And, the identified display area (523) among the captured image (620) may be displayed in the second area (142) of the electronic device (100). The image displayed in the second area (142) may include a user object (621). However, the image displayed in the second area (142) may not include some area (624) of the captured image (620).

[0136] However, as time passes, the electronic device (100) can identify the display area (623) of the captured image (620) based on the object area (622) identified in the captured image (620).

[0137] FIG. 7 is a flowchart for explaining the operation of identifying the size of a display area by comparing images of a user taking a first pose and a second pose.

[0138] Referring to FIG. 7, the electronic device (100) can identify an object region in a first captured image (S705). Here, the first captured image may refer to an image of a user (1000) taking a first pose. The first captured image may correspond to the captured image (520) of FIG. 5. Then, the electronic device (100) can identify a display area of ​​the first captured image based on the object region identified in the first captured image (S710). Then, the electronic device (100) can obtain size information of the identified display area of ​​the first captured image. Here, the display area of ​​the first captured image may correspond to the display area (523) of the captured image (520) in FIG. 5.

[0139] And, the electronic device (100) can identify an object region in the second captured image (S715). Here, the second captured image may refer to an image of a user (1000) taking a second pose. The second captured image may correspond to the captured image (620) of FIG. 6. And, the electronic device (100) can identify a display area of ​​the second captured image based on the object region identified in the second captured image (S720). And, the electronic device (100) can obtain size information of the identified display area of ​​the second captured image. Here, the display area of ​​the second captured image may correspond to the display area (823) of FIG. 8 and the display area (923) of FIG. 9.

[0140] And, the electronic device (100) can determine whether the size of the display area of ​​the second captured image is larger than the size of the display area of ​​the first captured image (S725).

[0141] If the size of the display area of ​​the second captured image is not larger than the size of the display area of ​​the first captured image, the electronic device (100) can identify the size of the display area of ​​the third captured image based on the size of the display area of ​​the first captured image (S730). That is, the size of the display area of ​​the third captured image may be the same as the size of the display area obtained from the first captured image. Here, the third captured image may refer to an image captured at a point in time after the second captured image.

[0142] If the size of the display area of ​​the second captured image is larger than the size of the display area of ​​the first captured image, the electronic device (100) can identify the size of the display area of ​​the third captured image based on the size of the display area of ​​the second captured image (S735). That is, the size of the display area of ​​the third captured image may be the same as the size of the display area obtained from the second captured image.

[0143] The electronic device (100) can maintain the larger display area between the size of the display area obtained from the first captured image and the size of the display area obtained from the second captured image. That is, a new captured image can be obtained over time, and the pose of the user (1000) in the new captured image can be varied. However, even if the pose of the user (1000) is varied, the electronic device (100) can maintain the largest display area. And, the electronic device (100) can display the image corresponding to the largest display area among the multiple captured images in the second area (142).

[0144] If the display area is maintained at a large size even when the captured image is changed, the problem of some area (624) of FIG. 6 not being displayed on the display (140) can be solved.

[0145] FIG. 8 is a diagram illustrating a display area identification operation according to one embodiment.

[0146] Referring to FIG. 8, the captured image (820) may include a user object (821). The electronic device (100) may identify an object region (822) based on the user object (821). Here, the captured image (820) may be an image of a user (1000) taking a second pose. The captured image (820) may include a user object (821), and the electronic device (100) may identify an object region (822) based on the user object (821).

[0147] Here, it is assumed that the size of the display area (823) of the captured image (520) is larger than the size of the display area (820). When the electronic device (100) identifies the display area of ​​the captured image (820), the display area can be extended in all directions—up, down, left, and right—from the object area (822).

[0148] If some area (824) of the image corresponding to the user object (821) included in the captured image (820) is not displayed on the display (140), the electronic device (100) can expand the display area by extending in all directions up, down, left, and right. The display area (823) of the captured image (820) corresponding to the second pose may be an area extended in all directions up, down, left, and right compared to the display area (523) of the captured image (520) corresponding to the first pose.

[0149] FIG. 9 is a drawing for explaining a display area identification operation according to another embodiment.

[0150] Referring to FIG. 9, the captured image (920) may include a user object (921). The electronic device (100) may identify an object region (922) based on the user object (921). Here, the captured image (920) may be an image of a user (1000) taking a second pose. The captured image (920) may include a user object (921), and the electronic device (100) may identify an object region (922) based on the user object (921).

[0151] Here, it is assumed that the size of the display area (923) of the captured image (520) is larger than the size of the display area (920). When the electronic device (100) identifies the display area of ​​the captured image (920), the display area can be extended in all directions—up, down, left, and right—from the object area (922).

[0152] If some area (924) of the image corresponding to the user object (921) included in the captured image (920) is not displayed on the display (140), the electronic device (100) may expand the display area by extending the upward direction. The display area (923) of the captured image (920) corresponding to the second pose may be an area extended in the upward direction compared to the display area (523) of the captured image (520) corresponding to the first pose.

[0153] Unlike FIG. 8, FIG. 9 describes that the display area is extended only in the upward direction. This is because, generally, the images capturing the user (1000) often have the floor fixed, so there is no need to unnecessarily extend the display area in the downward direction. Therefore, the display area is fixed to the floor in the downward direction and can be extended only in the upward direction due to the user's movement.

[0154] FIG. 10 is a diagram illustrating the operation of displaying a third captured image based on an identified display area.

[0155] Referring to FIG. 10, an electronic device (100) can capture a user (1000) taking a second pose through a camera (110) and obtain a captured image (1020). The captured image (1020) may include a user object (1021). The electronic device (100) can identify an object area (1022) based on the user object (1021). And, based on the identified object area (1022), a display area (1023) can be identified.

[0156] Unlike in FIG. 6, in the embodiment of FIG. 10, it is assumed that a certain amount of time has elapsed while the user (1000) is in a second pose.

[0157] The electronic device (100) can display an image corresponding to an identified display area (1023) among the captured images (1020) in a second area (142). Here, a user object (1021) may be displayed in the second area (142). Meanwhile, a guide object (1011) may be displayed in the first area (141).

[0158] FIG. 11 is a flowchart illustrating the operation of identifying a display area based on a threshold time.

[0159] Referring to FIG. 11, the electronic device (100) can identify a display area of ​​a first captured image and a display area of ​​a second captured image (S1105). Step S1105 may correspond to steps S705 to S720 of FIG. 7.

[0160] The electronic device (100) can determine whether the size of the display area of ​​the second captured image is larger than the size of the display area of ​​the first captured image (S1110). Step S1110 may correspond to step S725 of FIG. 7.

[0161] If the size of the display area of ​​the second captured image is not larger than the size of the display area of ​​the first captured image, the electronic device (100) can identify the size of the display area of ​​the third captured image based on the size of the display area of ​​the first captured image (S1115). That is, the size of the display area of ​​the third captured image may be the same as the size of the display area obtained from the first captured image. Here, the third captured image may refer to an image captured at a point in time after the second captured image.

[0162] If the size of the display area of ​​the second captured image is larger than the size of the display area of ​​the first captured image, the electronic device (100) can determine whether the display area of ​​the second captured image is identified for more than a threshold time (S1120). Here, if the display area of ​​the second captured image is not identified for more than a threshold time, the electronic device (100) can identify the size of the display area of ​​the third captured image based on the size of the display area of ​​the first captured image (S1115).

[0163] When the display area of ​​the second captured image is identified for more than a threshold time, the electronic device (100) can identify the size of the display area of ​​the third captured image based on the size of the display area of ​​the second captured image (S1125). That is, the size of the display area of ​​the third captured image may be the same as the size of the display area obtained from the second captured image.

[0164] FIG. 12 is a diagram illustrating the change in the display area over time by embodying the operation of FIG. 11.

[0165] Referring to FIG. 12, the table (1205) represents a series of multiple captured images. It is assumed that the series of multiple captured images includes a user object taking a first pose from 1 second to 4 seconds, a user object taking a second pose from 5 seconds to 8 seconds, and a user object taking a first pose from 9 seconds to 13 seconds.

[0166] Here, the image captured at 1 second is in the first pose, so it can correspond to the image captured in FIG. 5 (520).

[0167] And, since the image captured at 5 seconds is in the second pose, it can correspond to the image (620) of FIG. 6. Since the first pose is in the pose immediately before the image captured at 5 seconds, the display area (523) can be maintained as is. Therefore, some area (624) of the image (620) may not be displayed on the display (140). If the display area is changed immediately as the pose changes, the user (1000) viewing the display (140) may feel dizzy. Therefore, the electronic device (100) can delay the change of the display area for a threshold time.

[0168] However, when a threshold time (e.g., 2 seconds) has elapsed, the electronic device (100) can change to a display area (1023) corresponding to a second pose. Thus, the captured image (1020) acquired at 7 seconds can be displayed in the display area without the object area (1022) being omitted, based on the display area (1023).

[0169] Meanwhile, the captured image (1220) obtained at 9 seconds may include a user object (1221) that has taken the first pose from the second pose. Here, since it is the moment when the first pose is changed immediately from the second pose, the display area corresponding to the changed first pose may not change immediately. Therefore, in the captured image (1220), the object area is identified as (1222), but the display area may remain as the display area (1023) corresponding to the captured image (1020).

[0170] However, when a threshold time (e.g., 2 seconds) has elapsed, the electronic device (100) can be changed to a display area (523) corresponding to the first pose. Accordingly, the captured image (520) acquired at 11 seconds can be displayed on the display (140) based on the display area (523). This is because if the currently set display area (1023) is maintained as is even after the threshold time has elapsed, the display area is always determined to expand, and a problem may arise where it does not match the content image displayed on the display (140).

[0171] FIG. 13 is a flowchart illustrating the operation of identifying a display area based on a threshold number.

[0172] Referring to FIG. 13, steps S1305, S1310, and S1315 may correspond to steps S1105, S1110, and S1115 of FIG. 11. Therefore, a description that overlaps with this is omitted.

[0173] If the size of the display area of ​​the second captured image is larger than the size of the display area of ​​the first captured image, the electronic device (100) can determine whether the display area of ​​the second captured image is identified by more than a threshold number (S1320). Identifying the display area by more than a threshold number means that the display area can be acquired for each predetermined frame based on the object area being tracked in real time. Then, the electronic device (100) can count the number of acquired display areas. The reason for counting is to determine whether the user's (1000) action is repeated. For example, if the user takes the second pose only once, the existing display area can be considered without changing the display area. However, if the user takes the second pose two or more times, the electronic device (100) can change to the display area corresponding to the second pose. A detailed explanation related to this will be described later in FIG. 14.

[0174] If the display area of ​​the second captured image is not identified by more than a threshold number, the electronic device (100) can identify the size of the display area of ​​the third captured image based on the size of the display area of ​​the first captured image (S1315).

[0175] And, when the display area of ​​the second captured image is identified as more than a threshold number, the electronic device (100) can identify the size of the display area of ​​the third captured image based on the size of the display area of ​​the second captured image (S1125). That is, the size of the display area of ​​the third captured image may be the same as the size of the display area obtained from the second captured image.

[0176] FIG. 14 is a drawing for explaining the change in the display area over time by specifying the operation of FIG. 13.

[0177] Referring to FIG. 14, the table (1405) represents a series of multiple captured images. It is assumed that the series of multiple captured images includes a user object taking a first pose from 1 second to 2 seconds, a user object taking a second pose from 3 seconds to 4 seconds, a user object taking a first pose from 5 seconds to 6 seconds, a user object taking a second pose from 7 seconds to 8 seconds, and a user object taking a first pose from 9 seconds to 13 seconds.

[0178] That is, the embodiment of FIG. 14 may be a situation in which the user (1000) repeatedly takes the first pose and the second pose at 2-second intervals.

[0179] In the embodiment of FIG. 13, an embodiment is disclosed in which the display area is changed only when the same pose is taken for more than a threshold time, and in the embodiment of FIG. 14, an embodiment is disclosed in which the display area is changed only when the same pose is taken for more than a threshold number (threshold count).

[0180] The image captured at 1 second can correspond to the image captured in FIG. 5 (520) because the user object is taking the first pose.

[0181] And, since the image captured at 3 seconds is in the second pose, it can correspond to the image captured (620) of FIG. 6. In the image captured (620), the display area identified becomes larger as the user object takes the second pose, but since the display area identified (1023) is not identified by a threshold number (e.g., 2 times), the image can be transmitted to the display (140) by applying the existing display area (523) instead of applying the display area identified (1023) directly to the image captured.

[0182] And, the image captured at 5 seconds may be in the first pose again. Here, the electronic device (100) may maintain the initial display area (523) as is.

[0183] And, the image captured at 7 seconds may show the user object taking a second pose. Here, the display area (1023) corresponding to the image captured at 7 seconds following the image captured at 3 seconds may be identified at a threshold number (e.g., 2 times). Accordingly, here, the electronic device (100) can change from the display area (523) to the display area (1023) to display the image on the display (140).

[0184] And, the image captured at 9 seconds may be in the first pose again. Here, since the display area corresponding to the first pose is not larger than the display area corresponding to the second pose, the electronic device (100) can display the image on the display (140) while maintaining the existing display area (1023).

[0185] And, the image captured at 11 seconds also maintains the first pose, but since the display area corresponding to the first pose is not larger than the display area corresponding to the second pose, the electronic device (100) can display the image while maintaining the existing display area (1023).

[0186] Meanwhile, in the embodiment of FIG. 14, it was determined whether the display area is identified by a threshold number, but in addition to the threshold number, the threshold time mentioned in the embodiment of FIG. 13 can be applied simultaneously.

[0187] FIG. 15 is a flowchart illustrating the operation of identifying a display area based on content received from an external server.

[0188] Referring to FIG. 15, the electronic device (100) can identify a display area of ​​a first captured image and a display area of ​​a second captured image (S1505).

[0189] And, the electronic device (100) can receive content from an external server (S1510). Here, although step S1510 is described after step S1505, step S1510 can actually be performed first.

[0190] And, the electronic device (100) can identify a guide object in the received content (S1515). Here, the guide object is described in FIGS. 5, FIGS. 6, and FIGS. 10.

[0191] And, the electronic device (100) can track a guide area containing an identified guide object (S1520). The guide area may refer to an area containing a guide object in the received content. Therefore, if the size of the guide object changes, the size of the guide area may also change.

[0192] And, the electronic device (100) can identify the display area of ​​the third image based on the tracked guide area, the display area of ​​the first image, and the display area of ​​the second image.

[0193] Specifically, the electronic device (100) can obtain first ratio information based on a guide area obtained from a first content image corresponding to the time when the first captured image is received and a guide area obtained from a second content image corresponding to the time when the second captured image is received. Additionally, the electronic device (100) can obtain second ratio information based on a display area of ​​the first captured image and a display area of ​​the second captured image. Furthermore, the electronic device (100) can identify the larger ratio among the first ratio information and the second ratio information, and identify the display area of ​​the third captured image based on the identified larger ratio.

[0194] However, depending on the implementation example, ratio information may be obtained in an object area rather than a display area of ​​the captured image. An embodiment related to this is described later in FIGS. 16 to 19. Likewise, FIGS. 16 to 19 are described based on an object area, but this can be applied equally to a display area.

[0195] In the embodiment of FIG. 15, the electronic device (100) can determine the display area of ​​the third captured image by additionally considering the guide area included in the content as well as the captured image.

[0196] FIG. 16 is a flowchart illustrating the operation of considering ratio information to identify the size of the display area.

[0197] Referring to FIG. 16, the electronic device (100) can continuously receive a plurality of content images included in the content. Then, the electronic device (100) can obtain first ratio information based on the size of the guide area of ​​the first content image and the size of the guide area of ​​the second content image (S1610). Here, the first content image and the second content image may refer to consecutive images in chronological order.

[0198] And, the electronic device (100) can identify the object area of ​​the first captured image and the object area of ​​the second captured image (S1615). And, the electronic device (100) can obtain second ratio information based on the size of the object area of ​​the first captured image and the size of the object area of ​​the second captured image (S1620).

[0199] Here, the first ratio information and the second ratio information may refer to the ratio of the size change of the images being compared. For example, the first ratio information may be the size of the first content image : the size of the second content image, and the second ratio information may be the size of the object area of ​​the first captured image : the size of the object area of ​​the second captured image. The ratio information may be divided into vertical ratio information and horizontal ratio information.

[0200] The electronic device (100) can determine whether the first ratio information is a larger ratio than the second ratio information (S1625). Here, if the first ratio information is a larger ratio than the second ratio information, the electronic device (100) can identify the size of the display area of ​​the third captured image based on the first ratio information (S1630). Specifically, the size of the display area obtained from the third captured image can be identified based on the area obtained by multiplying the first ratio information by the object area identified in the third captured image. That is, the electronic device (100) can obtain a modified object area by multiplying the first ratio information corresponding to the content image by the object area of ​​the first captured image, and can identify the display area of ​​the third captured image based on the modified object area.

[0201] If, here, the first ratio information is smaller than the second ratio information, the electronic device (100) can identify the size of the display area of ​​the third captured image based on the second ratio information (S1635). If the second ratio information corresponding to the captured image is larger than the first ratio information corresponding to the content image, the electronic device (100) can obtain a modified object area by multiplying the second ratio information by the object area of ​​the first captured image, and can identify the display area of ​​the third captured image based on the modified object area. In conclusion, step S1635 can obtain the same result as step S735, which identifies the display area of ​​the third captured image based only on the captured image without considering information about the content.

[0202] Meanwhile, although it is described as applying ratio information to the object area, the display area may ultimately be identified based on the modified object area. Therefore, depending on the implementation example, ratio information may be applied directly to the display area rather than the object area.

[0203] Meanwhile, it was described that the object area of ​​the captured image is used to obtain the second ratio information in steps S1615 and S1620. However, according to another embodiment, the electronic device (100) may be implemented in a form that uses the display area of ​​the captured image to obtain the second ratio information.

[0204] FIG. 17 is a diagram illustrating the change in size between an image taking a first pose and an image taking a second pose.

[0205] Referring to FIG. 17, the electronic device (100) can acquire a first content image (510) and a first captured image (520). Here, the first content image (510) and the first captured image (520) may be images displayed simultaneously on a display (140). Additionally, the guide area (512) of the first content image (510) may have horizontal information h11 (e.g., 500, unit omitted) and vertical information w11 (e.g., 100, unit omitted). And, the object area (522) of the first captured image (520) may have horizontal information h21 (e.g., 500, unit omitted) and vertical information w21 (e.g., 100, unit omitted).

[0206] Additionally, the electronic device (100) can acquire a second content image (610) and a second captured image (620). Here, the second content image (610) and the second captured image (620) may be images displayed simultaneously on the display (140). Additionally, the guide area (612) of the second content image (610) may have horizontal information of h12 (e.g., 750, unit omitted) and vertical information of w12 (e.g., 150, unit omitted). And, the object area (622) of the second captured image (620) may have horizontal information of h22 (e.g., 650, unit omitted) and vertical information of w22 (e.g., 130, unit omitted).

[0207] Here, the electronic device (100) can obtain first ratio information based on the first content image (510) and the second content image (610), and obtain second ratio information based on the first captured image (520) and the second captured image (620).

[0208] FIG. 18 is a diagram for explaining ratio information between images disclosed in FIG. 17.

[0209] Referring to FIG. 18, the electronic device (100) can obtain horizontal ratio information (1805) and vertical ratio information (1815) from a first content image (510) and a second content image (610). The first ratio information may include horizontal ratio information (1805) and vertical ratio information (1815).

[0210] Additionally, the electronic device (100) can obtain horizontal ratio information (1810) and vertical ratio information (1820) from the first captured image (520) and the second captured image (620). The second ratio information may include horizontal ratio information (1810) and vertical ratio information (1820).

[0211] Specifically, since the horizontal information of the first content image (510) is w11 (100) and the horizontal information of the second content image (610) is w12 (150), the horizontal ratio of the first content image (510) and the second content image (610) may be w11:w12 (100:150) or 1:w12 / w11 (1:1.5). Also, since the horizontal information of the first captured image (520) is w21 (100) and the horizontal information of the second captured image (620) is w22 (130), the horizontal ratio of the first captured image (520) and the second captured image (620) may be w21:w22 (100:130) or 1:w22 / w21 (1:1.3).

[0212] Specifically, since the vertical information of the first content image (510) is h11 (500) and the vertical information of the second content image (610) is h12 (750), the vertical ratio of the first content image (510) and the second content image (610) may be h11:h12 (500:750) or 1:h12 / h11 (1:1.5). Also, since the vertical information of the first captured image (520) is h21 (500) and the vertical information of the second captured image (620) is h22 (650), the vertical ratio of the first captured image (520) and the second captured image (620) may be h21:h22 (500:650) or 1:h22 / h21 (1:1.3).

[0213] FIG. 19 is a diagram illustrating the operation of displaying an image by applying ratio information.

[0214] Referring to FIG. 19, the electronic device (100) can identify the larger ratio information between the first ratio information and the second ratio information based on step S1625 of FIG. 16. According to the embodiment of FIG. 17 to 18, it is assumed that the first ratio information (1:1.5) is the larger ratio.

[0215] The electronic device (100) can multiply the first ratio information to the object area (522) corresponding to the first captured image (520). Specifically, the electronic device (100) can obtain modified horizontal information w32 (e.g., 150, unit omitted) by multiplying the horizontal information w21 (100) of the object area (522) corresponding to the first captured image (520) by the horizontal ratio information (1805) among the first ratio information. Additionally, the electronic device (100) can obtain modified vertical information h32 (e.g., 750, unit omitted) by multiplying the vertical information h21 (100) of the object area (522) corresponding to the first captured image (520) by the vertical ratio information (1815) among the first ratio information.

[0216] And, the electronic device (100) can identify a changed object area based on the changed horizontal information w32 (150) and the changed vertical information h32 (750). Even though the image (1920) in which the changed object area is identified is in a first pose, the electronic device (100) can identify an object area (1922) that is enlarged more than the size of the existing object area (522).

[0217] And, the electronic device (100) can identify the display area of ​​the third captured image based on the enlarged object area (1922).

[0218] Meanwhile, FIG. 19 describes that ratio information is applied to an object area. However, the main operation of the electronic device (100) according to one embodiment of the present disclosure may be to identify the display area of ​​the third captured image. Accordingly, the electronic device (100) may be implemented in a form that applies ratio information directly to the display area rather than to the object area. For example, the electronic device (100) may identify the size of the new display area by applying the first ratio information directly to the display area (523) of the first captured image (520) of FIG. 5. Then, the electronic device (100) may display the image on the display (140) based on the size of the identified display area.

[0219] FIG. 20 is a flowchart illustrating a method for controlling an electronic device according to one embodiment of the present disclosure.

[0220] Referring to FIG. 20, a control method of an electronic device (100) according to one embodiment of the present disclosure can track an object region including a user object in an image and identify a display region in the image based on the tracked object region (S2005).

[0221] In addition, the control method can identify a display area of ​​the first image based on an object area identified in the first image (S2010).

[0222] Additionally, the control method can identify a display area of ​​the second image in the object area identified in the second image (S2015).

[0223] Additionally, the control method can identify the display area of ​​the third image based on the display area of ​​the first image and the display area of ​​the second image (S2020).

[0224] Meanwhile, the step of identifying the display area of ​​the third captured image (S2020) can identify the display area of ​​the third captured image based on the display area of ​​the second captured image when the size of the display area of ​​the second captured image is larger than the size of the display area of ​​the first captured image.

[0225] Meanwhile, the step of identifying a display area of ​​the first captured image (S2010) and the step of identifying a display area of ​​the second captured image (S2015) can identify an object area based on the height of a user object in the captured image and identify a display area based on the height of the identified object area.

[0226] Meanwhile, the step of identifying the display area of ​​the third captured image (S2020) can identify the display area of ​​the third captured image if a display area larger than the size of the display area of ​​the first captured image is identified in a plurality of second captured images taken after the first captured image, and the third captured image may be an image taken after the plurality of second captured images.

[0227] Meanwhile, the step of identifying the display area of ​​the third captured image (S2020) can identify the display area of ​​the third captured image if a display area larger than the size of the display area of ​​the first captured image is identified in the second captured image, which is captured in a number of consecutively captured threshold images.

[0228] Meanwhile, the control method may further include the step of displaying a screen in which a content image received from an external server is included in a first area and an identified display area is included in a second area.

[0229] Meanwhile, the control method further includes the step of tracking a guide area including a guide object in a content image, and the step of identifying a display area of ​​a third captured image (S2020) can identify a display area of ​​the third captured image based on the size information of the tracked guide area, the size information of the object area of ​​the first captured image, and the size information of the object area of ​​the second captured image.

[0230] Meanwhile, the control method may further include the step of obtaining first ratio information based on the size of a guide area identified in a first content image and the size of a guide area identified in a second content image, and the step of obtaining second ratio information based on the size of an object area of ​​a first captured image and the size of an object area of ​​a second captured image, and the step of identifying a display area of ​​a third captured image (S2020) may identify a display area of ​​the third captured image based on the first ratio information and the second ratio information.

[0231] Meanwhile, the step of identifying the display area of ​​the third captured image (S2020) can identify the display area of ​​the third captured image based on the relatively larger ratio information among the first ratio information and the second ratio information.

[0232] Meanwhile, the step of identifying the display area of ​​the first captured image (S2010), the step of identifying the display area of ​​the second captured image (S2015), and the step of identifying the display area of ​​the third captured image (S2020) can identify the display area to be displayed through the display (140) in the captured image based on the resolution information of the display (140) and the tracked object area.

[0233] Meanwhile, a control method for an electronic device (100) such as that of FIG. 20 can be executed on an electronic device (100) having the configuration of FIG. 2 or FIG. 3, and can also be executed on an electronic device having other configurations.

[0234] Meanwhile, the methods according to the various embodiments of the present disclosure described above can be implemented in the form of an application that can be installed on an existing electronic device.

[0235] In addition, the methods according to the various embodiments of the present disclosure described above can be implemented by software upgrades or hardware upgrades alone for existing electronic devices.

[0236] In addition, the various embodiments of the present disclosure described above may also be performed through an embedded server equipped in an electronic device, or through an external server among at least one of the electronic device and the display device.

[0237] Meanwhile, according to the exemplary embodiments of the present disclosure, the various embodiments described above may be implemented as software comprising instructions stored on a machine-readable storage medium (e.g., a computer). The machine may include electronic devices according to the disclosed embodiments, which are devices capable of calling instructions stored from the storage medium and operating according to the called instructions. When instructions are executed by a processor, the processor may perform a function corresponding to the instructions directly or by using other components under the control of the processor. Instructions may include code generated or executed by a compiler or an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, "non-transitory" means only that the storage medium does not contain a signal and is tangible, and does not distinguish whether data is stored semi-permanently or temporarily in the storage medium.

[0238] Additionally, according to one embodiment of the present disclosure, the method according to the various embodiments described above may be provided as included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)) or online through an application store (e.g., Play Store™). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily created in a storage medium such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0239] Additionally, each component (e.g., module or program) according to the various embodiments described above may be composed of a single or multiple entities, and some of the aforementioned sub-components may be omitted, or other sub-components may be further included in the various embodiments. Generally or additionally, some components (e.g., module or program) may be integrated into a single entity to perform the functions performed by each of the respective components prior to integration in the same or similar manner. The operations performed by the module, program, or other components according to the various embodiments may be executed sequentially, in parallel, iteratively, or heuristically, or at least some operations may be executed in a different order, omitted, or other operations added.

[0240] Although preferred embodiments of the present disclosure have been illustrated and described above, the present disclosure is not limited to the specific embodiments described above. It is understood that various modifications can be made by those skilled in the art without departing from the essence of the present disclosure as claimed in the claims, and such modifications should not be understood individually from the technical spirit or perspective of the present disclosure. Explanation of the symbols

[0241] 100: Electronic device 110: Camera 120: Processor

Claims

Claim 1 An electronic device comprising: a camera; and a processor that tracks an object region including a user object in a plurality of captured images acquired through the camera and identifies a display region in the captured images based on the tracked object region; wherein the processor identifies a display region of the first captured image based on an object region identified in the first captured image among the plurality of captured images, identifies a display region of the second captured image based on an object region identified in the second captured image acquired after the first captured image among the plurality of captured images, and identifies a display region of the third captured image acquired after the second captured image among the plurality of captured images based on the display region of the first captured image and the display region of the second captured image. Claim 2 An electronic device according to claim 1, wherein the processor identifies the display area of ​​the third image based on the display area of ​​the second image when the size of the display area of ​​the second image is larger than the size of the display area of ​​the first image. Claim 3 An electronic device according to claim 1, wherein the processor identifies the object region based on the height of the user object in the captured image, and identifies the display region of the first captured image and the display region of the second captured image based on the height of the identified object region. Claim 4 An electronic device according to claim 1, wherein the processor identifies a display area larger than the size of the display area of ​​the first image in a plurality of second images taken after the first image, and the third image is an image taken after the plurality of second images. Claim 5 An electronic device according to claim 4, wherein the processor identifies a display area of ​​the third image when a display area larger than the size of the display area of ​​the first image is identified in a second image that is continuously captured at a threshold number or more. Claim 6 An electronic device according to claim 1, further comprising a display; wherein the processor controls the display to display a screen in which a content image received from an external server is included in a first area and the identified display area is included in a second area. Claim 7 An electronic device according to claim 6, wherein the processor tracks a guide area including a guide object in the content image, and identifies a display area of ​​the third image based on size information of the tracked guide area, size information of an object area of ​​the first image, and size information of an object area of ​​the second image. Claim 8 An electronic device according to claim 7, wherein the processor obtains first ratio information based on the size of a guide area identified in a first content image and the size of a guide area identified in a second content image, obtains second ratio information based on the size of an object area of ​​the first captured image and the size of an object area of ​​the second captured image, and identifies a display area of ​​the third captured image based on the first ratio information and the second ratio information. Claim 9 An electronic device according to claim 8, wherein the processor identifies a display area of ​​the third captured image based on the relatively larger ratio information among the first ratio information and the second ratio information. Claim 10 An electronic device according to claim 1, wherein the processor identifies a display area of ​​the third captured image to be displayed through the display based on resolution information of the display and the tracked object area. Claim 11 A method for controlling an electronic device, comprising: a step of tracking an object region including a user object in a plurality of captured images and identifying a display region in the captured images based on the tracked object region; a step of identifying a display region of the first captured image based on an object region identified in the first captured image acquired after the first captured image among the plurality of captured images; a step of identifying a display region of the second captured image in an object region identified in the second captured image among the plurality of captured images; and a step of identifying a display region of the third captured image acquired after the second captured image among the plurality of captured images based on the display region of the first captured image and the display region of the second captured image. Claim 12 In claim 11, the step of identifying the display area of ​​the third captured image is a method for controlling an electronic device, wherein if the size of the display area of ​​the second captured image is larger than the size of the display area of ​​the first captured image, the display area of ​​the third captured image is identified based on the display area of ​​the second captured image. Claim 13 A method for controlling an electronic device according to claim 11, wherein the step of identifying a display area of ​​the first captured image and the step of identifying a display area of ​​the second captured image are to identify the object area based on the height of the user object in the captured image, and to identify the display area of ​​the first captured image and the display area of ​​the second captured image based on the height of the identified object area. Claim 14 A method for controlling an electronic device according to claim 11, wherein the step of identifying a display area of ​​the third captured image is to identify the display area of ​​the third captured image when a display area larger than the size of the display area of ​​the first captured image is identified in a plurality of second captured images taken after the first captured image, and the third captured image is an image taken after the plurality of second captured images. Claim 15 In claim 14, the step of identifying a display area of ​​the third captured image is a method for controlling an electronic device, wherein if a display area larger than the size of the display area of ​​the first captured image is identified in a second captured image that is continuously captured at a threshold number or more, the display area of ​​the third captured image is identified. Claim 16 A method for controlling an electronic device, further comprising the step of displaying a screen in which a content image received from an external server is included in a first area and the identified display area is included in a second area. Claim 17 A method for controlling an electronic device according to claim 16, further comprising the step of tracking a guide area including a guide object in the content image; and the step of identifying a display area of ​​the third captured image, wherein the display area of ​​the third captured image is identified based on size information of the tracked guide area, size information of the object area of ​​the first captured image, and size information of the object area of ​​the second captured image. Claim 18 A method for controlling an electronic device according to claim 17, further comprising: a step of obtaining first ratio information based on the size of a guide area identified in a first content image and the size of a guide area identified in a second content image; and a step of obtaining second ratio information based on the size of an object area of ​​the first captured image and the size of an object area of ​​the second captured image; wherein the step of identifying a display area of ​​the third captured image identifies the display area of ​​the third captured image based on the first ratio information and the second ratio information. Claim 19 In claim 18, the step of identifying the display area of ​​the third captured image is to identify the display area of ​​the third captured image based on the relatively larger ratio information among the first ratio information and the second ratio information, a control method for an electronic device. Claim 20 In claim 11, the step of identifying a display area of ​​the first captured image, the step of identifying a display area of ​​the second captured image, and the step of identifying a display area of ​​the third captured image are a method for controlling an electronic device, wherein the display area of ​​the third captured image to be displayed through the display is identified based on the resolution information of the display and the tracked object area.

Images