Electronic device and method for controlling same
The electronic device automates the application of 3D effects by analyzing content and combining layers to create a composite image, addressing inefficiencies in manual object selection and enhancing user experience through efficient 3D effect application.
Patent Information
- Application Number
- PCT/KR2024/000685
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-02
- Filing Date
- 2024-01-15
- Publication Date
- 2024-11-07
AI Technical Summary
Existing electronic devices face inconvenience in manually selecting objects for applying 3D effects in content, requiring content analysis to determine which objects to apply three-dimensional effects, leading to inefficiencies in providing a 3D effect.
An electronic device with a processor that analyzes received content to identify a target object, determines its location, and applies a 3D effect by combining a base layer and a 3D layer to create a composite image, including adding a letter box if necessary, to provide a 3D effect by analyzing the content and selecting the appropriate section for the effect.
Automates the process of applying 3D effects, reducing manual intervention and improving efficiency in providing a 3D effect by analyzing content and selecting the appropriate objects and sections for the effect, enhancing user experience.
Smart Images

Figure KR2024000685_07112024_PF_FP_ABST
Abstract
Description
Electronic device and method of controlling the same
[0001] The present disclosure relates to an electronic device and a control method thereof, and more particularly, to an electronic device and a control method thereof that provide a 3D effect by displaying an object in a letter box.
[0002] Electronic devices can apply occlusion effects to images using a monocular lens. This is one of several elements that can create a 3D effect.
[0003] This could be a way to exploit a cognitive property: when two different objects overlap, the occluded object can be perceived as being further away than the object it is covering.
[0004] Users who wish to provide a 3D effect can select an object among multiple objects included in the content to be used for applying the 3D effect.
[0005] Users can select a suitable section of the entire content playback time to apply the 3D effect.
[0006] There was an inconvenience in that content analysis was required to provide a 3D effect, and the user had to directly select which object in which image to apply the three-dimensional effect.
[0007] The present disclosure provides an electronic device and a control method thereof that provides a 3D effect by analyzing received content and displaying an object in a letter box in a specific target section.
[0008] According to one embodiment, an electronic device includes at least one memory storing content and a command, and at least one processor connected to the memory, wherein the at least one processor identifies a target object from the content stored in the at least one memory, obtains location information of the target object, and identifies a target section to which a 3D effect for displaying a part of the target object in a letter box is to be applied based on the location information of the target object, wherein the target section includes a target image, and obtains a base layer corresponding to a background of the target image and a 3D layer corresponding to the 3D effect based on the target image, and obtains a composite image including the letter box by combining the base layer and the 3D layer, and provides the composite image.
[0009] The at least one processor can obtain the composite image by identifying whether the content includes the letter box, and adding the letter box to at least one of the base layer or the 3D layer if the content does not include the letter box.
[0010] The at least one processor can obtain a 3D image corresponding to the target object and obtain the 3D layer including the 3D image.
[0011] The at least one processor determines the location of the letter box based on the location of content information included in the content, and the content information may include at least one of a graphical UI (User Interface) or a text UI representing the content.
[0012] The content includes a plurality of images, the target object has a changed position in the plurality of images, and the at least one processor can analyze characteristics of the plurality of objects identified in the content to identify the target object among the plurality of objects.
[0013] The at least one processor can identify a preset event based on the location information and identify the target section based on the time at which the preset event is identified.
[0014] The at least one processor can identify a letter box area and a content area among the entire area of the content, and identify a first event in which the target object moves from the letter box area to the content area or a second event in which the target object moves from the content area to the letter box area, and identify the first event or the second event as the preset event.
[0015] The at least one processor can obtain first location information of the target object at a first time point and second location information of the target object at a second time point, and identify the preset event based on the first location information and the second location information.
[0016] The first location information includes at least one of a first minimum coordinate value, a first maximum coordinate value, or a first average coordinate value of the target object at the first time point, the second location information includes at least one of a second minimum coordinate value, a second maximum coordinate value, or a second average coordinate value of the target object at the second time point, and the at least one processor can obtain movement information of the target object based on the first location information and the second location information, and identify the preset event based on the movement information.
[0017] The at least one processor can identify a bounding box of the target object, obtain the first location information based on a first plurality of coordinate values representing the bounding box at the first time point, and obtain the second location information based on a second plurality of coordinate values representing the bounding box at the second time point.
[0018] According to one embodiment, a method for controlling an electronic device storing content includes the steps of: identifying a target object in the content; obtaining location information of the target object; identifying a target section to which a 3D effect for displaying a part of the target object in a letter box is to be applied based on the location information of the target object, wherein the target section includes a target image; obtaining a base layer corresponding to a background of the target image and a 3D layer corresponding to the 3D effect based on the target image; obtaining a composite image including the letter box by combining the base layer and the 3D layer; and providing the composite image.
[0019] The control method further includes a step of identifying whether the content includes the letter box, and the step of obtaining the composite image may obtain the composite image by adding the letter box to at least one of the base layer or the 3D layer if the content does not include the letter box.
[0020] The step of acquiring the above 3D layer may include acquiring a 3D image corresponding to the target object and acquiring the 3D layer including the 3D image.
[0021] The above control method further includes a step of determining the position of the letter box based on the position of content information included in the content, and the content information may include at least one of a graphical UI (User Interface) or a text UI representing the content.
[0022] The above content includes a plurality of images, the position of the target object is changed in the plurality of images, and the step of identifying the target object can identify the target object among the plurality of objects by analyzing the characteristics of the plurality of objects identified in the content.
[0023] The step of identifying the target section may identify a preset event based on the location information, and identify the target section based on the time at which the preset event is identified.
[0024] The step of identifying the target section may include identifying a letter box area and a content area among the entire area of the content, identifying a first event in which the target object moves from the letter box area to the content area or a second event in which the target object moves from the content area to the letter box area, and identifying the first event or the second event as the preset event.
[0025] The step of identifying the above-described preset event may include obtaining first location information of the target object at a first time point and second location information of the target object at a second time point, and identifying the above-described preset event based on the first location information and the second location information.
[0026] The first location information includes at least one of a first minimum coordinate value, a first maximum coordinate value, or a first average coordinate value of the target object at the first time point, the second location information includes at least one of a second minimum coordinate value, a second maximum coordinate value, or a second average coordinate value of the target object at the second time point, and the step of identifying the preset event may obtain movement information of the target object based on the first location information and the second location information, and identify the preset event based on the movement information.
[0027] The step of obtaining the above location information may include identifying a bounding box of the target object, obtaining the first location information based on a first plurality of coordinate values representing the bounding box at the first time point, and obtaining the second location information based on a second plurality of coordinate values representing the bounding box at the second time point.
[0028] The above and other aspects, features and advantages of specific embodiments of the present invention will become more apparent from the following description taken in conjunction with the accompanying drawings.
[0029] FIG. 1 is a drawing illustrating an electronic device providing a 3D effect according to one or more embodiments.
[0030] FIG. 2 is a block diagram illustrating an electronic device according to one embodiment.
[0031] Figure 3 is a block diagram for explaining the specific configuration of the electronic device of Figure 2.
[0032] Figure 4 is a drawing for explaining an operation for identifying whether content includes a letter box.
[0033] Figure 5 is a drawing for explaining an operation of providing a synthetic image.
[0034] Figure 6 is a drawing for explaining an operation of applying a 3D effect to content without a letter box.
[0035] Figure 7 is a drawing for explaining an operation of applying a 3D effect to content with a letter box.
[0036] Figure 8 is a drawing for explaining an operation of applying a 3D effect to an image.
[0037] Figure 9 is a drawing for explaining an operation of identifying a target object.
[0038] Figure 10 is a drawing for explaining an operation of obtaining movement information of a target object.
[0039] FIG. 11 is a drawing for explaining an operation of obtaining movement information of a target object according to one embodiment.
[0040] FIG. 12 is a drawing for explaining an operation of obtaining movement information of a target object according to one embodiment.
[0041] FIG. 13 is a drawing for explaining an operation of obtaining movement information of a target object according to one embodiment.
[0042] Figure 14 is a drawing for explaining the letter box area and content area.
[0043] FIG. 15 is a diagram for explaining a preset event of a target object according to one embodiment.
[0044] FIG. 16 is a drawing for explaining a composite image with a 3D effect applied according to one embodiment.
[0045] FIG. 17 is a diagram for explaining a preset event of a target object according to one embodiment.
[0046] FIG. 18 is a drawing for explaining a composite image with a 3D effect applied according to one embodiment.
[0047] FIG. 19 is a diagram for explaining a preset event of a target object according to one embodiment.
[0048] FIG. 20 is a drawing for explaining a composite image with a 3D effect applied according to one embodiment.
[0049] FIG. 21 is a diagram for explaining a preset event of a target object according to one embodiment.
[0050] FIG. 22 is a drawing for explaining a composite image with a 3D effect applied according to one embodiment.
[0051] FIG. 23 is a diagram for explaining a preset event of a target object according to one embodiment.
[0052] FIG. 24 is a drawing for explaining a composite image with a 3D effect applied according to one embodiment.
[0053] FIG. 25 is a drawing for explaining a preset event of a target object according to one embodiment.
[0054] FIG. 26 is a drawing for explaining a composite image with a 3D effect applied according to one embodiment.
[0055] Figure 27 is a drawing for explaining an example in which content information is damaged.
[0056] FIG. 28 is a drawing for explaining an operation of providing content information according to one embodiment.
[0057] FIG. 29 is a diagram for explaining an operation of providing content information according to one embodiment.
[0058] FIG. 30 is a drawing for explaining an operation of generating a composite image when there is no letter box according to one embodiment.
[0059] FIG. 31 is a diagram illustrating an operation of generating a composite image when there is no letter box according to one embodiment.
[0060] FIG. 32 is a drawing for explaining an operation of generating a composite image when there is no letter box according to one embodiment.
[0061] FIG. 33 is a drawing for explaining an operation of generating a composite image when there is no letter box according to one embodiment.
[0062] FIG. 34 is a drawing for explaining an operation of generating a composite image when there is a letter box according to one embodiment.
[0063] FIG. 35 is a drawing for explaining an operation of generating a composite image when there is a letter box according to one embodiment.
[0064] FIG. 36 is a drawing for explaining an operation of generating a composite image when there is a letter box according to one embodiment.
[0065] FIG. 37 is a drawing for explaining an operation of generating a composite image when there is a letter box according to one embodiment.
[0066] Figure 38 is a drawing for explaining the operation of changing the size of a letter box.
[0067] Figure 39 is a drawing for explaining an operation of playing content by setting a delay time.
[0068] Figure 40 is a flowchart for explaining a control method of an electronic device according to one embodiment.
[0069] Hereinafter, the present disclosure will be described in detail with reference to the attached drawings.
[0070] The terms used in the embodiments of this disclosure have been selected from widely used, current terms, taking into account the functions of this disclosure. However, these terms may vary depending on the intentions of those skilled in the art, precedents, the emergence of new technologies, etc. Furthermore, in certain cases, terms may be arbitrarily selected by the applicant, and in such cases, their meanings will be described in detail in the description of the relevant disclosure. Therefore, the terms used in this disclosure should not be defined simply as names of terms, but rather based on the meanings of the terms and the overall content of this disclosure.
[0071] In this specification, expressions such as “has,” “can have,” “includes,” or “may include” indicate the presence of a feature (e.g., a number, function, operation, or component such as a part), and do not exclude the presence of additional features.
[0072] The expression "at least one of A or B" should be understood to mean either "A" or "B" or "A and B".
[0073] As used herein, the expressions “first,” “second,” “first,” or “second,” etc., may describe various components, regardless of order and / or importance, and are only used to distinguish one component from another, but do not limit the components.
[0074] When it is said that a component (e.g., a first component) is “(operatively or communicatively) coupled with / to” or “connected to” another component (e.g., a second component), it should be understood that the component may be directly coupled to the other component, or may be connected through another component (e.g., a third component).
[0075] Singular expressions include plural expressions unless the context clearly dictates otherwise. In this application, terms such as "comprise" or "consist of" are intended to indicate the presence of a feature, number, step, operation, component, part, or combination thereof described in the specification, but should be understood not to preclude the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.
[0076] In the present disclosure, a "module" or "part" performs at least one function or operation and may be implemented in hardware or software, or a combination of hardware and software. Furthermore, multiple "modules" or multiple "parts" may be integrated into at least one module and implemented as at least one processor, excluding any "modules" or "parts" that need to be implemented as specific hardware.
[0077] In this specification, the term user may refer to a person using an electronic device or a device using an electronic device (e.g., an artificial intelligence electronic device).
[0078] An embodiment of the present disclosure will be described in more detail with reference to the attached drawings below.
[0079] FIG. 1 is a drawing for explaining an electronic device (100) that provides a 3D effect.
[0080] Referring to FIG. 1, the electronic device (100) can provide content with a 3D effect applied. The 3D effect may refer to an occlusion effect. The occlusion effect may be an effect that provides a three-dimensional effect by displaying objects in an overlapping manner.
[0081] For example, a 3D effect may include an out-painting technique. An out-painting technique may be an effect that makes a first object appear closer than a second object by making the first object appear above (or in front of) the second object.
[0082] The electronic device (100) can process content to provide a 3D effect. Specifically, the electronic device (100) can receive content and convert the received content into content to which a 3D effect is applied. The electronic device (100) can provide the converted content to the user.
[0083] FIG. 2 is a block diagram illustrating an electronic device (100) according to one embodiment.
[0084] Referring to FIG. 2, the electronic device (100) may include at least one of at least one memory (110) or at least one processor (120).
[0085] The memory (110) can store content. The electronic device (100) can receive content from an external device or an external server. The electronic device (100) can store the received content in the memory (110). The electronic device (100) can receive a portion of the content in a streaming manner and store the portion of the received content in the memory (110). The memory (110) can be a temporary memory.
[0086] At least one processor (120) can identify a preset target object from content stored in a memory (110), obtain location information of the target object, identify a target section to which a 3D (Three Dimension) effect is to be applied that displays a portion of the target object in a letter box based on the location information of the target object, obtain a base layer corresponding to the background of the target image and a 3D layer corresponding to the 3D effect based on a target image included in the target section, obtain a composite image including a letter box by combining the base layer and the 3D layer, and provide the composite image.
[0087] The 3D effect to be provided by at least one processor (120) may be an out-painting technique. The out-painting technique may be a technique for providing a three-dimensional effect by adjusting the overlap of objects displayed in an image. The out-painting technique may be a method for providing a composite image by adding some information (e.g., a portion of an object) for applying a 3D effect to an original image.
[0088] The content may include multiple images. The content may identify a target object in at least one of the multiple images.
[0089] Among the multiple images included in the content, the image to be analyzed can be described as an input image or an analysis image. Depending on various embodiments, the content itself may represent a single analysis image.
[0090] In one embodiment, the target object may be a preset object. The preset object may be changed based on user settings. The preset object may be a movable object. For example, the preset object may be a ball, an animal, a person, a face, a car, an airplane, a bullet, etc.
[0091] According to one embodiment, the target object may be an object determined by the content. The electronic device (100) may determine an object to which a 3D effect is to be applied as the target object based on at least one image (or image frame) included in the content.
[0092] A target object may be an object whose position changes in multiple images of the content. At least one processor (120) may analyze the characteristics of multiple objects identified in the content to identify the target object among the multiple objects. At least one processor (120) may analyze the characteristics of the objects to determine whether at least one object included in the content is a target object.
[0093] At least one processor (120) can obtain location information of a target object. At least one processor (120) can obtain location information of a target object from each of a plurality of images included in the content.
[0094] At least one processor (120) can identify the location of a target object in the content (or an input image included in the content).
[0095] At least one processor (120) can identify location information of a target object based on a plurality of coordinate values representing the target object.
[0096] For example, the location information of a target object may be a set of coordinate values at which the target object is displayed.
[0097] The location information may include at least one of a minimum coordinate value, a maximum coordinate value, or an average coordinate value. In one embodiment, the coordinate value may include at least one of an x-axis or a y-axis according to a reference axis. For example, the coordinate value may be (x), (y), or (x,y).
[0098] In one embodiment, the coordinate values may include at least one of the x-axis, the y-axis, and the z-axis, depending on the reference axis. For example, the coordinate values may be (x,y,z).
[0099] At least one processor (120) can obtain at least one of a minimum coordinate value (21), a maximum coordinate value (22), or an average coordinate value (23) from a set of coordinate values at which a target object is displayed.
[0100] At least one processor (120) can obtain a minimum coordinate value including an x-axis minimum value and a y-axis minimum value from a set of coordinate values at which a target object is displayed.
[0101] At least one processor (120) can obtain a maximum coordinate value including an x-axis maximum value and a y-axis maximum value from a set of coordinate values at which a target object is displayed.
[0102] At least one processor (120) can obtain an average coordinate value including an x-axis average value and a y-axis average value from a set of coordinate values at which a target object is displayed.
[0103] In one embodiment, the mean may be an average over all coordinate values.
[0104] In one embodiment, the average value may be an average of the minimum and maximum values for the coordinate values.
[0105] At least one processor (120) can obtain first location information of a target object at a first time point and second location information of the target object at a second time point, and identify a preset event based on the first location information and the second location information.
[0106] The location of the target object may change in each of the multiple images included in the content. At least one processor (120) may track the movement of the target object based on the location information of the target object.
[0107] At least one processor (120) can analyze the movement of the target object by analyzing the image of the first time point and the image of the second time point.
[0108] The first position information may include at least one of a minimum coordinate value, a maximum coordinate value, or an average coordinate value of the target object at the first point in time.
[0109] The second position information may include at least one of a minimum coordinate value, a maximum coordinate value, or an average coordinate value of the target object at the second point in time.
[0110] At least one processor (120) can obtain movement information of a target object based on the first location information and the second location information, and identify a preset event based on the movement information.
[0111] The movement information may include at least one of direction information and speed information. At least one processor (120) may obtain the movement information based on the first location information and the second location information.
[0112] At least one processor (120) can identify a bounding box (20) of a target object, obtain first location information based on a plurality of coordinate values representing the bounding box at a first point in time, and obtain second location information based on a plurality of coordinate values representing the bounding box at a second point in time.
[0113] Specific details related to the location information of the target object are described in FIGS. 9 to 13.
[0114] At least one processor (120) can identify a target section to which a 3D effect is to be applied among the entire section of the content based on the location information of the target object.
[0115] For example, at least one processor (120) may determine to apply a 3D effect to an image between 10 and 11 seconds in a 60-second content. The target period may be between 10 and 11 seconds.
[0116] At least one processor (120) can identify a target section based on the time at which the preset event is identified, when a preset event is identified based on the location information of the target object.
[0117] At least one processor (120) can determine whether a preset event is identified based on location information of a target object. At least one processor (120) can identify a target section based on the time at which the preset event is identified.
[0118] At least one processor (120) can distinguish between a letter box area and a content area among the entire area of content, and identify a first event in which a target object moves from the letter box area to the content area or a second event in which a target object moves from the content area to the letter box area as a preset event.
[0119] A letterbox area can refer to an area in a composite image where letterboxing is displayed. A letterbox area can be an area with a dark background (e.g., black) where text information is displayed. A letterbox area can also be an area where only a preset number of color values are displayed.
[0120] For example, the content may include an area with a black background at the top or bottom of the entire area. The area with the black background may be the area where subtitles are displayed. The letterbox area may be the area where a background of a first color (e.g., black) and subtitles of a second color (e.g., white) are displayed.
[0121] The content area can refer to the area where content is displayed. The content area can be any area within the entire area, excluding the letterbox area. The content area can also be an area where various color values are displayed.
[0122] The letter box area may include two areas. The letter box area may include a first letter box area (31) and a second letter box area (32). The first letter box area (31) may be an area located at the upper side of the entire area. The second letter box area (32) may be an area located at the lower side of the entire area.
[0123] Information related to the letter box area and content area is described in Fig. 14.
[0124] At least one processor (120) can analyze whether the target object is located in the letter box area or the content area. At least one processor (120) can identify a preset event based on the location of the target object.
[0125] Specific descriptions of the preset events and 3D effects according to the preset events are described in FIGS. 15 to 26.
[0126] The operation of identifying a preset event in content without a letterbox is described in FIGS. 15 to 20.
[0127] The operation of identifying a preset event in content with a letter box is described in FIGS. 21 to 26.
[0128] At least one processor (120) can identify a third point in time at which the target object moves from the content area to the letter box area. At least one processor (120) can identify a fourth point in time at which the target object continues to be identified in the letter box area and then moves to the content area. At least one processor (120) can identify the period from the third point in time to the fourth point in time as the target section.
[0129] At least one processor (120) can identify a fifth point in time at which the target object moves from the letter box area to the content area. At least one processor (120) can identify a sixth point in time prior to the threshold time based on the fifth point in time. The sixth point in time may be prior to the fifth point in time. At least one processor (120) can identify the period from the sixth point in time to the fifth point in time as the target period.
[0130] At least one processor (120) can identify a seventh point in time at which the target object moves from the content area to the letterbox area. At least one processor (120) can identify an eighth point in time that is a threshold time after the seventh point in time. The eighth point in time may be a time after the seventh point in time. At least one processor (120) can identify the period from the seventh point in time to the eighth point in time as the target section.
[0131] At least one processor (120) can identify whether the content includes a letter box, and if the content does not include a letter box, obtain a composite image by adding a letter box to at least one of the base layer or the 3D layer.
[0132] At least one processor (120) can obtain a 3D image corresponding to a target object to be displayed in a letter box and obtain a 3D layer including the 3D image.
[0133] The operation of generating a composite image from content without letterboxing is described in FIGS. 15 to 20.
[0134] The operation of generating a composite image from content with letter boxes is described in FIGS. 34 to 37.
[0135] At least one processor (120) determines the location of the letter box based on the location of content information included in the content, and the content information may include at least one of a graphical UI (User Interface) or a text UI representing the content.
[0136] Specific descriptions related to this are described in Figs. 27 to 29.
[0137] The electronic device (100) can determine the length of the letter box based on the location of the target object and the location of the content information.
[0138] At least one processor (120) may delay playback of content by a threshold amount of time to secure the time required to generate a composite image. A description thereof is provided in FIG. 39.
[0139] According to various embodiments, the content may include one image, and the electronic device (100) may apply a 3D effect to the one image.
[0140] The electronic device (100) may further include a display (140). At least one processor (120) may display a composite image on the display (140). At least one processor (120) may control the display (140) to display the composite image.
[0141] FIG. 3 is a block diagram for explaining the specific configuration of the electronic device (100) of FIG. 2.
[0142] Referring to FIG. 3, the electronic device (100) may include at least one of a memory (110), at least one processor (120), a communication interface (130), a display (140), an operation interface (150), an input / output interface (160), a speaker (170), or a microphone (180).
[0143] At least one memory (110) may be implemented as an internal memory such as a ROM (e.g., an electrically erasable programmable read-only memory (EEPROM)) or a RAM included in at least one processor (120), or may be implemented as a separate memory from at least one processor (120). In this case, the memory (110) may be implemented as a memory embedded in the electronic device (100) or as a memory detachable from the electronic device (100) depending on the purpose of data storage. For example, data for driving the electronic device (100) may be stored in a memory embedded in the electronic device (100), and data for an extended function of the electronic device (100) may be stored in a memory detachable from the electronic device (100).
[0144] In the case of memory embedded in the electronic device (100), it may be implemented as at least one of volatile memory (e.g., dynamic RAM (DRAM), static RAM (SRAM), or synchronous dynamic RAM (SDRAM)), non-volatile memory (e.g., one time programmable ROM (OTPROM), programmable ROM (PROM), erasable and programmable ROM (EPROM), electrically erasable and programmable ROM (EEPROM), mask ROM, flash ROM, flash memory (e.g., NAND flash or NOR flash), etc.), hard drive, or solid state drive (SSD), and in the case of memory that can be attached or detached to the electronic device (100), it may be implemented in the form of a memory card (e.g., compact flash (CF), secure digital (SD), micro secure digital (Micro-SD), mini secure digital (Mini-SD), extreme digital (xD), multi-media card (MMC), etc.), external memory that can be connected to a USB port (e.g., USB memory), etc.
[0145] At least one processor (120) can perform overall control operations of the electronic device (100). Specifically, at least one processor (120) has a function of controlling overall operations of the electronic device (100).
[0146] At least one processor (120) may be implemented as a digital signal processor (DSP), a microprocessor, or a time controller (TCON) that processes digital signals. However, the present invention is not limited thereto, and may include one or more of a central processing unit (CPU), a micro controller unit (MCU), a micro processing unit (MPU), a controller, an application processor (AP), a graphics-processing unit (GPU), a communication processor (CP), or an advanced reduced instruction set computer (RISC) machines (ARM) processor, or may be defined by the relevant terms. In addition, at least one processor (120) may be implemented as a system on chip (SoC) having a built-in processing algorithm, a large scale integration (LSI), or may be implemented in the form of a field programmable gate array (FPGA). In addition, at least one processor (120) may perform various functions by executing computer executable instructions stored in a memory.
[0147] The communication interface (130) is a component that performs communication with various types of external devices according to various types of communication methods. The communication interface (130) may include a wireless communication module or a wired communication module. Here, each communication module may be implemented in the form of at least one hardware chip.
[0148] A wireless communication module may be a module that communicates wirelessly with an external device. For example, the wireless communication module may include at least one of a Wi-Fi module, a Bluetooth module, an infrared communication module, or other communication modules.
[0149] Wi-Fi and Bluetooth modules can communicate via Wi-Fi and Bluetooth, respectively. When using a Wi-Fi or Bluetooth module, various connection information, such as the service set identifier (SSID) and session key, is first transmitted and received. This information is then used to establish a communication connection before various other information can be transmitted and received.
[0150] Infrared communication modules perform communication based on infrared communication (IrDA, infrared Data Association) technology, which transmits data wirelessly over short distances using infrared light, which is between visible light and millimeter waves.
[0151] In addition to the above-described communication method, other communication modules may include at least one communication chip that performs communication according to various wireless communication standards such as zigbee, 3G (3rd Generation), 3GPP (3rd Generation Partnership Project), LTE (Long Term Evolution), LTE-A (LTE Advanced), 4G (4th Generation), 5G (5th Generation), etc.
[0152] A wired communication module may be a module that communicates with an external device via a wire. For example, the wired communication module may include at least one of a Local Area Network (LAN) module, an Ethernet module, a paired cable, a coaxial cable, a fiber optic cable, or an Ultra Wide-Band (UWB) module.
[0153] The display (140) may be implemented as a display of various forms, such as a liquid crystal display (LCD), an organic light emitting diode (OLED) display, a plasma display panel (PDP), etc. The display (140) may also include a driving circuit, a backlight unit, etc., which may be implemented as a form, such as an a-si TFT (amorphous silicon thin film transistor), an LTPS (low temperature poly silicon) TFT, an OTFT (organic TFT), etc. The display (140) may be implemented as a touch screen combined with a touch sensor, a flexible display, a three-dimensional display (3D display, three-dimensional dispaly), etc. In addition, according to an embodiment of the present disclosure, the display (140) may include a bezel that houses the display panel as well as a display panel that outputs an image. In particular, according to an embodiment of the present disclosure, the bezel may include a touch sensor for detecting user interaction.
[0154] The operation interface (150) may be implemented as a device such as a button, a touch pad, a mouse, and a keyboard, or as a touch screen capable of performing the above-described display function and operation input function. Here, the button may be a mechanical button, a touch pad, a wheel, or any other type of button formed in any area of the front, side, or back of the main body of the electronic device (100).
[0155] The input / output interface (160) may be any one of HDMI (High Definition Multimedia Interface), MHL (Mobile High-Definition Link), USB (Universal Serial Bus), DP (Display Port), Thunderbolt, VGA (Video Graphics Array) port, RGB port, D-SUB (D-subminiature), and DVI (Digital Visual Interface). The input / output interface (160) may input / output at least one of audio and video signals. Depending on the implementation example, the input / output interface (160) may include a port that inputs / outputs only audio signals and a port that inputs / outputs only video signals as separate ports, or may be implemented as a single port that inputs / outputs both audio signals and video signals. The electronic device (100) may transmit at least one of the audio and video signals to an external device (e.g., an external display device or an external speaker) through the input / output interface (160). Specifically, an output port included in the input / output interface (160) can be connected to an external device, and the electronic device (100) can transmit at least one of an audio or video signal to the external device through the output port.
[0156] Here, the input / output interface (160) can be connected to a communication interface. The input / output interface (160) can transmit information received from an external device to the communication interface or transmit information received through the communication interface to the external device.
[0157] The speaker (170) may be a component that outputs various audio data as well as various notification sounds or voice messages.
[0158] The microphone (180) is a component that receives a user's voice or other sounds and converts them into audio data. The microphone (180) can receive the user's voice when activated. For example, the microphone (180) may be formed integrally on the top, front, or side of the electronic device (100). The microphone (180) may include various components, such as a microphone that collects the user's voice in analog form, an amplifier circuit that amplifies the collected user's voice, an A / D conversion circuit that samples the amplified user's voice and converts it into a digital signal, and a filter circuit that removes noise components from the converted digital signal.
[0159] Figure 4 is a drawing for explaining an operation for identifying whether content includes a letter box.
[0160] Referring to FIG. 4, the electronic device (100) can acquire content (S405). The content can be stored in the memory of the electronic device (100). The electronic device (100) can receive content from an external device. The electronic device (100) can receive content from the external device in real time. The method for receiving content in real time may be streaming. The electronic device (100) can receive content in real time from an external device or an external server.
[0161] The electronic device (100) can determine whether the content includes a letter box (S405). A letter box may refer to a margin area displayed in a dark color on at least one of the upper or lower sides of the content.
[0162] If the content does not include a letter box (S410-N), the electronic device (100) can apply a 3D effect to the content that generated the letter box (S415). Embodiments that do not include a letter box are described in FIGS. 15 to 20 and FIGS. 30 to 33.
[0163] If the content includes a letter box (S410-Y), the electronic device (100) can apply a 3D effect to the content (S420). Embodiments including a letter box are described in FIGS. 21 to 26 and FIGS. 34 to 37.
[0164] Figure 5 is a drawing for explaining an operation of providing a synthetic image.
[0165] Referring to FIG. 5, the electronic device (100) can identify a target object in the content (S510).
[0166] In one embodiment, the target object may be a preset object. The preset object may be changed based on user settings. The preset object may be a movable object. For example, the preset object may be a ball, an animal, a person, a face, a car, an airplane, a bullet, etc.
[0167] According to one embodiment, the target object may be an object determined based on content. The electronic device (100) may determine an object to which a 3D effect is to be applied as a target object based on at least one image (or image frame) included in the content.
[0168] The electronic device (100) can obtain location information of a target object (S520).
[0169] The electronic device (100) can obtain movement information (direction information, speed information) based on location information. The electronic device (100) can obtain direction information based on location information. The electronic device (100) can obtain speed information based on location information.
[0170] The electronic device (100) can determine whether a preset event has been identified based on the location information of the target object (S530).
[0171] If a preset event is not identified (S530-N), the electronic device (100) can repeatedly obtain location information of the target object in real time.
[0172] When a preset event is identified (S530-Y), the electronic device (100) can identify a target section to which the 3D effect is to be applied (S540).
[0173] The electronic device (100) can obtain a target image corresponding to the target section (S550).
[0174] The electronic device (100) can obtain a base layer and a 3D layer based on a target image (S560).
[0175] The electronic device (100) can obtain a composite image by combining the base layer and the 3D layer (S570).
[0176] The electronic device (100) can provide a composite image (S580). The electronic device (100) can provide content including the composite image.
[0177] Figure 6 is a drawing for explaining an operation of applying a 3D effect to content without a letter box.
[0178] Referring to FIG. 6, the electronic device (100) can obtain a target image corresponding to the target section (S610).
[0179] The electronic device (100) can determine the size of the letter box based on the target image (S620).
[0180] The size may include at least one of the width or height.
[0181] In one embodiment, the size of the letter box may be a predetermined size.
[0182] According to one embodiment, the size of the letter box may be determined based on the resolution of the content. The electronic device (100) may store a mapping table including the sizes of the letter boxes corresponding to each of a plurality of resolutions. The electronic device (100) may obtain the resolution of the received content. The electronic device (100) may obtain the size of the letter box corresponding to (mapped to) the resolution of the content based on the mapping table.
[0183] In one embodiment, the size of the letter box may be determined based on the location of the content information. An embodiment related to this is described in FIG. 28.
[0184] The electronic device (100) can create a base layer by adding a letter box to the target image (S630).
[0185] The electronic device (100) can generate a 3D layer containing an object image at a letter box location (S640).
[0186] The electronic device (100) can obtain a composite image by combining the base layer and the 3D layer (S650).
[0187] Examples of generating a composite image from content that does not include letter boxes are described in FIGS. 30 to 33.
[0188] Figure 7 is a drawing for explaining an operation of applying a 3D effect to content with a letter box.
[0189] Referring to FIG. 7, the electronic device (100) can obtain a target image corresponding to the target section (S710).
[0190] The electronic device (100) can obtain a first converted image by removing a letter box from a target image (S720).
[0191] The electronic device (100) can obtain a second converted image by changing the resolution of the first converted image (S730).
[0192] The electronic device (100) can determine the size of the letter box based on the second converted image (S740).
[0193] The electronic device (100) can create a base layer by adding a letter box to the target image (S750).
[0194] The electronic device (100) can generate a 3D layer containing an object image at a letter box location (S760).
[0195] The electronic device (100) can obtain a composite image by combining a base layer and a 3D layer.
[0196] Examples of generating a composite image from content containing letter boxes are described in FIGS. 34 to 37.
[0197] Figure 8 is a drawing for explaining an operation of applying a 3D effect to an image.
[0198] Referring to FIG. 8, the electronic device (100) can identify a target section in content to which a 3D effect has been applied. The electronic device (100) can acquire a target image corresponding to the target section. The electronic device (100) can perform a function of applying a 3D effect using the target image as an input image (or input data).
[0199] The electronic device (100) may include at least one of an image preprocessing unit (810), an image analysis unit (820), or an image generation unit (830).
[0200] The image preprocessing unit (810) may include at least one of a letter box detection unit (811) or an image correction unit (812).
[0201] The image preprocessing unit (810) can analyze and transform the image before applying a 3D effect to the input image.
[0202] The letter box detection unit (811) may be a module that determines whether the content includes a letter box.
[0203] When an input image is input to the image preprocessing unit (810), the electronic device (100) can perform letter box detection in the letter box detection unit (811). The letter box detection unit (811) can identify whether a letter box is included in the content. The letter box detection unit (811) can obtain size information of the letter box. The size information can include the vertical length of the letter box. If the content does not include a letter box, the input image can be transmitted to the image analysis unit (820) without a preprocessing operation.
[0204] The image correction unit (812) may be a module that performs at least one of an action of removing a letter box included in the content or an action of changing the resolution.
[0205] If there is a letter box, the image correction unit (812) can perform letter box removal and image resizing (resolution change).
[0206] The image correction unit (812) can remove the letter box and resize the image to match the aspect ratio of the original image. Since distortion may occur compared to the aspect ratio of the original image during resizing, a method of resizing by cropping the left and right areas (horizontally) of the image according to the removed letter box ratio can be applied.
[0207] In various embodiments, even if there is a letterbox, the letterbox can be maintained without being removed.
[0208] Depending on the various embodiments, the resolution may not change even if there is letterboxing.
[0209] The image analysis unit (820) may include at least one of a target object detection unit (821), a movement information calculation unit (822), or a target section detection unit (823).
[0210] The image analysis unit (820) can analyze an input image and analyze an image with a 3D effect applied.
[0211] The target object detection unit (821) can determine whether a target object is included in the input image. The target object detection unit (821) can determine the target object.
[0212] A target object can be an object that attracts the viewer's attention. The target object detection unit (821) can utilize saliency information within the image. Saliency is a term that indicates the degree to which the viewer's gaze is drawn within the image. The higher the saliency, the higher the probability of attracting the viewer's attention. Saliency information can be described as interest level information or interest feature information.
[0213] The greater the color difference from adjacent pixels, the closer the location is to the center of the screen, and the higher the feature values such as texture (edge, detail, etc.), the higher the saliency.
[0214] The target object detection unit (821) can detect candidate pixels of an object by grouping pixels with high saliency. The target object detection unit (821) can cluster the shape of an object by utilizing texture components such as color, brightness, and edge of surrounding pixels based on the group of pixels with high saliency.
[0215] The target object detection unit (821) can use a set of clustered pixels as object information. The target object detection unit (821) can obtain information on whether the shape is maintained across multiple frames by considering the continuity of pixel information (saliency, color, brightness, edge, texture information, etc.) between adjacent frames. The target object detection unit (821) can identify the target object based on the various information obtained.
[0216] The target object may be an object within the image whose depth information is closer to the screen than its surroundings.
[0217] The target object detection unit (821) can utilize depth information of the input image. If there is no depth information in the content, the target object detection unit (821) can utilize a technology for obtaining 3D depth information based on 2D RGB image information.
[0218] The target object detection unit (821) can obtain depth information for each pixel by using a technique through deep learning such as 'single image depth estimation'.
[0219] The target object detection unit (821) can identify a target object by combining saliency information and depth information.
[0220] The target object detection unit (821) can determine the target object through judgment conditions such as whether the depth value of the object is smaller than a specific threshold value, or whether the depth value of object y is getting smaller (getting closer to the screen) for continuous frame information.
[0221] The target object detection unit (821) can obtain the number of object candidates (N) per frame of the image, a set of pixel coordinates forming the object for each candidate (object pixel), a bounding box of the minimum size containing the object (objectBoundBox), and the center coordinates of the object (objectCenter).
[0222] The target object detection unit (821) can filter the number of object candidates desired by the user and the size and location of each object by adjusting a predefined threshold value.
[0223] In one or more embodiments, the target object detection unit (821) can identify the target object by considering whether the size (length or area) of the object is greater than or equal to a threshold value.
[0224] The movement information calculation unit (822) can obtain (or calculate) movement information of a target object. The movement information may include at least one of direction information and speed information. The operation of obtaining the movement information is described in FIGS. 9 to 14. The movement information can be obtained based on location information.
[0225] Depending on the various expressions, the movement information may include at least one of position information, direction information, or speed information.
[0226] The target section detection unit (823) can identify at least one image included in the content to which a 3D effect is to be applied. For example, it can identify a time section within a 1-hour piece of content to which a 3D effect is to be applied. The electronic device (100) can determine the target section based on whether a preset event is identified. Specific descriptions of the preset events are provided in FIGS. 15 to 26 .
[0227] In one or more embodiments, the target section detection unit (823) may identify the target section by considering whether the object's movement speed is greater than or equal to a threshold value. The target section detection unit (823) may identify the target section based on an object whose movement speed exceeds a first threshold value and is less than or equal to a second threshold value. If the movement speed is less than or equal to the first threshold value or greater than or equal to the second threshold value, the target section detection unit (823) may determine that the object is not a target section.
[0228] The image generation unit (830) may include at least one of a main UI (User Interface) detection unit (831), a letter box generation unit (832), a layer generation unit (833), or a layer synthesis unit (834).
[0229] The image generation unit (830) can generate (or obtain) a composite image with a 3D effect applied.
[0230] The main UI (User Interface) detection unit (831) can identify a UI containing content information contained in an input image (or a preprocessed image). The content information may include a content name and information related to the channel providing the content. The channel-related information may include a channel name, channel number, and channel logo.
[0231] The letter box generation unit (832) can determine the size and position of the letter box based on the input image (or preprocessed image).
[0232] The letter box generation unit (832) can determine the size of the letter box to be inserted based on the movement information of the object.
[0233] The letter box generation unit (832) can determine the size of the letter box to be inserted into the target section.
[0234] The letter box generation unit (832) can gradually increase the size of the letter box from 0 before the target section to reach the maximum size of the letter box at the start of the effect section.
[0235] The letter box generation unit (832) can control the letter box to slowly disappear after the target section.
[0236] The letter box generation unit (832) can determine the maximum size of the letter box through content information when determining the maximum size so that the letter box is generated only up to the point where it does not cover the corresponding graphics.
[0237] The layer generation unit (833) can generate at least one of a basic layer or a 3D layer based on an input image (or a preprocessed image).
[0238] The layer synthesis unit (834) can obtain a synthesis image by combining a basic layer and a 3D layer.
[0239] The layer synthesis unit (834) can prevent unnatural appearance by applying a blurring filter to the object boundary portion. The electronic device (100) can obtain the synthesis image as an output image.
[0240] Figure 9 is a drawing for explaining an operation of identifying a target object.
[0241] Referring to the embodiment (910) of FIG. 9, a target object (10) may be included in the content. The electronic device (100) may identify the target object (10). The electronic device (100) may identify (or determine) a bounding box (20) based on the target object (10). The electronic device (100) may obtain location information of the target object (10) based on the bounding box (20) of the target object (10).
[0242] Referring to the embodiment (920) of FIG. 9, the electronic device (100) can obtain location information indicating the location of the target object (10).
[0243] An electronic device (100) can obtain a bounding box (20) of a target object (10). The bounding box (20) may be a set of at least one bounding point.
[0244] The electronic device (100) can determine the location information of the target object (10) based on the location information of the bounding box (20). The location information of the bounding box (20) can include the location of each of a plurality of bounding points representing the bounding box (20).
[0245] The electronic device (100) can obtain location information of the target object (10) in various ways based on the location information of the bounding box (20).
[0246] According to one embodiment, the electronic device (100) can obtain the minimum coordinate value (21) of the target object (10). The minimum coordinate value (21) can include the minimum value of the x-coordinates and the minimum value of the y-coordinates of the bounding box (20).
[0247] According to one embodiment, the electronic device (100) can obtain the maximum coordinate value (22) of the target object (10). The maximum coordinate value (22) can include the maximum value of the x-coordinates and the maximum value of the y-coordinates of the bounding box (20).
[0248] According to one embodiment, the electronic device (100) can obtain an average coordinate value (23) of a target object (10). The average coordinate value (23) can include an average value of x-coordinates and an average value of y-coordinates of a bounding box (20). The average coordinate value (23) can include an average value of a minimum coordinate value (21) and a maximum coordinate value (22).
[0249] Figure 10 is a drawing for explaining an operation of obtaining movement information of a target object.
[0250] Referring to embodiment (1010) of FIG. 10, a target object (10) may move over time. Embodiment (1010) may not represent an image (or frame) at a specific point in time. Embodiment (1010) represents the movement process of the target object (10) and the bounding box (20) as a single image. In reality, one image (frame) will only include one target object (10).
[0251] Referring to the embodiment (1020) of FIG. 10, the electronic device (100) can obtain first location information based on the bounding box (20) at a first point in time. The first location information can include at least one of a minimum coordinate value (Ps-min), a maximum coordinate value (Ps-max), or an average coordinate value (Ps-average). Ps can mean a point at which an object starts moving in the analysis section.
[0252] The electronic device (100) can obtain second location information based on the bounding box (20) at a second point in time. The second location information can include at least one of a minimum coordinate value (Pe-min), a maximum coordinate value (Pe-max), or an average coordinate value (Pe-average). Pe can mean a point where movement ends in the analysis section (or a point of an object at the current point in time).
[0253] The electronic device (100) can obtain (or calculate) movement information of the target object (10) based on the location information at the first point in time and the location information at the second point in time. The electronic device (100) can obtain (or calculate) movement information of the target object (10) 1, 2, 3) or at least one of the movement speeds (v1, v2, v3) can be acquired.
[0254] FIG. 11 is a drawing for explaining an operation of obtaining movement information of a target object according to one embodiment.
[0255] Referring to FIG. 11, the electronic device (100) can obtain movement information of the target object (10) based on the minimum coordinate value.
[0256] Referring to the embodiment (1110) of FIG. 11, the electronic device (100) can obtain the minimum coordinate value (Ps-min) of the first time point and the minimum coordinate value (Pe-min) of the second time point.
[0257] The electronic device (100) can obtain position information (Box_s) of the bounding box (20) at a first point in time.
[0258] The electronic device (100) can obtain the minimum value of x-coordinates (x_min(Box_s)) and the minimum value of y-coordinates (y_min(Box_s)) among the position information (Box_s) of the bounding box (20) at the first point in time.
[0259] The electronic device (100) can obtain a minimum coordinate value (Ps-min) including the minimum value of x-coordinates (x_min(Box_s)) and the minimum value of y-coordinates (y_min(Box_s)) at a first point in time.
[0260] The electronic device (100) can obtain position information (Box_e) of the bounding box (20) at a second point in time.
[0261] The electronic device (100) can obtain the minimum value of the x-coordinates (x_min(Box_e)) and the minimum value of the y-coordinates (y_min(Box_e)) among the position information (Box_e) of the bounding box (20) at the second point in time.
[0262] The electronic device (100) can obtain a minimum coordinate value (Pe-min) including the minimum value of x-coordinates (x_min(Box_e)) and the minimum value of y-coordinates (y_min(Box_e)) at the second point in time.
[0263] Referring to the embodiment (1120) of FIG. 11, the electronic device (100) can obtain the movement direction of the target object (10). The electronic device (100) can obtain the movement direction ( 1) can be obtained. The direction of movement can be described in degrees.
[0264] The first distance (x1) may be the absolute value of the minimum value of the x-coordinates at the second time point (x_min(Box_e)) minus the minimum value of the x-coordinates at the first time point (x_min(Box_s)).
[0265] The second distance (y1) may be the absolute value of the difference between the minimum value of the y-coordinates at the second time point (y_min(Box_e)) and the minimum value of the y-coordinates at the first time point (y_min(Box_s)).
[0266] Referring to the embodiment (1130) of FIG. 11, the electronic device (100) can obtain the movement speed (v1) using the movement distance (d1) and the time interval (△t).
[0267] The distance traveled (d1) may be the square root of the sum of the squares of the first distance (x1) and the squares of the second distance (y1).
[0268] The time interval (△t) can be the difference between the first time point and the second time point.
[0269] The electronic device (100) can obtain the movement speed (v1) by dividing the movement distance (d1) by the time interval (△t).
[0270] FIG. 12 is a drawing for explaining an operation of obtaining movement information of a target object according to one embodiment.
[0271] Referring to FIG. 12, the electronic device (100) can obtain movement information of the target object (10) based on the maximum coordinate value.
[0272] Referring to the embodiment (1210) of FIG. 12, the electronic device (100) can obtain the maximum coordinate value (Ps-max) of the first time point and the maximum coordinate value (Pe-max) of the second time point.
[0273] The electronic device (100) can obtain position information (Box_s) of the bounding box (20) at a first point in time.
[0274] The electronic device (100) can obtain the maximum value of x-coordinates (x_max(Box_s)) and the maximum value of y-coordinates (y_max(Box_s)) among the position information (Box_s) of the bounding box (20) at the first point in time.
[0275] The electronic device (100) can obtain a maximum coordinate value (Ps-max) including the maximum value of x-coordinates (x_max(Box_s)) and the maximum value of y-coordinates (y_max(Box_s)) at a first point in time.
[0276] The electronic device (100) can obtain position information (Box_e) of the bounding box (20) at a second point in time.
[0277] The electronic device (100) can obtain the maximum value of the x-coordinates (x_max(Box_e)) and the maximum value of the y-coordinates (y_max(Box_e)) among the position information (Box_e) of the bounding box (20) at the second point in time.
[0278] The electronic device (100) can obtain a maximum coordinate value (Pe-max) including the maximum value of x-coordinates (x_max(Box_e)) and the maximum value of y-coordinates (y_max(Box_e)) at a second point in time.
[0279] Referring to the embodiment (1220) of FIG. 12, the electronic device (100) can obtain the movement direction of the target object (10). The electronic device (100) can obtain the movement direction ( 2) can be obtained. The direction of movement can be described in degrees.
[0280] The first distance (x2) may be the absolute value of the maximum value of the x-coordinates at the second time point (x_max(Box_e)) minus the maximum value of the x-coordinates at the first time point (x_max(Box_s)).
[0281] The second distance (y2) may be the absolute value of the maximum value of the y-coordinates at the second time point (y_max(Box_e)) minus the maximum value of the y-coordinates at the first time point (y_max(Box_s)).
[0282] Referring to the embodiment (1230) of FIG. 12, the electronic device (100) can obtain the movement speed (v2) using the movement distance (d2) and the time interval (△t).
[0283] The distance traveled (d2) may be the square root of the sum of the squares of the first distance (x2) and the squares of the second distance (y2).
[0284] The time interval (△t) can be the difference between the first time point and the second time point.
[0285] The electronic device (100) can obtain the movement speed (v2) by dividing the movement distance (d2) by the time interval (△t).
[0286] FIG. 13 is a drawing for explaining an operation of obtaining movement information of a target object according to one embodiment.
[0287] Referring to FIG. 13, the electronic device (100) can obtain movement information of a target object (10) based on an average coordinate value.
[0288] Referring to the embodiment (1310) of FIG. 13, the electronic device (100) can obtain the average coordinate value (Ps-average) of the first time point and the average coordinate value (Pe-average) of the second time point.
[0289] The electronic device (100) can obtain position information (Box_s) of the bounding box (20) at a first point in time.
[0290] The electronic device (100) can obtain the average value of x-coordinates (x_average(Box_s)) and the average value of y-coordinates (y_average(Box_s)) among the position information (Box_s) of the bounding box (20) at the first point in time.
[0291] The electronic device (100) can obtain an average coordinate value (Ps-average) including an average value of x-coordinates (x_average(Box_s)) and an average value of y-coordinates (y_average(Box_s)) at a first point in time.
[0292] The electronic device (100) can obtain position information (Box_e) of the bounding box (20) at a second point in time.
[0293] The electronic device (100) can obtain the average value of x-coordinates (x_average(Box_e)) and the average value of y-coordinates (y_average(Box_e)) among the position information (Box_e) of the bounding box (20) at the second point in time.
[0294] The electronic device (100) can obtain an average coordinate value (Pe-average) including an average value of x-coordinates (x_average(Box_e)) and an average value of y-coordinates (y_average(Box_e)) at a second point in time.
[0295] Referring to the embodiment (1320) of FIG. 13, the electronic device (100) can obtain the movement direction of the target object (10). The electronic device (100) can obtain the movement direction ( 3) can be obtained. The direction of movement can be described in degrees.
[0296] The first distance (x3) may be the absolute value of the average value of the x-coordinates at the second time point (x_average(Box_e)) minus the average value of the x-coordinates at the first time point (x_average(Box_s)).
[0297] The second distance (y3) may be the absolute value of the average value of the y-coordinates at the second time point (y_average(Box_e)) minus the average value of the y-coordinates at the first time point (y_average(Box_s)).
[0298] Referring to the embodiment (1330) of FIG. 13, the electronic device (100) can obtain the movement speed (v3) using the movement distance (d3) and the time interval (△t).
[0299] The distance traveled (d3) may be the square root of the sum of the squares of the first distance (x3) and the squares of the second distance (y3).
[0300] The time interval (△t) can be the difference between the first time point and the second time point.
[0301] The electronic device (100) can obtain the movement speed (v3) by dividing the movement distance (d3) by the time interval (△t).
[0302] Figure 14 is a drawing for explaining the letter box area and content area.
[0303] Referring to the embodiment (1410) of FIG. 14, the electronic device (100) can divide the entire area included in the content into a letter box area (31, 32) and a content area (33) to identify a preset event.
[0304] When there are two letter box areas, the letter box placed at the top is described as the first letter box area (31), and the letter box placed at the bottom is described as the second letter box area (32).
[0305] If the content does not include a letter box, the electronic device (100) may arbitrarily distinguish between a letter box area and a content area. For example, the electronic device (100) may distinguish the entire area into a letter box area and a content area based on a letter box of a preset size.
[0306] If the content includes a letter box, the electronic device (100) can identify the letter box area and the content area included in the content.
[0307] The coordinates for representing the area can have the origin (0,0) at the upper left. The x-coordinate value can increase in the x-direction from the origin. The y-coordinate value can increase in the y-direction from the origin.
[0308] The vertical length of the entire area of the content can be Y_H.
[0309] The vertical length of the letter box area may be Y_L. If there are two letter boxes, the vertical lengths of each letter box (31, 32) may be the same.
[0310] The vertical length of the content area can be Y_H-2(Y_L). The vertical length of the content area can be the value (Y_H-2(Y_L)) obtained by subtracting the sum of the vertical lengths of two letter boxes (2(Y_L)) from the vertical length of the entire area (Y_H).
[0311] If the y-coordinate value of the target object (10) is greater than 0 and less than or equal to Y_L, the electronic device (100) can identify that the target object (10) is in the first letter box area (31).
[0312] If the y-coordinate value of the target object (10) is greater than Y_L and less than or equal to Y_H-Y_L, the electronic device (100) can identify that the target object (10) is in the content area (33).
[0313] If the y-coordinate value of the target object (10) is greater than Y_H-Y_L and less than or equal to Y_H, the electronic device (100) can identify that the target object (10) is in the second letter box area (32).
[0314] FIG. 15 is a diagram for explaining a preset event of a target object according to one embodiment.
[0315] Referring to the embodiment (1510) of FIG. 15, the electronic device (100) can identify a target object (10) in the content area (33).
[0316] When the minimum coordinate value (21) and the maximum coordinate value (22) are in the content area (33), the electronic device (100) can identify that the target object (10) is in the content area (33).
[0317] When the minimum coordinate value (21), maximum coordinate value (22), and average coordinate value (23) are in the content area (33), the electronic device (100) can identify that the target object (10) is in the content area (33).
[0318] In one or more embodiments, if the y-coordinate value of the minimum coordinate value (21) is greater than Y_L and less than or equal to Y_H-Y_L, and the y-coordinate value of the maximum coordinate value (22) is greater than Y_L and less than or equal to Y_H-Y_L, the electronic device (100) can identify that the target object (10) is in the content area (33).
[0319] Referring to the embodiment (1520) of FIG. 15, the electronic device (100) can identify the target object (10) at least one of the boundary line (y=Y_L) between the first letter box area (31) and the content area (33) or the boundary line (y=Y_H-Y_L) between the second letter box area (32) and the content area (33).
[0320] In one or more embodiments, if the y-coordinate value of the minimum coordinate value (21) is Y_L, the electronic device (100) can identify that the target object (10) is in contact with the boundary line (y=Y_L) between the first letter box area (31) and the content area (33).
[0321] In one or more embodiments, if the y-coordinate value of the maximum coordinate value (22) is less than or equal to Y_H-Y_L, the electronic device (100) can identify that the target object (10) is in contact with the boundary line (y=Y_H-Y_L) between the second letter box area (32) and the content area (33).
[0322] Referring to the embodiment (1530) of FIG. 15, the electronic device (100) can identify the target object (10) in at least one of the first letter box area (31) or the second letter box area (32).
[0323] In one or more embodiments, if the y-coordinate value of the minimum coordinate value (21) is greater than 0 and less than or equal to Y_L, the electronic device (100) can identify that at least a portion of the target object (10) is in the first letter box area (31).
[0324] In one or more embodiments, if the y-coordinate value of the maximum coordinate value (22) is greater than Y_H-Y_L and less than or equal to Y_H, the electronic device (100) can identify that the target object (10) is at least partially located in the second letter box area (32).
[0325] In one or more embodiments, if the y-coordinate value of the minimum coordinate value (21) is greater than 0 and less than or equal to Y_L, and the y-coordinate value of the maximum coordinate value (22) is greater than Y_H-Y_L and less than or equal to Y_H, the electronic device (100) can identify that the target object (10) is in the first letter box area (31), the second letter box area (32), and the content area (33).
[0326] The preset event may be an event in which the minimum coordinate value (21) is located in the content area (33) and then in the first letter box area (31) or the second letter box area (32).
[0327] The preset event may be an event in which the maximum coordinate value (22) is located in the content area (33) and then in the first letter box area (31) or the second letter box area (32).
[0328] The preset event may be an event in which the minimum coordinate value (21) is located in the content area (33) and then in the first letter box area (31), and the maximum coordinate value (22) is located in the content area (33) and then in the second letter box area (32).
[0329] FIG. 16 is a drawing for explaining a composite image with a 3D effect applied according to one embodiment.
[0330] Referring to FIG. 16, the electronic device (100) can obtain an input image (1610). The electronic device (100) can obtain a composite image (1620) by applying a 3D effect to the input image (1610).
[0331] When events that are identified in the order of the embodiments (1510, 1520, 1530) of FIG. 15 occur, the electronic device (100) can generate a composite image (1620).
[0332] FIG. 17 is a diagram for explaining a preset event of a target object according to one embodiment.
[0333] Referring to the embodiment (1710) of FIG. 17, the electronic device (100) can identify a target object (10) in the content area (33).
[0334] In one or more embodiments, if the y-coordinate value of the minimum coordinate value (21) is greater than Y_L and less than or equal to Y_H-Y_L, and the y-coordinate value of the maximum coordinate value (22) is greater than Y_L and less than or equal to Y_H-Y_L, the electronic device (100) can identify that the target object (10) is in the content area (33).
[0335] Referring to the embodiment (1720) of FIG. 17, the electronic device (100) can identify that the target object (10) is in contact with the boundary line (y=Y_L) between the first letter box area (31) and the content area (33).
[0336] In one or more embodiments, if the y-coordinate value of the minimum coordinate value (21) is Y_L and the y-coordinate value of the maximum coordinate value (22) is greater than Y_L and less than or equal to Y_H-Y_L, the electronic device (100) can identify that the target object (10) is in contact with the boundary line (y=Y_L) between the first letter box area (31) and the content area (33).
[0337] Referring to embodiment 1730 of FIG. 17, the electronic device (100) can identify that at least a portion of the target object (10) is in the first letter box area (31).
[0338] In one or more embodiments, if the y-coordinate value of the minimum coordinate value (21) is greater than 0 and less than or equal to Y_L, and the y-coordinate value of the maximum coordinate value (22) is greater than Y_L and less than or equal to Y_H-Y_L, the electronic device (100) can identify that at least a portion of the target object (10) is in the first letter box area (31).
[0339] The preset event may be an event in which the minimum coordinate value (21) is located in the content area (33) and then in the first letter box area (31).
[0340] FIG. 18 is a drawing for explaining a composite image with a 3D effect applied according to one embodiment.
[0341] Referring to FIG. 18, the electronic device (100) can obtain an input image (1810). The electronic device (100) can obtain a composite image (1820) by applying a 3D effect to the input image (1810).
[0342] When events that are identified in the sequence of the embodiments (1710, 1720, 1730) of FIG. 17 occur, the electronic device (100) can generate a composite image (1820).
[0343] FIG. 19 is a diagram for explaining a preset event of a target object according to one embodiment.
[0344] Referring to the embodiment (1910) of FIG. 19, the electronic device (100) can identify the target object (10) in the first letter box area (31).
[0345] In one or more embodiments, if the y-coordinate value of the minimum coordinate value (21) is greater than 0 and less than or equal to Y_L, and the y-coordinate value of the maximum coordinate value (22) is greater than 0 and less than or equal to Y_L, the electronic device (100) can identify that the target object (10) is in the first letter box area (31).
[0346] Referring to the embodiment (1920) of FIG. 19, the electronic device (100) can identify that the target object (10) is in contact with the boundary line (y=Y_L) between the first letter box area (31) and the content area (33).
[0347] In one or more embodiments, if the y-coordinate value of the minimum coordinate value (21) is greater than 0 and less than or equal to Y_L, and the y-coordinate value of the maximum coordinate value (22) is Y_L, the electronic device (100) can identify that the target object (10) is in contact with the boundary line (y=Y_L) between the first letter box area (31) and the content area (33).
[0348] Referring to the embodiment (1930) of FIG. 19, the electronic device (100) can identify a target object (10) in the content area (33).
[0349] In one or more embodiments, if the y-coordinate value of the minimum coordinate value (21) is greater than Y_L and less than or equal to Y_H-Y_L, and the y-coordinate value of the maximum coordinate value (22) is greater than Y_L and less than or equal to Y_H-Y_L, the electronic device (100) can identify that the target object (10) is in the content area (33).
[0350] The preset event may be an event in which the maximum coordinate value (22) is located in the first letter box area (31) and then in the content area (33).
[0351] The preset event may be an event in which the first letter box area (31) is located in the second letter box area (32) and then in the content area (33).
[0352] FIG. 20 is a drawing for explaining a composite image with a 3D effect applied according to one embodiment.
[0353] Referring to FIG. 20, the electronic device (100) can obtain an input image (2010). The electronic device (100) can obtain a composite image (2020) by applying a 3D effect to the input image (2010).
[0354] When events that are identified in the sequence of the embodiments (1910, 1920, 1930) of FIG. 19 occur, the electronic device (100) can generate a composite image (2020).
[0355] FIG. 21 is a diagram for explaining a preset event of a target object according to one embodiment.
[0356] Embodiments (2110) and (2120) of FIG. 21 may correspond to embodiments (1510) and (1520) of FIG. 15. Therefore, redundant descriptions are omitted. FIG. 15 is an embodiment for analyzing content without letter boxes, and FIG. 21 is an embodiment for analyzing content with letter boxes.
[0357] Referring to the embodiment (2130) of FIG. 21, the electronic device (100) may not be able to identify the minimum coordinate value (21) and the average coordinate value (23). This is because the target object (10) may be outside the content area (33). In a content with a letter box, if the target object (10) is outside the content area (33), the electronic device (100) may not be able to accurately determine the location of the target object (10) in the area with the letter box.
[0358] The electronic device (100) can identify the location of the target object (10) using the average coordinate value (23) of the target object (10).
[0359] If the y-coordinate value of the minimum coordinate value (21) is not identified, the y-coordinate value of the maximum coordinate value (22) is not identified, and the y-coordinate value of the average coordinate value (23) is greater than Y_L and less than or equal to Y_H-Y_L, the electronic device (100) can identify that the target object (10) is in the first letter box area (31), the second letter box area (32), and the content area (33).
[0360] The electronic device (100) can identify that the vertical length of the target object (10) is greater than the vertical length of the content area (33).
[0361] If at least one of the minimum coordinate value (21), maximum coordinate value (22), and average coordinate value (23) is not identified, the electronic device (100) can obtain (or estimate or calculate) unidentified location information based on the first letter box area (31), the second letter box area (32), the content area (33), the size of the target object (10), and the image size.
[0362] FIG. 22 is a drawing for explaining a composite image with a 3D effect applied according to one embodiment.
[0363] Referring to FIG. 22, the electronic device (100) can obtain an input image (2210). The electronic device (100) can obtain a composite image (2220) by applying a 3D effect to the input image (2210).
[0364] When events that are identified in the order of the embodiments (2110, 2120, 2130) of FIG. 21 occur, the electronic device (100) can generate a composite image (2220).
[0365] FIG. 23 is a diagram for explaining a preset event of a target object according to one embodiment.
[0366] Embodiments (2310) and (2320) of FIG. 23 may correspond to embodiments (1710) and (1720) of FIG. 17. Therefore, redundant descriptions are omitted. FIG. 17 is an embodiment for analyzing content without letter boxes, and FIG. 23 is an embodiment for analyzing content with letter boxes.
[0367] Referring to the embodiment (2330) of FIG. 23, the electronic device (100) may not be able to identify the minimum coordinate value (21). This is because the target object (10) may be outside the content area (33). In a content with a letter box, if the target object (10) is outside the content area (33), the electronic device (100) may not be able to accurately determine the location of the target object (10) in the area with the letter box.
[0368] The electronic device (100) can identify the location of the target object (10) using the average coordinate value (23) of the target object (10).
[0369] If the y-coordinate value of the minimum coordinate value (21) is not identified and the y-coordinate value of the maximum coordinate value (22) is greater than Y_L and less than or equal to Y_H-Y_L, the electronic device (100) can identify that the target object (10) is in the first letter box area (31) and the content area (33).
[0370] FIG. 24 is a drawing for explaining a composite image with a 3D effect applied according to one embodiment.
[0371] Referring to FIG. 24, the electronic device (100) can obtain an input image (2410). The electronic device (100) can obtain a composite image (2420) by applying a 3D effect to the input image (2410).
[0372] When events that are identified in the order of the embodiments (2310, 2320, 2330) of FIG. 23 occur, the electronic device (100) can generate a composite image (2420).
[0373] FIG. 25 is a drawing for explaining a preset event of a target object according to one embodiment.
[0374] Embodiments (2520) and (2530) of FIG. 25 may correspond to Embodiments (1920) and (1930) of FIG. 19. Therefore, redundant descriptions are omitted. FIG. 19 is an embodiment for analyzing content without letter boxes, and FIG. 25 is an embodiment for analyzing content with letter boxes.
[0375] Referring to the embodiment (2510) of FIG. 25, the electronic device (100) may not be able to identify the minimum coordinate value (21), the maximum coordinate value (22), and the average coordinate value (23). This is because the target object (10) may be outside the content area (33). In a content with a letter box, if the target object (10) is outside the content area (33), the electronic device (100) may not be able to accurately determine the location of the target object (10) in the area with the letter box.
[0376] FIG. 26 is a drawing for explaining a composite image with a 3D effect applied according to one embodiment.
[0377] Referring to FIG. 26, the electronic device (100) can obtain an input image (2610). The electronic device (100) can obtain a composite image (2620) by applying a 3D effect to the input image (2610).
[0378] When events that are identified in the order of the embodiments (2510, 2520, 2530) of FIG. 25 occur, the electronic device (100) can generate a composite image (2620).
[0379] Figure 27 is a drawing for explaining an example in which content information is damaged.
[0380] Referring to FIG. 27, the electronic device (100) can obtain an input image (2710) including content information (2711, 2712). Depending on the location where the content information (2711, 2712) is displayed, at least a portion of the content information (2711, 2712) may not be displayed in the composite image (2720).
[0381] If the content information (2711, 2712) is key information that needs to be conveyed to the viewer, there may be a problem in which the key information is omitted from the composite image (2720).
[0382] FIG. 28 is a drawing for explaining an operation of providing content information according to one embodiment.
[0383] Referring to the embodiment (2810) of FIG. 28, the electronic device (100) can determine the size (or vertical length in the y-axis direction) of the letter box based on the position information of the content information (2811, 2812). The electronic device (100) can adjust the size of the letter box so that the content information (2811, 2812) is not deleted depending on the letter box being created (or added).
[0384] The electronic device (100) can identify content information (2811, 2812) in an input image.
[0385] The electronic device (100) can obtain location information of content information (2811, 2812). The electronic device (100) can obtain the minimum y-coordinate value among the location information of the content information (2811, 2812). The electronic device (100) can determine the minimum y-coordinate value among the location information of the content information (2811, 2812) as the vertical length of the letter box.
[0386] FIG. 29 is a diagram for explaining an operation of providing content information according to one embodiment.
[0387] Referring to the embodiment (2910) of FIG. 29, the electronic device (100) can generate a composite image by changing the location of content information (2911, 2912).
[0388] When content information (2911, 2912) is identified in the input image, the electronic device (100) can identify the location of the content information (2911, 2912). The electronic device (100) can extract only the content information (2911, 2912). The electronic device (100) can include the extracted content information (2911, 2912) in the composite image.
[0389] The electronic device (100) can generate (or obtain) a second composite image by adding extracted content information (2911, 2912) to the first composite image.
[0390] The location where content information (2911, 2912) is displayed may vary depending on the user's settings.
[0391] In one or more embodiments, the electronic device (100) may display content information (2911, 2912) at a preset distance from the generated letter box.
[0392] In one or more embodiments, the electronic device (100) may display content information (2911, 2912) at a second location that is spaced apart by the vertical length of a letter box generated from the first location from which the content information (2911, 2912) was extracted.
[0393] FIG. 30 is a drawing for explaining an operation of generating a composite image when there is no letter box according to one embodiment.
[0394] Referring to FIG. 30, the electronic device (100) can obtain an input image (3010).
[0395] The electronic device (100) can obtain a base layer (3021) based on an input image (3010). The base layer (3021) may be an image to which a letter box has been added to the input image (3010). The electronic device (100) can generate a base layer (3121) including a letter box while the resolution of the input image (3010) remains unchanged.
[0396] The electronic device (100) can obtain a 3D layer (3022) based on the input image (3010). The 3D layer (3022) can include a portion of a target object. The portion of the target object included in the 3D layer (3022) can be information displayed in a letter box. The electronic device (100) can extract a portion of the target object displayed in the letter box area from the input image (3010) and include it in the 3D layer (3022).
[0397] The 3D layer (3022) may not store data in a location corresponding to the content area (33) or may contain null data.
[0398] The electronic device (100) can generate (or obtain) a composite image (3030) by combining the base layer (3021) and the 3D layer (3022). The electronic device (100) can provide the composite image (3030) to the user as an output image.
[0399] FIG. 31 is a diagram illustrating an operation of generating a composite image when there is no letter box according to one embodiment.
[0400] The input image (3110), base layer (3121), 3D layer (3122), and composite image (3130) of FIG. 31 may correspond to the input image (3010), base layer (3021), 3D layer (3022), and composite image (3030) of FIG. 30. Therefore, redundant description is omitted.
[0401] The 3D layer (3122) of FIG. 31 may include a letter box, unlike the 3D layer (3022) of FIG. 30. The electronic device (100) may obtain a 3D layer (3122) including a letter box and a portion of a target object to be displayed on the letter box.
[0402] When letter boxes are added to both the base layer (3121) and the 3D layer (3122) as in Fig. 31, the letter boxes can be displayed overlappingly, and the target object can be displayed more naturally.
[0403] FIG. 32 is a drawing for explaining an operation of generating a composite image when there is no letter box according to one embodiment.
[0404] The input image (3210), base layer (3221), 3D layer (3222), and composite image (3230) of FIG. 32 may correspond to the input image (3110), base layer (3121), 3D layer (3122), and composite image (3130) of FIG. 31. Therefore, redundant description is omitted.
[0405] The base layer (3221) of FIG. 32 may not have a letter box added, unlike the base layer (3121) of FIG. 31. Since the 3D layer (3122) includes a letter box, the electronic device (100) may not add a letter box to the base layer (3121).
[0406] As in Fig. 32, adding a letter box to the 3D layer (3122) without adding a letter box to the base layer (3121) can reduce the amount of computation.
[0407] FIG. 33 is a drawing for explaining an operation of generating a composite image when there is no letter box according to one embodiment.
[0408] Referring to FIG. 33, the electronic device (100) can obtain an input image (3310).
[0409] The electronic device (100) can extract a target object from an input image (3310).
[0410] The electronic device (100) can obtain a base layer (3321) by removing a target object from an input image (3310) and changing the resolution of the input image (3310). The electronic device (100) can obtain a base layer (3321) from which a target object has been removed based on the input image (3310).
[0411] The base layer (3321) may be an image with a letter box added to the input image (3310). The electronic device (100) may change the resolution of the input image (3310). The electronic device (100) may change the resolution of the input image (3310) based on the content area (33). The electronic device (100) may change the resolution of the input image (3310) so that all image information included in the input image (3310) is displayed in the content area (33). With the resolution of the input image (3310) changed, the electronic device (100) may generate a base layer (3321) including a letter box.
[0412] The electronic device (100) can obtain a 3D layer (3322) based on the input image (3310). The 3D layer (3322) can include extracted target objects. Some of the target objects included in the 3D layer (3322) can be displayed in a letter box, and other parts can be displayed in a content area.
[0413] The electronic device (100) can generate (or obtain) a composite image (3330) by combining the base layer (3321) and the 3D layer (3322). The electronic device (100) can provide the composite image (3330) to the user as an output image.
[0414] FIG. 34 is a drawing for explaining an operation of generating a composite image when there is a letter box according to one embodiment.
[0415] Referring to FIG. 34, the electronic device (100) can obtain an input image (3410).
[0416] The electronic device (100) can obtain a first base layer (3421-1) by removing a letter box from an input image (3410). The first base layer (3421-1) may be a layer with a changed resolution. The size of the input image (3410) may increase (or expand) as the letter box is removed from the input image (3410). The electronic device (100) can obtain a second base layer (3421-2) by adding a letter box of a preset size to the first base layer (3421-1). The electronic device (100) can obtain the second base layer (3421-2) by maintaining the resolution of the first base layer (3421-1).
[0417] The electronic device (100) can obtain a 3D layer (3422) based on the input image (3410). The 3D layer (3422) can include a portion of a target object. The portion of the target object included in the 3D layer (3422) can be information displayed in a letter box. The electronic device (100) can extract a portion of the target object displayed in the letter box area from the first basic layer (3421-1) and include it in the 3D layer (3422).
[0418] The 3D layer (3422) may not store data in a location corresponding to the content area (33) or may contain null data.
[0419] The electronic device (100) can generate (or obtain) a composite image (3430) by combining the second base layer (3421-2) and the 3D layer (3422). The electronic device (100) can provide the composite image (3430) to the user as an output image.
[0420] FIG. 35 is a drawing for explaining an operation of generating a composite image when there is a letter box according to one embodiment.
[0421] Referring to FIG. 35, the electronic device (100) can obtain an input image (3510). The electronic device (100) can identify a letter box in the input image (3510). The electronic device (100) can identify a target object that is outside the content area (33) in the input image (3510).
[0422] The electronic device (100) can obtain an input image (3510) as a base layer (3521).
[0423] The electronic device (100) can determine whether the target object can be displayed in the letter box. If the target object can be displayed in the letter box (or a preset event is identified), the electronic device (100) can obtain a 3D layer (3522) by generating a portion of the target object so that the target object can be displayed in the letter box.
[0424] Information (or shape) of the target object included in the 3D layer (3522) may not be included in the input image (3510). The electronic device (100) may additionally generate a portion of the target object to be displayed and include it in the 3D layer (3522).
[0425] The electronic device (100) can generate (or obtain) a composite image (3530) by combining the base layer (3521) and the 3D layer (3522). The electronic device (100) can provide the composite image (3530) to the user as an output image.
[0426] FIG. 36 is a drawing for explaining an operation of generating a composite image when there is a letter box according to one embodiment.
[0427] The input image (3610), base layer (3621), 3D layer (3622), and composite image (3630) of FIG. 36 may correspond to the input image (3510), base layer (3521), 3D layer (3522), and composite image (3530) of FIG. 35. Therefore, redundant description is omitted.
[0428] The 3D layer (3622) of FIG. 36 may include a letter box, unlike the 3D layer (3522) of FIG. 35. The electronic device (100) may obtain a 3D layer (3622) including a letter box and a portion of a target object to be displayed on the letter box.
[0429] When letter boxes are added to both the base layer (3621) and the 3D layer (3622) as in Fig. 36, the letter boxes can be displayed overlappingly, and the target object can be displayed more naturally.
[0430] FIG. 37 is a drawing for explaining an operation of generating a composite image when there is a letter box according to one embodiment.
[0431] Referring to FIG. 37, the electronic device (100) can obtain an input image (3710).
[0432] The electronic device (100) can extract a target object from an input image (3710).
[0433] The electronic device (100) can obtain a base layer (3721) by removing a target object from an input image (3710). The electronic device (100) can obtain the base layer (3721) while maintaining the resolution of the input image (3710).
[0434] The electronic device (100) may acquire a 3D layer (3722) based on the input image (3710). The 3D layer (3722) may include a portion of the extracted target object and an additionally generated target object. Some of the target objects included in the 3D layer (3722) may be displayed in a letter box, and other portions may be displayed in a content area. Some of the target objects displayed in the letter box (or letter box area) may be newly generated information by the electronic device (100).
[0435] The electronic device (100) can generate (or obtain) a composite image (3730) by combining a base layer (3721) and a 3D layer (3722). The electronic device (100) can provide the composite image (3730) to a user as an output image.
[0436] Figure 38 is a drawing for explaining the operation of changing the size of a letter box.
[0437] Referring to FIG. 38, the electronic device (100) can identify a target section to which a 3D effect is to be applied among the entire section of the content.
[0438] Assume that a letter box is added to apply a 3D effect to content that does not include a letter box. The electronic device (100) can identify a target section (t2).
[0439] The target interval (t2) can be from t_start to t_end.
[0440] The electronic device (100) can gradually (or linearly) change the vertical length of the letter box to provide a natural 3D effect.
[0441] The electronic device (100) can change the vertical length of the letter box from 0 to Y_L for a first threshold time (t1).
[0442] The electronic device (100) can change the vertical length of the letter box from Y_L to 0 for a second threshold time (t3).
[0443] The first critical time (t1) may be equal to the second critical time (t3).
[0444] The first threshold time (t1) and the second threshold time (t3) can be changed according to the user's settings.
[0445] The first critical time (t1) and the second critical time (t3) can be determined based on the target interval (t2). If the target interval (t2) becomes longer, the first critical time (t1) and the second critical time (t3) can also become longer. If the target interval (t2) becomes shorter, the first critical time (t1) and the second critical time (t3) can also become shorter.
[0446] Within a cycle, the starting point where a letter box is first created can be t_in, and the ending point where a letter box last exists can be t_out.
[0447] Prior to the t_in time, the electronic device (100) may provide an image (3810) without letterboxing.
[0448] At a first threshold time (t1), the electronic device (100) may provide an image (3820) including a gradually increasing letter box.
[0449] In the target section (t2), the electronic device (100) can provide an image (3830) including a fixed letter box.
[0450] At a second threshold time (t3), the electronic device (100) can provide an image (3840) including a gradually decreasing letter box.
[0451] After the t_out point, the electronic device (100) can provide an image (3850) without letterboxing.
[0452] The electronic device (100) can change the color information of the letter box as the size of the letter box changes. As the size of the letter box changes, the color of the letter box can also change. As the size of the letter box increases, the color of the letter box can become darker. As the size of the letter box increases, the brightness of the letter box can decrease (a darker feeling in the same color). As the size of the letter box decreases, the brightness of the letter box can increase (a brighter feeling in the same color).
[0453] Figure 39 is a drawing for explaining an operation of playing content by setting a delay time.
[0454] Referring to FIG. 39, the electronic device (100) can provide content in a streaming manner. When using the streaming manner, the electronic device (100) can provide content received from an external server (or external device) to the user in real time.
[0455] The electronic device (100) can delay the playback time by a delay time to secure processing time for applying a 3D effect. The electronic device (100) can display a preset screen (3910) during the third threshold time (t4) during which the playback time is delayed.
[0456] The third critical time (t4) may be a delay time.
[0457] The preset screen (3910) may include at least one of a graphic UI (3911) representing a 3D effect or a text UI (3912) representing a 3D effect.
[0458] The electronic device (100) can provide an image (3920) with a 3D effect applied to the target section (t2).
[0459] The third threshold time (t4) may be the time required to apply a 3D effect to a single target section. The third threshold time (t4) may be a preset time. When the third threshold time (t4) is set, the electronic device (100) can seamlessly provide the user with the 3D effect even without downloading all of the content.
[0460] Figure 40 is a flowchart for explaining a control method of an electronic device (100) according to one embodiment.
[0461] Referring to FIG. 40, a method for controlling an electronic device storing content includes a step of identifying a preset target object in the content (S4005), a step of obtaining location information of the target object (S4010), a step of identifying a target section to which a 3D effect for displaying a part of the target object in a letter box is to be applied based on the location information of the target object (S4015), a step of obtaining a base layer corresponding to a background of the target image and a 3D layer corresponding to the 3D effect based on a target image included in the target section (S4020), a step of obtaining a composite image including a letter box by combining the base layer and the 3D layer (S4025), and a step of providing the composite image (S4030).
[0462] The control method further includes a step of identifying whether the content includes a letter box, and the step of obtaining a composite image (S4025) may obtain a composite image by adding a letter box to at least one of the base layer or the 3D layer if the content does not include a letter box.
[0463] The step of acquiring a 3D layer can acquire a 3D image corresponding to a target object to be displayed in a letter box, and acquire a 3D layer including the 3D image.
[0464] The control method further includes a step of determining a position of a letter box based on a position of content information included in the content, wherein the content information may include at least one of a graphical UI (User Interface) or a text UI representing the content.
[0465] The target object is an object whose position changes in multiple images included in the content, and the step of identifying the target object (S4005) can identify the target object among the multiple objects by analyzing the characteristics of the multiple objects identified in the content.
[0466] The step of identifying a target section (S4015) can identify a target section based on the time at which the preset event is identified when a preset event is identified based on location information.
[0467] The step of identifying a target section (S4015) distinguishes between a letter box area and a content area among the entire area of the content, and when an event in which a target object moves from the letter box area to the content area or an event in which a target object moves from the content area to the letter box area is identified, it can be identified that a preset event has been identified.
[0468] The step of identifying a preset event may include obtaining first location information of a target object at a first time point and second location information of the target object at a second time point, and identifying a preset event based on the first location information and the second location information.
[0469] The first location information includes at least one of a minimum coordinate value, a maximum coordinate value, or an average coordinate value of the target object at a first point in time, the second location information includes at least one of a minimum coordinate value, a maximum coordinate value, or an average coordinate value of the target object at a second point in time, and the step of identifying a preset event may obtain movement information of the target object based on the first location information and the second location information, and identify a preset event based on the movement information.
[0470] The step of obtaining location information (S4010) may identify a bounding box of a target object, obtain first location information based on a plurality of coordinate values representing the bounding box at a first point in time, and obtain second location information based on a plurality of coordinate values representing the bounding box at a second point in time.
[0471] The control method of an electronic device (100) such as FIG. 40 can be executed on an electronic device (100) having the configuration of FIG. 2 or FIG. 3, and can also be executed on an electronic device having other configurations.
[0472] The methods according to the various embodiments of the present disclosure described above can be implemented in the form of an application that can be installed on an existing electronic device.
[0473] Additionally, the methods according to the various embodiments of the present disclosure described above can be implemented only with a software upgrade or a hardware upgrade for an existing electronic device.
[0474] Additionally, the various embodiments of the present disclosure described above may also be performed through an embedded server provided in an electronic device, or an external server of at least one of the electronic device and the display device.
[0475] According to an example embodiment of the present disclosure, the various embodiments described above may be implemented as software including instructions stored in a machine-readable storage medium that can be read by a machine (e.g., a computer). The device may include an electronic device according to the disclosed embodiments, which is a device that can call instructions stored in the storage medium and operate according to the called instructions. When the instructions are executed by a processor, the processor may directly or under the control of the processor use other components to perform a function corresponding to the instructions. The instructions may include code generated or executed by a compiler or interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' means that the storage medium does not contain signals and is tangible, but does not distinguish between data being stored semi-permanently or temporarily in the storage medium.
[0476] Furthermore, according to one embodiment of the present disclosure, the method according to the various embodiments described above may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)) or online through an application store (e.g., Play Store™). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated in a storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.
[0477] In addition, each of the components (e.g., modules or programs) according to the various embodiments described above may be composed of a single or multiple entities, and some of the corresponding sub-components described above may be omitted, or other sub-components may be further included in various embodiments. Alternatively or additionally, some components (e.g., modules or programs) may be integrated into a single entity, which may perform the same or similar functions as those performed by each of the corresponding components prior to integration. Operations performed by modules, programs or other components according to various embodiments may be executed sequentially, in parallel, iteratively or heuristically, or at least some operations may be executed in a different order, omitted, or other operations may be added.
[0478] While specific embodiments of the present disclosure have been specifically illustrated and described, it will be understood that various changes in form and detail may be made therein without departing from the spirit and scope of the following claims.
Claims
1. In electronic devices, At least one memory for storing content and instructions; comprising at least one processor connected to said memory; At least one processor executing said command, Identifying a target object from the content stored in at least one memory, Obtain location information of the above target object, Identifying a target section to which a 3D effect for displaying a part of the target object in a letter box is applied based on the position information of the target object, the target section including a target image, Based on the target image, a base layer corresponding to the background of the target image and a 3D layer corresponding to the 3D effect are obtained, By combining the above base layer and the above 3D layer, a composite image including the letter box is obtained, An electronic device providing the above composite image.
2. In paragraph 1, At least one processor executing said command, Identify whether the above content includes the above letterbox, An electronic device that obtains the composite image by adding the letter box to at least one of the base layer or the 3D layer if the content does not include the letter box.
3. In paragraph 1, At least one processor executing said command, Acquire a 3D image corresponding to the above target object, An electronic device for obtaining the 3D layer including the 3D image.
4. In paragraph 1, At least one processor executing said command, Determine the location of the letter box based on the location of the content information included in the above content, The above content information is: An electronic device comprising at least one of a graphical UI (User Interface) or a text UI representing the above content.
5. In paragraph 1, The above content contains multiple images, The above target object is, The position is changed in the above multiple images, At least one processor executing said command, An electronic device that analyzes the characteristics of a plurality of objects identified in the above content and identifies the target object among the plurality of objects.
6. In paragraph 1, At least one processor executing said command, Identify the preset event based on the above location information, An electronic device that identifies the target section based on the point in time at which the above-described preset event is identified.
7. In paragraph 6, At least one processor executing said command, Identify the letterbox area and the content area among the entire area of the above content, Identify a first event in which the target object moves from the letter box area to the content area or a second event in which the target object moves from the content area to the letter box area, An electronic device that identifies the first event or the second event as the preset event.
8. In paragraph 7, At least one processor executing said command, Obtaining first location information of the target object at a first time point and second location information of the target object at a second time point, An electronic device that identifies the preset event based on the first location information and the second location information.
9. In paragraph 8, The above first location information is, At least one of the first minimum coordinate value, the first maximum coordinate value or the first average coordinate value of the target object at the first time point, The second location information is, At the second point in time, at least one of the second minimum coordinate value, the second maximum coordinate value or the second average coordinate value of the target object is included, At least one processor executing said command, Obtain movement information of the target object based on the first location information and the second location information, An electronic device that identifies the preset event based on the movement information.
10. In paragraph 9, At least one processor executing said command, Identify the bounding box of the above target object, Obtaining the first location information based on the first plurality of coordinate values representing the bounding box at the first point in time, An electronic device that obtains the second location information based on a second plurality of coordinate values representing the bounding box at the second time point.
11. A method for controlling an electronic device storing content, A step of identifying a target object in the above content; A step of obtaining location information of the above target object; A step of identifying a target section to which a 3D effect for displaying a part of the target object in a letter box is to be applied based on the position information of the target object, the target section including a target image; A step of obtaining a base layer corresponding to the background of the target image and a 3D layer corresponding to the 3D effect based on the target image; A step of combining the above base layer and the above 3D layer to obtain a composite image including the letter box; and A control method, comprising: a step of providing the above composite image; 12. In paragraph 11, The above control method is, Further comprising a step of identifying whether the above content includes the letter box; The step of obtaining the above synthetic image is: A control method for obtaining the composite image by adding the letter box to at least one of the base layer or the 3D layer if the above content does not include the letter box.
13. In paragraph 11, The steps for obtaining the above 3D layer are: Acquire a 3D image corresponding to the above target object, A control method for obtaining the 3D layer including the 3D image.
14. In paragraph 11, The above control method is, Further comprising a step of determining the position of the letter box based on the position of the content information included in the content; The above content information is: A control method comprising at least one of a graphical UI (User Interface) or a text UI representing the above content.
15. In paragraph 11, The above content contains multiple images, The above target object is, The position is changed in the above multiple images, The step of identifying the above target object is: A control method for identifying a target object among a plurality of objects by analyzing the characteristics of the plurality of objects identified in the above content.
Citation Information
Patent Citations
MOBILE TERMINAL AND METHOD FOR CONTROLLING A THREE DIMENSION IMAGE in thereof
KR101674956B1
System and method for combining text with three-dimensional content
KR1020090120492A
Mobile terminal and method for controlling the same
KR1020120022352A
Stereoscopic mapping
US20170142404A1
KR20230014336A
