Electronic apparatus and control method therefor

Abstract

This electronic apparatus comprises: a memory storing instructions; and at least one processor including processing circuitry, wherein the instructions, when individually or collectively executed by the at least one processor: obtain a video frame of a first color space and a UI frame of a second color space; identify a video area corresponding to the video frame and a UI area corresponding to the UI frame, from a display area in which the video frame and the UI frame are displayed; identify a non-overlapping video area that does not overlap the UI area from the video area; identify an overlapping video area that overlaps the UI area from the video area; obtain first encoded data by encoding the non-overlapping video area; obtain combined data by combining the overlapping video area and the UI frame; obtain second encoded data by encoding the combined data; and obtain final encoded data by merging the first encoded data and the second encoded data.

Description

Electronic device and method of controlling the same

[0001] The present disclosure relates to an electronic device and a control method thereof, and more particularly, to an electronic device that merges a video frame and a UI frame and a control method thereof.

[0002] A screen output through a display or projection lens may include multiple layers. The screen may be output by combining layers representing video frames and layers representing UI frames. A user can visually perceive a single screen composed of video frames and UI frames.

[0003] A screen that is visually perceptible to a user can be generated based on data that includes both video frames and UI frames. The electronic device can perform video processing operations to generate a screen that combines the video frames and UI frames.

[0004] Video processing operations may include combining different image elements (or image frames). Video processing operations may include layer combining operations. Performing video processing operations may require processing time.

[0005] In technologies that perform real-time video processing operations, screen generation can take a long time. Users may experience screen tearing or slow response times.

[0006] The present disclosure is designed to improve the above-described problem, and an object of the present disclosure is to provide an electronic device and a control method thereof for partially generating encoding data by combining an overlapping area of ​​a video frame and a UI frame by considering whether a video frame and a UI frame overlap.

[0007] According to one embodiment, an electronic device includes a memory storing instructions, and at least one processor including processing circuitry, wherein the instructions, when individually or collectively executed by the at least one processor, obtain a video frame of a first color space and a UI frame of a second color space, identify a video region corresponding to the video frame and a UI region corresponding to the UI frame among display regions in which the video frame and the UI frame are displayed, identify a non-overlapping video region among the video regions that does not overlap with the UI region, identify an overlapping video region among the video regions that overlaps with the UI region, encode the non-overlapping video region to obtain first encoded data, combine the overlapping video region and the UI frame to obtain combined data, encode the combined data to obtain second encoded data, and merge the first encoded data and the second encoded data to obtain final encoded data.

[0008] The above instructions, when individually or collectively executed by the at least one processor, can divide the video frame and the UI frame into a plurality of blocks, and identify the video area and the UI area based on the divided plurality of blocks.

[0009] The instructions, when individually or collectively executed by the at least one processor, may identify a first proportion of the display area occupied by the video area, identify a second proportion of the display area occupied by the UI area, obtain a target proportion by dividing the first proportion by the second proportion, and obtain the combined data based on the target proportion.

[0010] The instructions, when individually or collectively executed by the at least one processor, identify whether video rendering is possible for converting from the second color space to the first color space if the target ratio is greater than or equal to a threshold ratio, and if the video rendering is possible, obtain the combined data by applying the video rendering to the UI frame, and if the video rendering is impossible, obtain the combined data by applying UI rendering to the UI frame.

[0011] The above instructions, when individually or collectively executed by the at least one processor, can perform video rendering on the UI frame, if the video rendering is possible, to obtain a first rendering UI frame, and combine the overlapping video area and the first rendering UI frame to obtain the combined data.

[0012] When the above instructions are individually or collectively executed by the at least one processor, if the video rendering is impossible, the UI rendering can be performed on the UI frame to obtain a second rendering UI frame, the first color space conversion can be performed on the second rendering UI frame to obtain a converted UI frame, and the combined data can be obtained by combining the overlapping video area and the converted UI frame.

[0013] The above first color space conversion may include an operation of converting pixel information of the second color space into pixel information of the first color space.

[0014] The above instructions, when individually or collectively executed by the at least one processor, can identify whether the video frame is a 2D image, and if the video frame is not a 2D image, perform 2D modeling on the video frame.

[0015] The first color space may be YUV or YUVA, and the second color space may be RGB or RGBA.

[0016] The electronic device includes a communication interface connected to an XR (Extended Reality) device, and can transmit the final encoded data to the XR device through the communication interface.

[0017] According to one embodiment, a control method of an electronic device includes the steps of obtaining a video frame of a first color space and a UI frame of a second color space, identifying a video region corresponding to the video frame and a UI region corresponding to the UI frame among display regions in which the video frame and the UI frame are displayed, identifying a non-overlapping video region among the video regions that does not overlap with the UI region, identifying an overlapping video region among the video regions that overlaps with the UI region, encoding the non-overlapping video region to obtain first encoded data, combining the overlapping video region and the UI frame to obtain combined data, encoding the combined data to obtain second encoded data, and merging the first encoded data and the second encoded data to obtain final encoded data.

[0018] The step of identifying the UI video area and the UI area may include dividing the video frame and the UI frame into a plurality of blocks, and identifying the video area and the UI area based on the divided plurality of blocks.

[0019] The control method includes a step of identifying a first proportion of the display area occupied by the video area, a step of identifying a second proportion of the display area occupied by the UI area, and a step of obtaining a target proportion by dividing the first proportion by the second proportion, and the step of obtaining the combined data can obtain the combined data based on the target proportion.

[0020] The control method includes a step of identifying whether video rendering for converting from the second color space to the first color space is possible if the target ratio is greater than or equal to a threshold ratio, and the step of obtaining the combined data includes obtaining the combined data by applying the video rendering to the UI frame if the video rendering is possible, and obtaining the combined data by applying UI rendering to the UI frame if the video rendering is impossible.

[0021] The step of obtaining the combined data may include, if the video rendering is possible, performing the video rendering on the UI frame to obtain a first rendering UI frame, and combining the overlapping video area and the first rendering UI frame to obtain the combined data.

[0022] The step of obtaining the combined data may include, if the video rendering is impossible, performing the UI rendering on the UI frame to obtain a second rendering UI frame, performing a first color space conversion on the second rendering UI frame to obtain a converted UI frame, and combining the overlapping video area and the converted UI frame to obtain the combined data.

[0023] The above first color space conversion may include an operation of converting pixel information of the second color space into pixel information of the first color space.

[0024] The control method may include a step of identifying whether the video frame is a 2D image and a step of performing 2D modeling on the video frame if the video frame is not a 2D image.

[0025] The first color space may be YUV or YUVA, and the second color space may be RGB or RGBA.

[0026] The above control method may include a step of transmitting the final encoded data to an XR (Extended Reality) device.

[0027] The above and other aspects, features and advantages of specific embodiments of the present invention will become more apparent from the following description taken in conjunction with the accompanying drawings.

[0028] FIG. 1 is a diagram illustrating an operation of combining a video frame and a UI frame according to one embodiment.

[0029] FIG. 2 is a block diagram illustrating an electronic device according to one embodiment.

[0030] FIG. 3 is a block diagram illustrating a specific configuration of the electronic device of FIG. 2, according to one embodiment.

[0031] FIG. 4 is a diagram for explaining a 2D video processing structure according to one embodiment.

[0032] FIG. 5 is a diagram for explaining a 3D video processing structure according to one embodiment.

[0033] FIG. 6 is a diagram for explaining a color space conversion operation according to one embodiment.

[0034] FIG. 7 is a diagram illustrating an operation of identifying a video area and a UI area to obtain a target ratio according to one embodiment.

[0035] FIG. 8 is a diagram for explaining an operation of dividing a window buffer and a frame buffer according to one embodiment.

[0036] FIG. 9 is a diagram for explaining an operation of identifying a video area and a UI area according to one embodiment.

[0037] FIG. 10 is a diagram illustrating a frame in which the video area occupies a larger proportion than the UI area, according to one embodiment.

[0038] FIG. 11 is a diagram illustrating a frame in which the UI area occupies a larger proportion than the video area, according to one embodiment.

[0039] FIG. 12 is a drawing for explaining an operation of performing different operations depending on a target ratio, according to one embodiment.

[0040] FIG. 13 is a diagram for explaining an operation of obtaining final encoding data when the target ratio is greater than or equal to a threshold ratio, according to one embodiment.

[0041] FIG. 14 is a diagram illustrating an operation for obtaining final encoding data when video rendering is possible, according to one embodiment.

[0042] FIG. 15 is a diagram illustrating an operation for obtaining final encoding data when video rendering is impossible, according to one embodiment.

[0043] FIG. 16 is a diagram illustrating an operation for determining whether video rendering is possible, according to one embodiment.

[0044] FIG. 17 is a diagram for explaining an operation of determining a processing unit that performs a combining operation, according to one embodiment.

[0045] FIG. 18 is a diagram for explaining an operation of processing an alpha value in a combining operation according to one embodiment.

[0046] FIG. 19 is a diagram for explaining an operation of obtaining final encoding data when the target ratio is less than a threshold ratio, according to one embodiment.

[0047] FIG. 20 is a diagram for explaining an operation of obtaining final encoding data when the target ratio is less than a threshold ratio, according to one embodiment.

[0048] FIG. 21 is a diagram for explaining an operation of updating a PNSR table through a screen output from an XR device, according to one embodiment.

[0049] FIG. 22 is a diagram for explaining an operation of updating a PSNR table based on a lookup table transmitted from an external display device, according to one embodiment.

[0050] FIG. 23 is a diagram illustrating an operation of selecting a processing unit to perform a combining operation based on a vertical synchronization time, according to one embodiment.

[0051] FIG. 24 is a drawing for explaining a method of controlling an electronic device according to one embodiment.

[0052] Hereinafter, the present disclosure will be described in detail with reference to the attached drawings.

[0053] The terms used in the embodiments of this disclosure have been selected from widely used, current terms, taking into account the functions of this disclosure. However, these terms may vary depending on the intentions of those skilled in the art, precedents, the emergence of new technologies, etc. Furthermore, in certain cases, terms may be arbitrarily selected by the applicant, and in such cases, their meanings will be described in detail in the description of the relevant disclosure. Therefore, the terms used in this disclosure should not be defined simply as names of terms, but rather based on the meanings of the terms and the overall content of this disclosure.

[0054] In this specification, expressions such as “has,” “can have,” “includes,” or “may include” indicate the presence of a feature (e.g., a number, function, operation, or component such as a part), and do not exclude the presence of additional features.

[0055] The expression "at least one of A or B" should be understood to mean "A" or "B" or "A and B".

[0056] As used herein, the expressions “first,” “second,” “first,” or “second,” etc., may describe various components, regardless of order and / or importance, and are only used to distinguish one component from another, but do not limit the components.

[0057] When it is said that a component (e.g., a first component) is “(operatively or communicatively) coupled with / to” or “connected to” another component (e.g., a second component), it should be understood that the component may be directly coupled to the other component, or may be connected through another component (e.g., a third component).

[0058] Singular expressions include plural expressions unless the context clearly dictates otherwise. In this application, terms such as "comprise" or "consist of" are intended to indicate the presence of a feature, number, step, operation, component, part, or combination thereof described in the specification, but should be understood not to preclude the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0059] In the present disclosure, a "module" or "part" performs at least one function or operation and may be implemented in hardware or software, or a combination of hardware and software. Furthermore, multiple "modules" or multiple "parts" may be integrated into at least one module and implemented as at least one processor, excluding any "modules" or "parts" that need to be implemented as specific hardware.

[0060] In this specification, the term user may refer to a person using an electronic device or a device using an electronic device (e.g., an artificial intelligence electronic device).

[0061] An embodiment of the present disclosure will be described in more detail with reference to the attached drawings below.

[0062] FIG. 1 is a diagram illustrating an operation of combining a video frame and a UI frame according to one embodiment.

[0063] Referring to FIG. 1, an electronic device (100) can receive a video frame (10) and a UI frame (20). The electronic device (100) can combine the video frame (10) and the UI frame (20) to obtain final encoded data (40).

[0064] The video frame (10) may include pixel information expressed in a first color space. The first color space may be YUV. YUV may represent a color space defined by Y (Luminance), U (Chrominance Blue), and V (Chrominance Red). Y may represent brightness or luminance. U may represent a value obtained by subtracting a brightness value from a blue component. V may represent a value obtained by subtracting a brightness value from a red component.

[0065] The UI frame (20) may include pixel information expressed in a second color space. The second color space may be RGB. RGB may represent a color space defined by R (red), G (green), and B (blue).

[0066] An electronic device (100) can obtain final encoded data (40) by combining a video frame (10) expressed in a first color space and a UI frame (20) expressed in a second color space. The final encoded data can include pixel information expressed in the first color space.

[0067] According to one embodiment, pixel information stored in a video frame (10) may be expressed in YUVA. YUBA may represent a color space in which A (Alpha) is additionally defined in YUV. A (Alpha) may represent transparency. A (Alpha) may indicate whether a pixel is transparent or opaque.

[0068] According to one embodiment, pixel information stored in the UI frame (20) may be expressed in RGBA. RGBA may represent a color space in which A (Alpha) is additionally defined in RGB. A (Alpha) may represent transparency. A (Alpha) may indicate whether a pixel is transparent or opaque.

[0069] For example, the video frame (10) may be an XR (Extended Reality) frame. The XR frame may include a VR (Virtual Reality) frame, an AR (Augmented Reality) frame, or an MR (Mixed Reality) frame.

[0070] For example, the video frame (10) may be a general video frame. In an electronic device including a display, such as a smartphone, the video frame (10) and the UI frame (20) may be combined and displayed on a single screen.

[0071] FIG. 2 is a block diagram illustrating an electronic device according to one embodiment.

[0072] Referring to FIG. 2, the electronic device (100) may include at least one of a memory (110) or at least one processor (120).

[0073] An electronic device (100) may include a memory (110) for storing instructions and at least one processor (120) including processing circuitry. The instructions may be executed individually or collectively by the at least one processor (120).

[0074] The electronic device (100) can perform an operation of merging video frames and UI frames. The electronic device (100) can generate a screen including the video frames and the UI frames. The electronic device (100) can generate a single screen by combining a layer corresponding to the video frame and a layer corresponding to the UI frame. The electronic device (100) may be a device that performs an operation of merging video frames and UI frames.

[0075] For example, the electronic device (100) may be a server providing a screen. The electronic device (100) may receive a control command to generate a screen by merging video frames and UI frames. The electronic device (100) may generate final encoded data by merging the video frames and UI frames. The electronic device (100) may transmit the final encoded data to an external device. The external device may be an XR device (200) or a display device (300).

[0076] For example, the electronic device (100) may be a terminal device. The electronic device (100) may be a user terminal device (e.g., a smartphone, a tablet, a wearable device). The electronic device (100) may transmit final encoded data that merges video frames and UI frames to an external device. The external device may be an XR device (200) or a display device (300). In other words, the electronic device (100) may transmit the final encoded data to a device that displays an image based on the final encoded data.

[0077] For example, the electronic device (100) may be an XR device. The electronic device (100) may output a screen by merging video frames and UI frames. If the electronic device (100) is implemented as an XR device, an encoding operation may not be required. If an encoding operation is not required, the encoding data below may be described as video data.

[0078] The description below is based on an embodiment in which the electronic device (100) generates final encoded data.

[0079] At least one processor (120) can obtain a video frame of a first color space and a UI frame of a second color space.

[0080] Video frames and UI frames can be expressed in different color spaces. Video frames can include pixel information in a first color space. UI frames can include pixel information in a second color space.

[0081] When trying to output video frames and UI frames expressed in different color spaces onto a single screen, processing speed and processing time can be an issue.

[0082] At least one processor (120) may receive a control command to output a video frame and a UI frame together. At least one processor (120) may generate encoding data for displaying the video frame and the UI frame on a single screen. At least one processor (120) may generate the encoding data by taking into account the overlapping area of ​​the video frame and the UI frame.

[0083] At least one processor (120) can identify a video area corresponding to a video frame and a UI area corresponding to a UI frame among the entire area (or display area) in which video frames and UI frames are displayed.

[0084] At least one processor (120) can divide a video frame and a UI frame into a plurality of blocks and identify a video area and a UI area based on the divided plurality of blocks. A description related to this is provided in FIGS. 8 and 9.

[0085] At least one processor (120) can identify a non-overlapping video region that does not overlap with a UI region among the video regions. At least one processor (120) can identify an overlapping video region that overlaps with a UI region among the video regions.

[0086] At least one processor (120) can identify an area (overlapping area) where a video area where a video frame is displayed and a UI area where a UI frame is displayed overlap each other. At least one processor (120) can divide the video area into a non-overlapping video area and an overlapping video area.

[0087] Non-overlapping video regions may be described as frames corresponding to the non-overlapping video regions. Overlapping video regions may be described as frames corresponding to the overlapping video regions. Regions may be described as parts or blocks. Descriptions related to overlapping regions are described in FIGS. 14, 15, and 20.

[0088] At least one processor (120) can encode a non-overlapping video region to obtain first encoded data.

[0089] At least one processor (120) can perform a partial encoding operation on an area of ​​the video area that does not overlap with the UI area.

[0090] At least one processor (120) can combine overlapping video regions and UI frames to obtain combined data.

[0091] At least one processor (120) can obtain combined data by performing a combining operation on the overlapping video area and UI frame, which are the remaining portions not reflected in the first encoding data.

[0092] At least one processor (120) can encode the combined data to obtain second encoded data.

[0093] At least one processor (120) can obtain final encoded data by merging the first encoded data and the second encoded data.

[0094] The final encoded data can contain all information about the video frames and UI frames.

[0095] At least one processor (120) can identify a first proportion of the entire area occupied by the video area and a second proportion of the entire area occupied by the UI area. At least one processor (120) can obtain a target proportion by dividing the first proportion by the second proportion and obtain combined data based on the target proportion.

[0096] The first ratio may be described as a video ratio. The second ratio may be described as a UI ratio. At least one processor (120) may calculate a target ratio based on the first ratio and the second ratio.

[0097] The target ratio can represent the ratio of the video area or the UI area to the total area. The target ratio can indicate which area, the video area or the UI area, occupies a larger portion of the total area. Descriptions of the target ratio are provided in Figures 10 and 11.

[0098] At least one processor (120) can identify whether video rendering is possible by converting from a second color space to a first color space if the target ratio is greater than or equal to a threshold ratio. The threshold ratio can be changed according to a user's settings.

[0099] Examples in which the target ratio is greater than or equal to the critical ratio are described in FIGS. 13 to 15.

[0100] Examples in which the target ratio is less than the critical ratio are described in FIGS. 19 and 20.

[0101] If video rendering is enabled, at least one processor (120) can obtain combined data by applying video rendering to the UI frame.

[0102] If video rendering is enabled, at least one processor (120) can perform video rendering on the UI frame to obtain a first rendering UI frame. At least one processor (120) can combine the overlapping video region and the first rendering UI frame to obtain combined data.

[0103] An example of generating final encoded data when video rendering is possible is described in FIG. 14.

[0104] If video rendering is not possible, at least one processor (120) can obtain combined data by applying UI rendering to the UI frame.

[0105] If video rendering is not possible, at least one processor (120) can perform UI rendering on the UI frame to obtain a second rendering UI frame. At least one processor (120) can perform a first color space conversion on the second rendering UI frame to obtain a converted UI frame. At least one processor (120) can combine the overlapping video region and the converted UI frame to obtain combined data.

[0106] An example of generating final encoding data when video rendering is not possible is described in FIG. 15.

[0107] The first color space conversion may include an operation of converting pixel information of the second color space into pixel information of the first color space. The second color space conversion may include an operation of converting pixel information of the first color space into pixel information of the second color space. The first color space conversion and the second color space conversion are described in FIG. 6.

[0108] At least one processor (120) can identify whether a video frame is a 2D image. If the video frame is not a 2D image, the at least one processor (120) can perform 2D modeling on the video frame. A description thereof is provided in FIG. 12.

[0109] According to one embodiment, the first color space may be YUV or YUVA, and the second color space may be RGB or RGBA. Without being limited by the above limitations, the first and second color spaces may represent various color spaces. The color spaces may represent CMYK (Cyan, Magenta, Yellow, Key / Black), HSV (Hue, Saturation, Value), etc.

[0110] The electronic device (100) may include a communication interface (130) connected to an XR (Extended Reality) device (200). At least one processor (120) may transmit final encoded data to the XR device (200) via the communication interface (130).

[0111] The XR device (200) can receive final encoded data from the electronic device (100). The XR device (200) can decode the received final encoded data. The XR device (200) can output (or display) a screen including video frames and UI frames based on the decoded data.

[0112] According to one embodiment, the electronic device (100) can decode the final encoded data on its own. The electronic device (100) can output (or display) the decoded data.

[0113] The electronic device (100) may not simply merge video frames and UI frames and output them as a single layer (or screen). The electronic device (100) may acquire encoding data by considering the overlapping portion of the video frames and UI frames.

[0114] When the color spaces of video and UI frames differ, there are problems such as poor quality or long processing times during the merging process. Separately combining the overlapping portion of the video frame with the UI frame can improve the final display quality and increase the encoding processing speed. The electronic device (100) can independently control the display (140) to display the frame (or image).

[0115] The above description describes generating final encoded data. According to one embodiment, the electronic device (100) may receive video frames and UI frames, and directly combine and display the received video frames and UI frames. The electronic device (100) may perform the same operations related to the video frames and UI frames, but may generate merged data instead of generating encoded data, and display the merged data. For this embodiment, the encoded data may be described as video data or partial data.

[0116] For example, with respect to the operation of FIG. 13, the electronic device (100) may obtain first video data by encoding a non-overlapping video region in operation S1310. The electronic device (100) may obtain second video data by encoding combined data in operation S1350. The electronic device (100) may obtain final video data by merging the first video data and the second video data in operation S1360. The electronic device (100) may output (or display) the final video data through the display (140). The same method (replacing encoded data with video data) related thereto may be applied to various embodiments of the present disclosure.

[0117] FIG. 3 is a block diagram illustrating a specific configuration of the electronic device of FIG. 2, according to one embodiment.

[0118] Referring to FIG. 3, the electronic device (100) may include at least one of a memory (110), at least one processor (120), a communication interface (130), a display (140), an operation interface (150), an input / output interface (160), a speaker (170), a microphone (180), and a camera (190).

[0119] The memory (110) may be implemented as an internal memory such as a ROM (e.g., an electrically erasable programmable read-only memory (EEPROM)) or RAM included in at least one processor (120), or may be implemented as a separate memory from at least one processor (120). The memory (110) may be implemented as a memory embedded in the electronic device (100) or as a memory detachable from the electronic device (100) depending on the purpose of data storage. For example, data for driving the electronic device (100) may be stored in a memory embedded in the electronic device (100), and data for expanding functions of the electronic device (100) may be stored in a memory detachable from the electronic device (100).

[0120] In the case of memory embedded in the electronic device (100), it may be implemented as at least one of volatile memory (e.g., dynamic RAM (DRAM), static RAM (SRAM), or synchronous dynamic RAM (SDRAM)), non-volatile memory (e.g., one time programmable ROM (OTPROM), programmable ROM (PROM), erasable and programmable ROM (EPROM), electrically erasable and programmable ROM (EEPROM), mask ROM, flash ROM, flash memory (e.g., NAND flash or NOR flash), etc.), hard drive, or solid state drive (SSD), and in the case of memory that can be detachably attached to the electronic device (100), it may be implemented in the form of a memory card (e.g., compact flash (CF), secure digital (SD), micro secure digital (Micro-SD), mini secure digital (Mini-SD), extreme digital (xD), multi-media card (MMC), etc.), external memory that can be connected to a USB port (e.g., USB memory), etc.

[0121] The memory (110) can store at least one instruction. Based on the instruction stored in the memory (110), at least one processor (120) can perform various operations.

[0122] At least one processor (120) may be implemented as a digital signal processor (DSP), a microprocessor, or a time controller (TCON) that processes digital signals. However, the present invention is not limited thereto, and may include one or more of a central processing unit (CPU), a micro controller unit (MCU), a micro processing unit (MPU), a controller, an application processor (AP), a graphics-processing unit (GPU), a communication processor (CP), or an advanced reduced instruction set computer (RISC) machines (ARM) processor, or may be defined by the relevant terminology. At least one processor (120) may be implemented as a system on chip (SoC) having a built-in processing algorithm, a large scale integration (LSI), or may be implemented in the form of a field programmable gate array (FPGA). At least one processor (120) may perform various functions by executing computer executable instructions stored in a memory.

[0123] The communication interface (130) is a component that performs communication with various types of external devices according to various types of communication methods. The communication interface (130) may include a wireless communication module or a wired communication module. Each communication module may be implemented in the form of at least one hardware chip.

[0124] A wireless communication module may be a module that communicates wirelessly with an external device. For example, the wireless communication module may include at least one of a Wi-Fi module, a Bluetooth module, an infrared communication module, or other communication modules.

[0125] Wi-Fi and Bluetooth modules can communicate via Wi-Fi and Bluetooth, respectively. When using a Wi-Fi or Bluetooth module, various connection information, such as the service set identifier (SSID) and session key, is first transmitted and received. This information is then used to establish a communication connection before various other information can be transmitted and received.

[0126] Infrared communication modules perform communication based on infrared communication (IrDA, infrared Data Association) technology, which transmits data wirelessly over short distances using infrared light, which is between visible light and millimeter waves.

[0127] In addition to the above-described communication method, other communication modules may include at least one communication chip that performs communication according to various wireless communication standards such as zigbee, 3G (3rd Generation), 3GPP (3rd Generation Partnership Project), LTE (Long Term Evolution), LTE-A (LTE Advanced), 4G (4th Generation), 5G (5th Generation), etc.

[0128] A wired communication module may be a module that communicates with an external device via a wire. For example, the wired communication module may include at least one of a Local Area Network (LAN) module, an Ethernet module, a paired cable, a coaxial cable, a fiber optic cable, or an Ultra Wide-Band (UWB) module.

[0129] According to one embodiment, the communication interface (130) may utilize the same communication module (e.g., a Wi-Fi module) to communicate with an external device such as a remote control device and an external server.

[0130] According to one embodiment, the communication interface (130) may utilize different communication modules to communicate with external devices, such as a remote control device, and external servers. For example, the communication interface (130) may utilize at least one of an Ethernet module or a Wi-Fi module to communicate with an external server, and may also utilize a Bluetooth module to communicate with an external device, such as a remote control device. However, this is merely one embodiment, and the communication interface (130) may utilize at least one of various communication modules when communicating with multiple external devices or external servers.

[0131] The display (140) may be implemented as a variety of displays such as a liquid crystal display (LCD), an organic light emitting diode (OLED) display, a plasma display panel (PDP), etc. The display (140) may also include a driving circuit, a backlight unit, etc., which may be implemented as a form such as an a-si TFT (amorphous silicon thin film transistor), an LTPS (low temperature poly silicon) TFT, an OTFT (organic TFT), etc. The display (140) may be implemented as a touch screen combined with a touch sensor, a flexible display, a three-dimensional display (3D display, three-dimensional display), etc. According to an embodiment of the present disclosure, the display (140) may include not only a display panel that outputs an image, but also a bezel that houses the display panel. In particular, according to an embodiment of the present disclosure, the bezel may include a touch sensor for detecting user interaction.

[0132] The operating interface (150) may be implemented as a device such as a button, a touch pad, a mouse, and a keyboard, or as a touch screen capable of performing the above-described display function and operating input function. The button may be a mechanical button, a touch pad, a wheel, or any other type of button formed in any area of ​​the front, side, or back of the main body of the electronic device (100).

[0133] The input / output interface (160) may be any one of HDMI (High Definition Multimedia Interface), MHL (Mobile High-Definition Link), USB (Universal Serial Bus), DP (Display Port), Thunderbolt, VGA (Video Graphics Array) port, RGB port, D-SUB (D-subminiature), and DVI (Digital Visual Interface). The input / output interface (160) may input / output at least one of audio and video signals. Depending on the implementation example, the input / output interface (160) may include a port that inputs / outputs only audio signals and a port that inputs / outputs only video signals as separate ports, or may be implemented as a single port that inputs / outputs both audio signals and video signals. The electronic device (100) may transmit at least one of the audio and video signals to an external device (e.g., an external display device or an external speaker) through the input / output interface (160). An output port included in the input / output interface (160) can be connected to an external device, and the electronic device (100) can transmit at least one of an audio and video signal to the external device through the output port.

[0134] The input / output interface (160) can be connected to a communication interface. The input / output interface (160) can transmit information received from an external device to the communication interface or transmit information received through the communication interface to the external device.

[0135] The speaker (170) may be a component that outputs various audio data as well as various notification sounds or voice messages.

[0136] The microphone (180) is a component that receives a user's voice or other sounds and converts them into audio data. The microphone (180) can receive the user's voice in an activated state. For example, the microphone (180) can be formed integrally on the upper side, the front side, the side side, etc. of the electronic device (100). The microphone (180) can include various components such as a microphone that collects the user's voice in analog form, an amplifier circuit that amplifies the collected user's voice, an A / D conversion circuit that samples the amplified user's voice and converts it into a digital signal, and a filter circuit that removes noise components from the converted digital signal.

[0137] The camera (190) is a device configured to capture a subject and generate a captured image, and the captured image includes both moving images and still images. The camera (190) can acquire images for at least one external device and can be implemented with a camera, lens, infrared sensor, or the like.

[0138] The camera (190) may include a lens and an image sensor. The type of lens may include a general-purpose lens, a wide-angle lens, a zoom lens, etc., and may be determined according to the type, characteristics, usage environment, etc. of the electronic device (100). The image sensor may include a complementary metal oxide semiconductor (CMOS) and a charge-coupled device (CCD).

[0139] FIG. 4 is a diagram for explaining a 2D video processing structure according to one embodiment.

[0140] Referring to the embodiment (400) of FIG. 4, the electronic device (100) can obtain a two-dimensional video frame (10). The electronic device (100) can output a UI frame (20) together with the video frame (10).

[0141] The electronic device (100) may include at least one of a decoding module (121), a video rendering module (122), a combining module (123), and an encoding module (124). The decoding module (121), the video rendering module (122), the combining module (123), and the encoding module (124) may be included in at least one processor (120).

[0142] The decoding module (121) may be a module that performs a decoding operation on an input frame. The decoding module (121) may decode original data. The decoding module (121) may be a module that performs an operation of converting encoded (or encrypted or coded) information into original information.

[0143] For example, the decoding module (121) may be included in a DPU (Display Processing Unit). The DPU may be included in at least one processor (120).

[0144] An electronic device (100) can input a video frame (10) into a decoding module (121). The electronic device (100) can obtain a decoded video frame through the decoding module (121).

[0145] The electronic device (100) can identify at least one of a non-overlapping video area and an overlapping video area in a video frame (10).

[0146] The electronic device (100) can input a non-overlapping video region into the encoding module (124). The electronic device (100) can obtain first encoding data through the encoding module (124).

[0147] For example, the encoding module (124) may be included in the DPU.

[0148] For example, the encoding module (124) may be included in an NPU (Neural Processing Unit). The NPU may be included in at least one processor (120).

[0149] The electronic device (100) can input the overlapping video area into the combining module (123).

[0150] The electronic device (100) can input a UI frame (20) to the video rendering module (122). The electronic device (100) can obtain a first rendering UI frame through the video rendering module (122).

[0151] The video rendering module (122) can perform a rendering operation to change pixel information of the second color space (RGBA) into pixel information of the first color space (YUV).

[0152] For example, the video rendering module (122) may be included in a GPU (Graphics Processing Unit). The GPU may be included in at least one processor (120).

[0153] The electronic device (100) can input the first rendering UI frame to the combining module (123).

[0154] The combining module (123) may be a module that combines (or composes) at least one frame (or data).

[0155] For example, the coupling module (123) may be included in a GPU.

[0156] For example, the coupling module (123) may be included in the NPU.

[0157] For example, the combination module (123) may be included in each of the GPU and NPU.

[0158] The electronic device (100) can obtain combined data by combining the overlapping video area and the first rendering UI frame received from the video rendering module (122).

[0159] The electronic device (100) can input combined data into the encoding module (124). The electronic device (100) can obtain second encoded data through the encoding module (124).

[0160] The electronic device (100) can obtain final encoded data (40) by merging the first encoded data and the second encoded data.

[0161] FIG. 5 is a diagram for explaining a 3D video processing structure according to one embodiment.

[0162] Referring to the embodiment (500) of FIG. 5, the electronic device (100) can obtain a three-dimensional video frame (10). The three-dimensional video frame (10) may be a 360-degree video frame. For example, the three-dimensional video frame (10) may be an XR frame provided to an XR (Extended Reality) device. The electronic device (100) can output a UI frame (20) together with the video frame (10).

[0163] The decoding module (121), video rendering module (122), combining module (123), and encoding module (124) of Fig. 5 are described in Fig. 4. Duplicate descriptions are omitted.

[0164] The electronic device (100) may include at least one of a decoding module (121), a video rendering module (122), a combining module (123), an encoding module (124), and a 2D modeling module (125).

[0165] The 2D modeling module (125) may be a module that converts a three-dimensional video frame into a two-dimensional video frame. The electronic device (100) may input a decoded video frame (three-dimensional) received from the decoding module (121) into the 2D modeling module (125).

[0166] For example, a 2D modeling module (125) may be included in the GPU.

[0167] For example, a 2D modeling module (125) may be included in the NPU.

[0168] The electronic device (100) can obtain a two-dimensional decoded video frame through the 2D modeling module (125). The electronic device (100) can identify at least one of a non-overlapping video region and an overlapping video region in the two-dimensional decoded video frame.

[0169] The electronic device (100) can transmit the overlapping video region to the combining module (123). The electronic device (100) can transmit the non-overlapping video region to the encoding module (124). Other operations may correspond to the operations of FIG. 4. Duplicate descriptions are omitted.

[0170] FIG. 6 is a diagram for explaining a color space conversion operation according to one embodiment.

[0171] Referring to the embodiment (610) of FIG. 6, the electronic device (100) may perform a first color space conversion to convert pixel information from a second color space (RGB) to a first color space (YUV). The first color space conversion may include an operation of converting pixel information of the second color space (RGB) to pixel information of the first color space (YUV).

[0172] The electronic device (100) can perform a first color space conversion using the first function information (611). The electronic device (100) can convert the second color space (RGB) into the first color space (YUV) using the first function information (611).

[0173] The electronic device (100) can perform video rendering. The video rendering can include an operation of converting a second color space (RGB) into a first color space (YUV) using first function information (611).

[0174] The operation of converting the second color space (RGB) to the first color space (YUV) using the first function information (611) can be described as video rendering.

[0175] Referring to the embodiment (620) of FIG. 6, the electronic device (100) may perform a second color space conversion to convert pixel information from a first color space (YUV) to a second color space (RGB). The second color space conversion may include an operation of converting pixel information of the first color space (YUV) to pixel information of the second color space (RGB).

[0176] The electronic device (100) can perform a second color space conversion using the second function information (621). The electronic device (100) can convert the first color space (YUV) to the second color space (RGB) using the second function information (621).

[0177] Referring to the embodiment (630) of FIG. 6, the electronic device (100) can perform the first color space conversion or the second color space conversion based on the color space lookup table (631). The color space lookup table (631) can include preset mapping information. The mapping information can include information mapping RGB values ​​corresponding to specific YUV values.

[0178] For example, pixel information of the first color space (YUV) representing black may be (0,0,0), and pixel information of the second color space (RGB) representing black may be (16, 128, 128). The color space lookup table (631) may include information in which YUV (16, 128, 128) and RGB (0,0,0) are mapped.

[0179] For example, if the pixel value in the second color space (RGB) is 0≤R,G,B≤255, the pixel value in the first color space (YUV) may be 16≤Y≤235, 16≤U,V≤240.

[0180] For example, dividing the window buffer into blocks can enable fast parallel processing using the NPU.

[0181] For example, the color space lookup table (631) may include information on which YUV and RGB are mapped.

[0182] For example, the color space lookup table (631) may include information on which YUV and RGBA are mapped.

[0183] For example, the color space lookup table (631) may include information to which YUVA and RGB are mapped.

[0184] For example, the color space lookup table (631) may include information on which YUVA and RGBA are mapped.

[0185] FIG. 7 is a diagram illustrating an operation of identifying a video area and a UI area to obtain a target ratio according to one embodiment.

[0186] Referring to FIG. 7, the electronic device (100) can obtain a video frame (10) and a UI frame (20). The electronic device (100) can receive a control command to output the video frame (10) and the UI frame (20) together.

[0187] For example, the video frame (10) may be an XR (Extended Reality) frame. The XR frame may include a VR (Virtual Reality) frame, an AR (Augmented Reality) frame, or an MR (Mixed Reality) frame.

[0188] The electronic device (100) can identify whether a device outputting a frame (e.g., an XR device) is operating in XR mode (S710). The XR mode may include a VR mode, an AR mode, or an MR mode. If the XR mode is operating (S710-Y), the electronic device (100) can determine whether a video layer is identified.

[0189] The electronic device (100) may perform video processing to display a video frame (10). The video processing may include an operation of displaying a video layer. The electronic device (100) may determine whether a video layer is identified. The electronic device (100) may generate a video layer based on the video frame (10). The electronic device (100) may determine whether a video layer is generated (or displayed).

[0190] In one embodiment, steps S710 and S715 may be omitted.

[0191] In one embodiment, step S710 may be replaced with a step for determining whether the device is operating in external display mode. External display mode may indicate a mode in which frames (or sub-frames) are output to an additional display device in addition to the XR device outputting the frames.

[0192] Once the video layer is identified (S715-Y), the electronic device (100) can divide the buffer into multiple blocks. The electronic device (100) can divide the window buffer or frame buffer into multiple blocks. The multiple blocks may be a preset number of blocks.

[0193] A window buffer can be a memory space that stores pixel information for a specific area of ​​the screen.

[0194] A frame buffer can be a memory space that stores pixel information for the entire area of ​​the screen.

[0195] The electronic device (100) can encode blocks stored in a window buffer and a frame buffer. The electronic device (100) can perform an encoding operation through an encoding module (124).

[0196] The electronic device (100) can identify a video area and a UI area based on a plurality of segmented blocks (S725). The electronic device (100) can identify a video area where a video frame (10) is displayed on the entire screen where the frames are displayed. The electronic device (100) can identify a UI area where a UI frame (20) is displayed on the entire screen.

[0197] The electronic device (100) can identify a first ratio (or video ratio) occupied by the video area among the entire area representing the entire screen (S730).

[0198] The electronic device (100) can identify a second ratio (or UI ratio) occupied by the UI area among the entire area representing the entire screen (S735).

[0199] The electronic device (100) can obtain a target ratio based on the first ratio and the second ratio (S740). The target ratio may be a value obtained by dividing the first ratio by the second ratio. The target ratio may indicate the relationship between the proportion of the entire area occupied by the video area or the UI area. The target ratio may indicate which area, the video area or the UI area, occupies a larger portion of the entire area.

[0200] Once the target ratio is obtained, the electronic device (100) can perform the operations disclosed in FIG. 12.

[0201] In one embodiment, the electronic device (100) may not calculate the target ratio. The electronic device (100) may compare the first ratio and the second ratio. The electronic device (100) may determine which value is greater, the first ratio or the second ratio.

[0202] FIG. 8 is a diagram for explaining an operation of dividing a window buffer and a frame buffer according to one embodiment.

[0203] Referring to the embodiment (810) of FIG. 8, the electronic device (100) can divide the window buffer into a preset number of buffers. The window buffer can represent a buffer for displaying at least one of a video area or a UI area.

[0204] Referring to the embodiment (820) of FIG. 8, the electronic device (100) can divide the frame buffer into a preset number of buffers. The frame buffer can represent a buffer for displaying the entire area.

[0205] In the embodiments (810, 820) of FIG. 8, the sizes of the window buffer and the frame buffer may be the same. When the video frame (10) is displayed on the entire screen, the sizes of the window buffer and the frame buffer may be the same.

[0206] The electronic device (100) can divide a plurality of blocks using a window buffer or a frame buffer. The electronic device (100) can identify a video area and a UI area using the divided plurality of blocks.

[0207] FIG. 9 is a diagram for explaining an operation of identifying a video area and a UI area according to one embodiment.

[0208] Referring to the embodiment (910) of FIG. 9, the video area (911) on the screen (901) may be larger than the UI area (912). The size at which the video frame (10) is displayed may be larger than the size at which the UI frame (20) is displayed. The video area (911) may be the entire 16-square area out of the total 16-square area. The UI area (912) may be an area including the middle 4 squares.

[0209] Referring to the embodiment (920) of FIG. 9, the UI area (922) on the screen (902) may be larger than the video area (921). The size at which the UI frame (20) is displayed may be larger than the size at which the video frame (10) is displayed. The video area (921) may be an area of ​​8 spaces below (all of the 3rd and 4th rows). The UI area (922) may be an area of ​​12 spaces (all of the 1st, 2nd, and 3rd rows).

[0210] FIG. 10 is a diagram illustrating a frame in which the video area occupies a larger proportion than the UI area, according to one embodiment.

[0211] Fig. 10 may correspond to the embodiment (910) of Fig. 9. The electronic device (100) may identify a video area (1011) and a UI area (1012) in the entire screen (1001).

[0212] Referring to table (1020) of FIG. 10, the electronic device (100) can calculate a target ratio for the screen (1001).

[0213] The electronic device (100) can identify the number of blocks in the entire area (e.g., 16). The electronic device (100) can identify the number of blocks occupied by the video area (1011) among the number of blocks in the entire area (e.g., 16). The electronic device (100) can identify the number of blocks occupied by the UI area (1012) among the number of blocks in the entire area (e.g., 4).

[0214] The electronic device (100) can identify a first ratio (e.g., 1) of the number of blocks occupied by the video area (1011) among the total number of blocks. The first ratio can be described as video ratio or r_YUV.

[0215] The electronic device (100) can identify a second ratio (e.g., 0.25) of the number of blocks occupied by the UI area (1012) among the total number of blocks. The second ratio can be described as UI ratio or r_RGB.

[0216] The electronic device (100) can obtain a target ratio based on the first ratio and the second ratio. The target ratio may be a value (e.g., 4) obtained by dividing the first ratio by the second ratio. The target ratio may be expressed as r_total.

[0217] If the target ratio is greater than 1, the electronic device (100) can identify that the video area (1011) is larger than the UI area (1012).

[0218] If the target ratio is 1, the electronic device (100) can identify that the video area (1011) and the UI area (1012) are of the same size.

[0219] If the target ratio is less than 1, the electronic device (100) can identify that the video area (1011) is smaller than the UI area (1012).

[0220] Since the target ratio is greater than 1, the electronic device (100) can identify that the video area (1011) is larger than the UI area (1012).

[0221] As the value of the target ratio becomes greater than 1, the electronic device (100) can identify that the video area (1011) is relatively larger.

[0222] The closer the target ratio value is to 0, the more the electronic device (100) can identify that the UI area (1012) is relatively larger.

[0223] FIG. 11 is a diagram illustrating a frame in which the UI area occupies a larger proportion than the video area, according to one embodiment.

[0224] Fig. 11 may correspond to the embodiment (920) of Fig. 9. The electronic device (100) may identify a video area (1111) and a UI area (1112) in the entire screen (1001).

[0225] Referring to table (1120) of FIG. 11, the electronic device (100) can calculate a target ratio for the screen (1001).

[0226] The electronic device (100) can identify the number of blocks in the entire area (e.g., 16). The electronic device (100) can identify the number of blocks occupied by the video area (1111) among the number of blocks in the entire area (e.g., 8). The electronic device (100) can identify the number of blocks occupied by the UI area (1112) among the number of blocks in the entire area (e.g., 12).

[0227] The electronic device (100) can identify a first ratio (e.g., 0.5) of the number of blocks occupied by the video area (1111) to the total number of blocks. The first ratio can be described as video ratio or r_YUV.

[0228] The electronic device (100) can identify a second ratio (e.g., 0.75) of the number of blocks occupied by the UI area (1112) among the total number of blocks. The second ratio can be described as UI ratio or r_RGB.

[0229] The electronic device (100) can obtain a target ratio based on the first ratio and the second ratio. The target ratio may be a value (e.g., 0.67) obtained by dividing the first ratio by the second ratio. The target ratio may be expressed as r_total.

[0230] Descriptions related to the target ratio are described in FIG. 10. Duplicate descriptions are omitted. Since the target ratio is less than 1, the electronic device (100) can identify that the UI area (1012) is larger than the video area (1011).

[0231] FIG. 12 is a drawing for explaining an operation of performing different operations depending on a target ratio, according to one embodiment.

[0232] When the target ratio is obtained according to the embodiment of FIG. 7, the electronic device (100) can identify whether the target ratio is greater than or equal to a threshold ratio (S1205). The threshold ratio may be a preset ratio. The threshold ratio may be changed according to the user's settings.

[0233] The threshold ratio may be a criterion for the electronic device (100) to determine whether the video area is the main area in the entire screen.

[0234] If the target ratio is greater than or equal to the threshold ratio, the electronic device (100) can combine the non-overlapping video region and UI frame based on the first color space (YUV). Specific operations related thereto are described in FIGS. 13 to 15.

[0235] If the target ratio is greater than or equal to the threshold ratio (S1205-Y), the electronic device (100) can identify whether the video frame (10) is a 2D image (S1210). The electronic device (100) can identify whether the video frame (10) is a 2D image based on the dimension of pixel information included in the video frame (10).

[0236] If the video frame (10) is a 2D image (S1210-Y), the electronic device (100) can perform the operations disclosed in FIG. 13.

[0237] If the video frame (10) is not a 2D image (S1210-N), the electronic device (100) can perform 2D modeling (S1215). The 2D modeling can be performed by the 2D modeling module (125) of FIG. 5. The 2D modeling may not include an operation of converting from a first color space (YUV) to a second color space (RGB). The 2D modeling may include an operation of converting a 3D frame or 3D image into a 2D frame or 2D image.

[0238] When performing 2D modeling, the electronic device (100) can utilize an NPU or GPU. The electronic device (100) can perform 2D modeling by selecting a processing unit with the fastest processing speed. This is to shorten the processing time required to perform 2D modeling.

[0239] The electronic device (100) can obtain at least one of the frequency of the GPU, the workload of the GPU, the frequency of the NPU, or the workload of the NPU. The electronic device (100) can select a processing unit to perform a 2D modeling operation based on at least one of the frequency of the GPU, the workload of the GPU, the frequency of the NPU, or the workload of the NPU.

[0240] If the target ratio is less than the threshold ratio, the electronic device (100) may combine the non-overlapping video region and UI frame based on the second color space (RGB). Specific operations related to this are described in FIGS. 19 and 20.

[0241] If the target ratio is not greater than the threshold ratio (S1205-N), the electronic device (100) can identify whether the video frame (10) is a 2D image (S1210). The electronic device (100) can identify whether the video frame (10) is a 2D image based on the dimension of pixel information included in the video frame (10).

[0242] If the video frame (10) is a 2D image (S1220-Y), the electronic device (100) can perform the operations disclosed in FIG. 19.

[0243] If the video frame (10) is not a 2D image (S1220-N), the electronic device (100) can perform 2D modeling (S1225). The electronic device (100) can perform 2D modeling using a GPU.

[0244] The electronic device (100) can perform a second color space conversion on the video frame (10) to obtain a converted video frame (S1230). The electronic device (100) can obtain a video frame converted from a first color space (YUV) to a second color space (RGB). The electronic device (100) can perform the second color space conversion using an NPU.

[0245] In step S1215, a processing unit to perform 2D modeling is selected based on at least one of the frequency of the GPU, the workload of the GPU, the frequency of the NPU, or the workload of the NPU.

[0246] In step S1225, 2D modeling can be performed on the GPU. If the UI area occupies a larger proportion than the video area, the importance of processing speed may be relatively low. The electronic device (100) can permanently utilize the GPU to perform 2D modeling for high (or stable) quality. The electronic device (100) can perform 2D modeling using the GPU and perform a second color space conversion using the NPU to obtain a converted video frame.

[0247] The electronic device (100) can perform the operations disclosed in FIG. 19 based on the converted video frame.

[0248] An electronic device (100) can convert a three-dimensional video frame (10) into a two-dimensional video frame. Based on the converted video frame, the electronic device (100) can perform the operations disclosed in FIG. 13.

[0249] FIG. 13 is a diagram illustrating an operation for obtaining final encoded data when the target ratio is greater than or equal to a threshold ratio, according to one embodiment. The operations of FIG. 13 may be performed after step S1210 or S1215 of FIG. 12 is performed.

[0250] Referring to FIG. 13, the electronic device (100) can identify a non-overlapping video area that does not overlap with the UI area among the video areas, and can identify an overlapping video area that overlaps with the UI area among the video areas (S1305).

[0251] The electronic device (100) can obtain first encoding data by encoding a non-overlapping video region (S1310). The electronic device (100) can store the non-overlapping video region using a window buffer. The electronic device (100) can store the first encoding data in a frame buffer. The electronic device (100) can obtain the first encoding data using the encoding module (124) of FIG. 4. The first encoding data can include pixel information of a first color space.

[0252] The electronic device (100) can identify whether video rendering is possible (S1315). The electronic device (100) can identify whether the video rendering function can be performed.

[0253] Video rendering may include an operation of converting a frame of a second color space (RGB) to a frame of a first color space (YUV). Video rendering may include an operation of converting a second color space (RGB) to a first color space (YUV) using first function information (611). Video rendering may be described as YUV / YUVA rendering.

[0254] The electronic device (100) can store a list of functions that the electronic device (100) can perform in the memory (110). The electronic device (100) can identify whether the stored list of functions includes a video rendering function.

[0255] If the function list includes a rendering function, the electronic device (100) can be identified as capable of performing video rendering.

[0256] If it is determined that video rendering is possible (S1315-Y), the electronic device (100) may perform video rendering on the UI frame (20) to obtain a first rendering UI frame (S1320). The first rendering UI frame may include pixel information of a first color space. The first rendering UI frame may include pixel information of YUV / YUVA.

[0257] The electronic device (100) can obtain combined data by composing the overlapping video area and the first rendering UI frame (S1325). The electronic device (100) can perform a combining operation based on the first color space. The electronic device (100) can obtain combined data of the first color space by composing the first rendering UI frame of the first color space and the overlapping video area of ​​the first color space.

[0258] For example, the first color space may be YUV. The electronic device (100) may perform a YUV combining operation.

[0259] For example, the electronic device (100) can utilize the NPU in performing the combining operation of step S1325.

[0260] For example, the electronic device (100) can utilize a GPU to perform the combining operation of step S1325.

[0261] If video rendering is not identified as being performable (S1315-N), the electronic device (100) may perform UI rendering on the UI frame (20) to obtain a second rendering UI frame (S1330). The electronic device (100) may obtain the second rendering UI frame of the second color space.

[0262] UI rendering may indicate applying a preset rendering function to a UI frame. UI rendering may include an operation of applying a preset rendering function based on a second color space. The preset rendering function may include at least one of a function for changing the resolution, a function for changing the color space, a function for changing the position, and a function for changing style information (such as font, color, size, etc.).

[0263] For example, UI rendering may include preprocessing operations required for the combined operation.

[0264] For example, UI rendering may be RGB / RGBA rendering. The electronic device (100) may perform UI rendering on the UI frame (20) to obtain a second rendering UI frame of RGB / RGBA.

[0265] The electronic device (100) can perform a first color space conversion on the second rendering UI frame to obtain a conversion UI frame (S1335). By performing the first color space conversion, the electronic device (100) can convert the second rendering UI frame of the second color space into a conversion UI frame of the first color space.

[0266] For example, the first color space may be YUV / YUVA, and the second color space may be RGB / RGBA.

[0267] For example, the electronic device (100) can perform a first color space conversion using a color space lookup table (631).

[0268] For example, the electronic device (100) can perform a first color space conversion using the first function information (611).

[0269] For example, the electronic device (100) can perform the first color space conversion using the NPU.

[0270] For example, the electronic device (100) may perform a first color space conversion using the first function information (611) via the NPU. The electronic device (100) may generate a color space lookup table (631) as a result of performing the first color space conversion. The color space lookup table (631) may be updated according to a result of performing a new first color space conversion. The color space lookup table (631) may be used for the first color space conversion.

[0271] The electronic device (100) can obtain combined data by composing the overlapping video area and the transformation UI frame (S1340). The electronic device (100) can perform a combining operation based on the first color space. The electronic device (100) can obtain combined data of the first color space by composing the transformation UI frame of the first color space and the overlapping video area of ​​the first color space.

[0272] For example, the first color space may be YUV. The electronic device (100) may perform a YUV combining operation.

[0273] For example, the electronic device (100) can utilize the NPU in performing the combining operation of step S1340.

[0274] For example, the electronic device (100) can utilize a GPU to perform the combining operation of step S1340.

[0275] The electronic device (100) can obtain second encoded data by encoding the combined data obtained in step S1325 or step S1340 (S1350). The second encoded data can include pixel information of the first color space.

[0276] The electronic device (100) can obtain final encoded data by merging the first encoded data and the second encoded data (S1360). The final encoded data can include pixel information of the first color space.

[0277] FIG. 14 is a diagram illustrating an operation for obtaining final encoding data when video rendering is possible, according to one embodiment.

[0278] FIG. 14 may illustrate an embodiment of obtaining final encoding data when the video rendering function determined in step S1315 of FIG. 13 is performable.

[0279] The electronic device (100) can obtain a video frame (1410) and a UI frame (1420). The electronic device (100) can receive a control command to output the video frame (1410) and the UI frame (1420) together.

[0280] The electronic device (100) can identify a non-overlapping video area (1411) that does not overlap with a UI frame (1420) for the video frame (1410).

[0281] The electronic device (100) can identify an overlapping video area (1412) that overlaps a UI frame (1420) with respect to a video frame (1410).

[0282] The electronic device (100) can obtain first encoded data (1413) by encoding a non-overlapping video area (1411).

[0283] The electronic device (100) may obtain a first rendering UI frame (1421) by performing video rendering on the UI frame (1420). The video rendering may include a first color space conversion operation that converts pixel information of a second color space into pixel information of a first color space. The video rendering may include an operation that modifies the UI frame (1420) for a combining operation.

[0284] The electronic device (100) can obtain combined data (1430) by combining the overlapping video region (1412) and the first rendering UI frame (1421). The electronic device (100) can perform a combining operation based on the first color space. As an example, the combining operation can be described as YUV composition.

[0285] The electronic device (100) can encode the combined data (1430) to obtain second encoded data (1431).

[0286] The electronic device (100) can obtain final encoded data (1440) by merging the first encoded data (1413) and the second encoded data (1431).

[0287] For example, the video frame (1410), the non-overlapping video region (1411), the overlapping video region (1412), the first encoding data (1413), the first rendering UI frame (1421), the combined data (1430), the second encoding data (1431), and the final encoding data (1440) may correspond to a first color space. Corresponding to a first color space may indicate defining pixel information based on the first color space.

[0288] For example, the UI frame (1420) may correspond to a second color space. Corresponding to a second color space may indicate defining pixel information based on the second color space.

[0289] For example, the first color space may be YUV and the second color space may be RGB.

[0290] For example, the first color space may be YUVA and the second color space may be RGBA.

[0291] For example, the first color space may be YUV and the second color space may be RGBA.

[0292] For example, the first color space may be YUVA and the second color space may be RGB.

[0293] FIG. 15 is a diagram illustrating an operation for obtaining final encoding data when video rendering is impossible, according to one embodiment.

[0294] FIG. 15 may illustrate an embodiment of obtaining final encoding data when the video rendering function determined in step S1315 of FIG. 13 is not executable.

[0295] The electronic device (100) can obtain a video frame (1510) and a UI frame (1520). The electronic device (100) can receive a control command to output the video frame (1510) and the UI frame (1520) together.

[0296] The electronic device (100) can identify a non-overlapping video area (1511) that does not overlap with a UI frame (1520) for the video frame (1510).

[0297] The electronic device (100) can identify an overlapping video area (1512) that overlaps a UI frame (1520) with respect to a video frame (1510).

[0298] The electronic device (100) can encode a non-overlapping video area (1511) to obtain first encoded data (1513).

[0299] The electronic device (100) can obtain a second rendering UI frame (1521) by performing UI rendering on the UI frame (1520). The UI rendering may include an operation of changing the UI frame (1520) for a combining operation.

[0300] The electronic device (100) can obtain a converted UI frame (1522) by performing a first color space conversion on the second rendering UI frame (1521). The electronic device (100) can convert pixel information of the second color space into pixel information of the first color space based on the color space lookup table (631).

[0301] The electronic device (100) can obtain combined data (1530) by combining the overlapping video region (1512) and the transformed UI frame (1522). The electronic device (100) can perform a combining operation based on a first color space. As an example, the combining operation can be described as YUV composition.

[0302] The electronic device (100) can encode the combined data (1530) to obtain second encoded data (1531).

[0303] The electronic device (100) can obtain final encoded data (1540) by merging the first encoded data (1513) and the second encoded data (1531).

[0304] For example, a video frame (1510), a non-overlapping video region (1511), an overlapping video region (1512), first encoded data (1513), a transformed UI frame (1522), combined data (1530), second encoded data (1531), and final encoded data (1540) may correspond to a first color space. Corresponding to a first color space may indicate defining pixel information based on the first color space.

[0305] For example, the UI frame (1520) and the second rendering UI frame (1521) may correspond to a second color space. Corresponding to a second color space may indicate defining pixel information based on the second color space.

[0306] For example, the first color space may be YUV and the second color space may be RGB.

[0307] For example, the first color space may be YUV and the second color space may be RGBA.

[0308] For example, the first color space may be YUVA and the second color space may be RGBA.

[0309] For example, the first color space may be YUVA and the second color space may be RGB.

[0310] FIG. 16 is a diagram illustrating an operation for determining whether video rendering is possible, according to one embodiment.

[0311] Referring to FIG. 16, the electronic device (100) can determine whether the electronic device (100) can perform video rendering based on code information (1610).

[0312] The code information (1610) may include code for identifying whether a library used by an application for rendering images includes video rendering capabilities.

[0313] FIG. 17 is a diagram for explaining an operation of determining a processing unit that performs a combining operation, according to one embodiment.

[0314] FIG. 17 may include an operation of determining a processing unit that performs the combining operation described in step S1325 of FIG. 13, step S1340 of FIG. 13, or step S1925 of FIG. 19.

[0315] The electronic device (100) can select one of the processing units, either an NPU or a GPU. The electronic device (100) can perform a combining operation (S1325, S1340, S1925) using the selected processing unit.

[0316] The electronic device (100) can calculate (or obtain) a first time (T_NPU) expected (or estimated) to perform a combination operation using the NPU (S1705).

[0317] The electronic device (100) can calculate (or obtain) a second time (T_GPU) expected (or estimated) to perform a combination operation using a GPU (S1710).

[0318] The electronic device (100) can operate in a normal mode or a power-saving mode. The normal mode may be described as a performance mode. The power-saving mode may be described as a low-power mode. The modes may be replaced by states.

[0319] The electronic device (100) can identify whether the electronic device (100) is operating in power saving mode (S1715).

[0320] If not operating in power saving mode (S1715-N), the electronic device (100) can identify whether the first time (T_NPU) is less than the second time (T_GPU) (S1720).

[0321] If the first time (T_NPU) is less than the second time (T_GPU) (S1720-Y), the electronic device (100) can perform a combining operation using the NPU (S1730).

[0322] If the first time (T_NPU) is not less than the second time (T_GPU) (S1720-Y), the electronic device (100) can perform a combining operation using the GPU (S1740).

[0323] When operating in power saving mode (S1715-Y), the electronic device (100) can identify whether the first time (T_NPU) is less than the second time (T_GPU) (S1730).

[0324] When the electronic device (100) operates in normal mode, processing speed may be important to the user. The electronic device (100) may compare the expected times for the binding operation (first time, second time) and select a processing unit capable of faster processing.

[0325] If the first time (T_NPU) is less than the second time (T_GPU) (S1730-Y), the electronic device (100) can perform a combining operation using the GPU (S1740).

[0326] If the first time (T_NPU) is not less than the second time (T_GPU) (S1730-Y), the electronic device (100) can perform a combining operation using the NPU (S1730).

[0327] When the electronic device (100) operates in power saving mode, the quality of the combined data may be more important to the user than the processing speed. Even if the first time (T_NPU) is less than the second time (T_GPU), the electronic device (100) may select the GPU instead of the NPU.

[0328] According to one embodiment, when the electronic device (100) operates in power saving mode, the electronic device (100) may not perform step S1730. When the electronic device (100) operates in power saving mode, the electronic device (100) may obtain combined data using the NPU (S1730). In power saving mode, the supplied power may be low. Therefore, a processing unit that minimizes power consumption may be selected to perform the combining operation. Using the NPU rather than the GPU may result in lower power consumption.

[0329] In one embodiment, if the NPU consumes more power than the GPU, the electronic device (100) may perform the combining operation using the GPU in power saving mode.

[0330] FIG. 18 is a diagram for explaining an operation of processing an alpha value in a combining operation according to one embodiment.

[0331] Referring to FIG. 18, the electronic device (100) can perform a coupling operation using mathematical expression (1810).

[0332] The combined data (1430) may include pixel information of the first color space (YUV). The combined data (1430) may not include an alpha value. The alpha value may be a value indicating transparency. Information regarding transparency may be reflected in the first color space (YUV). The electronic device (100) may reflect the alpha value in the pixel information using the mathematical equation (1810).

[0333] According to one embodiment, the electronic device (100) can generate the combined data (1430) of FIG. 14 using the mathematical expression (1810).

[0334] α_0 may be the alpha value of the combined data (1430).

[0335] α_a may be the alpha value of the overlapping overlapping video area (1412).

[0336] α_b may be the alpha value of the combined first rendering UI frame (1421).

[0337] C_0 may be the color composition (or pixel information) of the combined data (1430).

[0338] α_a may be a color composition (or pixel information) of the overlapping overlapping video area (1412).

[0339] α_b may be a color configuration (or pixel information) of the combined first rendering UI frame (1421).

[0340] The above description can be applied to the combined data generation operation of FIGS. 15 and 20. Duplicate description is omitted.

[0341] FIG. 19 is a diagram for explaining an operation of obtaining final encoding data when the target ratio is less than a threshold ratio, according to one embodiment.

[0342] FIG. 19 is a diagram illustrating an operation for obtaining final encoding data when the target ratio is less than a threshold ratio. The operations of FIG. 19 may be performed after step S1220 or S1230 of FIG. 12 is performed.

[0343] Referring to FIG. 19, the electronic device (100) can identify a non-overlapping video area that does not overlap with the UI area among the video areas, and can identify an overlapping video area that overlaps with the UI area among the video areas (S1905).

[0344] The electronic device (100) can obtain third encoding data by encoding the non-overlapping video region (S1910). The electronic device (100) can store the non-overlapping video region using a window buffer. The electronic device (100) can store the third encoding data in a frame buffer. The electronic device (100) can obtain the third encoding data using the encoding module (124) of FIG. 4. The third encoding data can include pixel information of the first color space.

[0345] The electronic device (100) may perform a second color space conversion on the overlapping video region to obtain a converted video region (S1915). The second color space conversion may be an operation of converting a first color space into a second color space. The second color space conversion may include an operation of converting pixel information of the first color space into pixel information of the second color space. The electronic device (100) may perform the second color space conversion using a color space lookup table (631).

[0346] The electronic device (100) can perform UI rendering on the UI frame (20) to obtain a third rendering UI frame (S1920). The UI rendering is described in FIG. 13. Duplicate description is omitted.

[0347] The electronic device (100) can obtain combined data by combining the converted video area and the third rendering UI frame (S1925).

[0348] The electronic device (100) can perform a combining operation based on a second color space. The electronic device (100) can obtain combined data of the second color space by combining a third rendering UI frame of the second color space and a converted video area of ​​the second color space.

[0349] For example, the first color space may be RGB or RGBA. The electronic device (100) may perform RGB or RGBA combining operations.

[0350] For example, the electronic device (100) can utilize the NPU in performing the combining operation of step S1925.

[0351] For example, the electronic device (100) can utilize a GPU to perform the combining operation of step S1925.

[0352] The electronic device (100) can perform a first color space conversion on the combined data to obtain conversion data (S1930). The first color space conversion may include an operation of converting pixel information of a second color space into pixel information of a first color space.

[0353] For example, the electronic device (100) can perform the first color space conversion using the NPU.

[0354] For example, the electronic device (100) can perform a first color space conversion based on a color space lookup table (631).

[0355] For example, the electronic device (100) can perform a first color space conversion using the first function information (611).

[0356] For example, the electronic device (100) may perform a first color space conversion using the first function information (611) via the NPU. The electronic device (100) may generate a color space lookup table (631) as a result of performing the first color space conversion. The color space lookup table (631) may be updated according to a result of performing a new first color space conversion. The color space lookup table (631) may be used for the first color space conversion.

[0357] The electronic device (100) can encode the conversion data to obtain fourth encoding data (S1935).

[0358] The electronic device (100) can obtain final encoding data by merging the third encoding data and the fourth encoding data (S1940).

[0359] FIG. 20 is a diagram for explaining an operation of obtaining final encoding data when the target ratio is less than a threshold ratio, according to one embodiment.

[0360] Fig. 20 can represent an example of obtaining final encoding data by the operation of Fig. 19.

[0361] The electronic device (100) can obtain a video frame (2010) and a UI frame (2020). The electronic device (100) can receive a control command to output the video frame (2010) and the UI frame (2020) together.

[0362] The electronic device (100) can identify a non-overlapping video area (2011) that does not overlap with a UI frame (2020) for the video frame (2010).

[0363] The electronic device (100) can identify an overlapping video area (2012) that overlaps a UI frame (2020) with respect to a video frame (2010).

[0364] The electronic device (100) may perform a second color space conversion on the overlapping video area (2012) to obtain a converted video area (2012-1). The second color space conversion may include an operation of converting pixel information of a first color space into pixel information of a second color space. The electronic device (100) may convert pixel information of the first color space into pixel information of the second color space based on a color space lookup table (631).

[0365] The electronic device (100) can encode the non-overlapping video region (2011) to obtain third encoded data (2013).

[0366] The electronic device (100) can obtain a third rendering UI frame (2021) by performing UI rendering on the UI frame (2020). The UI rendering may include an operation of changing the UI frame (2020) for a combining operation.

[0367] The electronic device (100) can obtain combined data (2030) by combining the converted video region (2012-1) and the third rendering UI frame (2021). The electronic device (100) can perform a combining operation based on the second color space. As an example, the combining operation can be described as RGB composition.

[0368] The electronic device (100) can obtain conversion data (2031) by performing a first color space conversion based on the combined data (2030). The first color space conversion may include an operation of converting pixel information of a second color space into pixel information of a first color space. The electronic device (100) can perform the first color space conversion based on a color space lookup table (631).

[0369] The electronic device (100) can encode the conversion data (2031) to obtain fourth encoded data (2032).

[0370] The electronic device (100) can obtain final encoding data (2040) by merging the third encoding data (2013) and the fourth encoding data (2032).

[0371] For example, the video frame (2010), the non-overlapping video region (2011), the overlapping video region (2012), the third encoding data (2013), the conversion data (2031), the fourth encoding data (2032), and the final encoding data (2040) may correspond to a first color space. Corresponding to a first color space may indicate defining pixel information based on the first color space.

[0372] For example, the UI frame (2020), the third rendering UI frame (2021), and the combined data (2030) may correspond to a second color space. Corresponding to a second color space may indicate that pixel information is defined based on the second color space.

[0373] For example, the first color space may be YUV and the second color space may be RGB.

[0374] For example, the first color space may be YUVA and the second color space may be RGBA.

[0375] For example, the first color space may be YUV and the second color space may be RGBA.

[0376] For example, the first color space may be YUVA and the second color space may be RGB.

[0377] FIG. 21 is a diagram for explaining an operation of updating a PNSR table through a screen output from an XR device, according to one embodiment.

[0378] FIG. 21 may be an operation performed after step S1925 of FIG. 19. The operations disclosed in FIG. 19 may also be applied to FIG. 21. Duplicate explanations are omitted.

[0379] The electronic device (100) may be connected to an XR device (200). The XR device (200) may be a device that outputs final encoded data provided by the electronic device (100).

[0380] The electronic device (100) can store a PSNR (Peak-Signal to Noise Ratio) table. The electronic device (100) can perform a first color space conversion on the combined data based on the PSNR table to obtain conversion data (S2130).

[0381] A PSNR table can contain information indicating video (or image) quality. The PSNR table can contain information indicating the quality difference between the original data and the compressed (or encoded) data. The PSNR table can contain PSNR values ​​measured in dB. A higher PSNR value can indicate better quality.

[0382] The electronic device (100) can encode the conversion data to obtain fourth encoding data (S2135).

[0383] The electronic device (100) can obtain final encoding data by merging the third encoding data and the fourth encoding data (S2140).

[0384] The electronic device (100) can transmit the final encoded data to the XR device (200) (S2145).

[0385] The XR device (200) can receive final encoding data from the electronic device (100). The XR device (200) can decode the final encoding data to obtain final decoded data (S2150). The XR device (200) can perform video rendering on the final decoded data to generate a final screen (S2155). The XR device (200) can output (or display) the final screen (S2160).

[0386] The XR device (200) can transmit the final screen to the electronic device (100) (S2165). The XR device (200) can transmit information about the final screen to the electronic device (100). The information about the final screen can include pixel information representing the final screen. The final screen can be described as an image frame corresponding to the final screen, image information corresponding to the final screen, or image data corresponding to the final screen.

[0387] The electronic device (100) can receive a final screen (or information about the final screen) from the XR device (200). The electronic device (100) can update the PSNR table based on the final screen.

[0388] The electronic device (100) can perform a first color space transformation on the combined data based on the updated PSNR table.

[0389] FIG. 22 is a diagram for explaining an operation of updating a PSNR table based on a color space lookup table transmitted from an external display device, according to one embodiment.

[0390] Referring to FIG. 22, it can be connected to a display device (300).

[0391] For example, the electronic device (100) may be connected to a display device (300) in addition to the XR device (200). The display device (300) may be a device that outputs content (video frames and UI frames) separately from the XR device (200).

[0392] The electronic device (100) can store a PSNR table. The electronic device (100) can perform a combining operation based on the PSNR table (S2205). The electronic device (100) can transmit video frames and UI frames to the display device (300) (S2210).

[0393] The display device (300) can store a color space lookup table. The display device (300) can receive video frames and UI frames from the electronic device (100). The display device (300) can perform a first color space conversion on the UI frame to obtain a converted frame (S2215).

[0394] The display device (300) can output (or display) the converted frame (S2220). The display device (300) can update the color space lookup table based on the result of performing the first color space conversion (S2225). The display device (300) can transmit the updated color space lookup table to the electronic device (100) (S2230).

[0395] The electronic device (100) can receive an updated color space lookup table from the display device (300). The electronic device (100) can update the PSNR table based on the updated color space lookup table (S2235).

[0396] The electronic device (100) can perform a combining operation based on the updated PSNR table. The combining operation can represent the combining operations of S1325 and S1340 of FIG. 13.

[0397] FIG. 23 is a diagram illustrating an operation of selecting a processing unit to perform a combining operation based on a vertical synchronization time, according to one embodiment.

[0398] Steps S2305 and S2310 of Fig. 23 may correspond to steps S1705 and S1710 of Fig. 17. Duplicate explanation is omitted.

[0399] The electronic device (100) can obtain a vertical synchronization time (T_V-sync) (S2315). The vertical synchronization time (T_V-sync) may represent a time for synchronizing the frame provision time and the frame display time. The vertical synchronization time (T_V-sync) may be described as a screen update cycle, a frame timing adjustment time, or an input delay time.

[0400] The electronic device (100) can identify whether the first time (T_NPU) is less than the vertical synchronization time (T_V-sync) (S2320).

[0401] If the first time (T_NPU) is less than the vertical synchronization time (T_V-sync) (S2320-Y), the electronic device (100) can obtain combined data using the NPU (S2325). The electronic device (100) can perform a combined operation using the NPU.

[0402] If the first time (T_NPU) is not less than the vertical synchronization time (T_V-sync), the electronic device (100) can identify whether the second time (T_GPU) is less than the vertical synchronization time (T_V-sync) (S2330).

[0403] If the second time (T_GPU) is less than the vertical synchronization time (T_V-sync) (S2330-Y), the electronic device (100) can obtain combined data using the GPU (S2335). The electronic device (100) can perform the combined operation using the GPU.

[0404] If the second time (T_GPU) is not less than the vertical synchronization time (T_V-sync) (S2330-N), the electronic device (100) can acquire combined data using both the NPU and the GPU (S2340). The electronic device (100) can perform the combined operation using both the NPU and the GPU. Some of the multiple operations performed in the combined operation can be performed through the NPU. The remaining operations can be performed through the GPU.

[0405] FIG. 24 is a drawing for explaining a method of controlling an electronic device according to one embodiment.

[0406] Referring to FIG. 24, a control method of an electronic device (100) includes a step of obtaining a video frame of a first color space and a UI frame of a second color space (S2405), a step of identifying a video region corresponding to a video frame and a UI region corresponding to a UI frame among the entire region where the video frame and the UI frame are displayed (S2410), a step of identifying a non-overlapping video region that does not overlap with the UI region among the video regions (S2415), a step of identifying an overlapping video region that overlaps with the UI region among the video regions (S2420), a step of encoding the non-overlapping video region to obtain first encoded data (S2425), a step of combining the overlapping video region and the UI frame to obtain combined data (S2430), a step of encoding the combined data to obtain second encoded data (S2435), and a step of merging the first encoded data and the second encoded data to obtain final encoded data (S2445).

[0407] The step of identifying the UI video area and the UI area (S2410) can divide the video frame and the UI frame into preset blocks, and identify the video area and the UI area based on the divided plurality of blocks.

[0408] The control method includes a step of identifying a first ratio of the entire area occupied by the video area, a step of identifying a second ratio of the entire area occupied by the UI area, and a step of obtaining a target ratio by dividing the first ratio by the second ratio, and the step of obtaining combined data (S2430) can obtain combined data based on the target ratio.

[0409] The control method includes a step of identifying whether video rendering is possible for converting from a second color space to a first color space if the target ratio is greater than or equal to a threshold ratio, and the step of obtaining combined data (S2430) includes obtaining combined data by applying video rendering to a UI frame if video rendering is possible, and obtaining combined data by applying UI rendering to a UI frame if video rendering is not possible.

[0410] The step of obtaining combined data (S2430) may include, if video rendering is possible, performing video rendering on a UI frame to obtain a first rendering UI frame, and combining the overlapping video area and the first rendering UI frame to obtain combined data.

[0411] The step of obtaining combined data (S2430) may include, if video rendering is impossible, performing UI rendering on a UI frame to obtain a second rendering UI frame, performing a first color space conversion on the second rendering UI frame to obtain a converted UI frame, and combining the overlapping video area and the converted UI frame to obtain combined data.

[0412] The first color space conversion may include an operation of converting pixel information of the second color space into pixel information of the first color space.

[0413] The control method may include the step of identifying whether the video frame is a 2D image and, if the video frame is not a 2D image, the step of performing 2D modeling on the video frame.

[0414] The first color space can be YUV or YUVA, and the second color space can be RGB or RGBA.

[0415] The control method may include a step of transmitting final encoded data to an Extended Reality (XR) device.

[0416] Embodiments of the method and device of the present disclosure improve the functionality of a computer by enhancing its display and user interface (UI) capabilities. These problems exist in the computing and networking domains. Therefore, the present embodiment relies on computer technology to address these problems arising in the computer network domain.

[0417] The methods according to the various embodiments of the present disclosure described above can be implemented in the form of an application that can be installed on an existing electronic device.

[0418] The methods according to the various embodiments of the present disclosure described above can be implemented only with a software upgrade or a hardware upgrade for an existing electronic device.

[0419] The various embodiments of the present disclosure described above may also be performed through an embedded server provided in an electronic device, or an external server of at least one of the electronic device and the display device.

[0420] According to an example embodiment of the present disclosure, the various embodiments described above may be implemented as software including instructions stored in a machine-readable storage medium that can be read by a machine (e.g., a computer). The device may include an electronic device according to the disclosed embodiments, which is a device that can call instructions stored in the storage medium and operate according to the called instructions. When the instructions are executed by a processor, the processor may directly or under the control of the processor use other components to perform a function corresponding to the instructions. The instructions may include code generated or executed by a compiler or interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' means that the storage medium does not contain signals and is tangible, but does not distinguish between whether data is stored semi-permanently or temporarily in the storage medium.

[0421] According to one embodiment of the present disclosure, the method according to the various embodiments described above may be provided as included in a computer program product. The computer program product may be traded as a commodity between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)) or online through an application store. In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated in a storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0422] Each of the components (e.g., modules or programs) according to the various embodiments described above may be composed of a single or multiple entities, and some of the sub-components described above may be omitted, or other sub-components may be further included in various embodiments. Alternatively or additionally, some components (e.g., modules or programs) may be integrated into a single entity, which may perform the same or similar functions as those performed by each of the respective components prior to integration. Operations performed by modules, programs or other components according to various embodiments may be executed sequentially, in parallel, iteratively or heuristically, or at least some operations may be executed in a different order, omitted, or other operations may be added.

[0423] Although the preferred embodiments of the present disclosure have been illustrated and described above, the present disclosure is not limited to the specific embodiments described above, and various modifications may be made by a person having ordinary skill in the art to which the present disclosure pertains without departing from the scope of the present disclosure as claimed in the claims, and such modifications should not be understood individually from the technical idea of ​​the present disclosure.

Claims

1. In electronic devices, Memory that stores instructions; At least one processor comprising processing circuitry; The above instructions, when individually or collectively executed by the at least one processor, Obtain a video frame of the first color space and a UI frame of the second color space, Identifying a video area corresponding to the video frame and a UI area corresponding to the UI frame among the display areas where the video frame and the UI frame are displayed, Identify a non-overlapping video area that does not overlap with the UI area among the above video areas, Identify an overlapping video area that overlaps the UI area among the above video areas, Encode the above non-overlapping video region to obtain first encoded data, Combine the above overlapping video area and the above UI frame to obtain combined data, Encode the above combined data to obtain second encoded data, An electronic device that obtains final encoded data by merging the first encoded data and the second encoded data.

2. In paragraph 1, The above instructions, when individually or collectively executed by the at least one processor, Divide the above video frame and the above UI frame into a plurality of blocks, An electronic device that identifies the video area and the UI area based on the plurality of divided blocks.

3. In paragraph 1, The above instructions, when individually or collectively executed by the at least one processor, Identify a first proportion of the above display area occupied by the video area, Identify a second ratio of the above display area occupied by the UI area, The target ratio is obtained by dividing the first ratio by the second ratio, An electronic device that obtains the combined data based on the target ratio.

4. In paragraph 3, The above instructions, when individually or collectively executed by the at least one processor, If the target ratio is greater than or equal to the threshold ratio, it is determined whether video rendering is possible by converting from the second color space to the first color space, If the above video rendering is possible, the combined data is obtained by applying the video rendering to the UI frame, An electronic device that obtains the combined data by applying UI rendering to the UI frame when the above video rendering is impossible.

5. In paragraph 4, The above instructions, when individually or collectively executed by the at least one processor, If the above video rendering is possible, the video rendering is performed on the UI frame to obtain the first rendering UI frame, An electronic device that obtains the combined data by combining the above overlapping video area and the first rendering UI frame.

6. In paragraph 4, The above instructions, when individually or collectively executed by the at least one processor, If the above video rendering is not possible, the UI rendering is performed on the UI frame to obtain a second rendering UI frame, Perform a first color space conversion on the second rendering UI frame to obtain a conversion UI frame, An electronic device that obtains the combined data by combining the above overlapping video area and the above transformation UI frame.

7. In paragraph 6, The above first color space transformation is, An electronic device comprising an operation of converting pixel information of the second color space into pixel information of the first color space.

8. In paragraph 4, The above instructions, when individually or collectively executed by the at least one processor, Identify whether the above video frame is a 2D image, An electronic device that performs 2D modeling on the video frame if the video frame is not a 2D image.

9. In paragraph 1, The above first color space is YUV or YUVA, An electronic device wherein the second color space is RGB or RGBA.

10. In paragraph 1, The above electronic device, Includes a communication interface that connects to an XR (Extended Reality) device, An electronic device that transmits the final encoded data to the XR device via the communication interface.

11. In a method for controlling an electronic device, A step of obtaining a video frame of a first color space and a UI frame of a second color space; A step of identifying a video area corresponding to the video frame and a UI area corresponding to the UI frame among the display areas in which the video frame and the UI frame are displayed; A step of identifying a non-overlapping video area that does not overlap with the UI area among the video areas; A step of identifying an overlapping video area overlapping the UI area among the above video areas; A step of encoding the above non-overlapping video region to obtain first encoding data; A step of obtaining combined data by combining the above overlapping video area and the UI frame; A step of encoding the above combined data to obtain second encoded data; and A control method comprising: a step of obtaining final encoding data by merging the first encoding data and the second encoding data.

12. In paragraph 11, The step of identifying the UI video area and the UI area is: Divide the above video frame and the above UI frame into a plurality of blocks, A control method for identifying the video area and the UI area based on the divided plurality of blocks.

13. In paragraph 11, The above control method is, A step of identifying a first proportion of the display area occupied by the video area; A step of identifying a second ratio of the UI area among the display areas; and A step of obtaining a target ratio by dividing the first ratio by the second ratio; The step of obtaining the above combined data is: A control method for obtaining the combined data based on the target ratio.

14. In paragraph 13, The above control method is, If the target ratio is greater than or equal to a threshold ratio, a step of identifying whether video rendering that converts from the second color space to the first color space is possible; The step of obtaining the above combined data is: If the above video rendering is possible, the combined data is obtained by applying the video rendering to the UI frame, A control method for obtaining the combined data by applying UI rendering to the UI frame when the above video rendering is impossible.

15. In paragraph 14, The step of obtaining the above combined data is: If the above video rendering is possible, the video rendering is performed on the UI frame to obtain the first rendering UI frame, A control method for obtaining the combined data by combining the above overlapping video area and the first rendering UI frame.

Images