Electronic device for video editing and operating method thereof

Abstract

Disclosed is an electronic device including: a memory; and at least one processor. The at least one processor may be configured to: acquire a first image; generating a first input frame to be mixed with the first image, in which each pixel data of the first input frame includes RGB data and mask information indicating a mixing method of each pixel data; assigning, to the first input frame, a first alpha map comprising per-pixel alpha values of the first input frame; determining a mixing method for frame color mixing of the first input frame and the first image based on the mask information; performing frame color mixing on the first input frame and a second input frame of the first image based on the determined mixing method and the first alpha map; and outputting the second image generated by frame color mixing.

Description

Technical Field

[0001] Various embodiments of the present disclosure relate to an electronic device for performing video editing and an operating method thereof. Background Art

[0002] More and more services and additional functions are being provided through electronic devices such as user terminals (e.g., smartphones). In order to increase the usefulness of electronic devices and meet various user needs, communication service providers or electronic device manufacturers are competing to develop electronic devices that provide various functions. Therefore, the various functions provided by electronic devices are becoming more and more complex.

[0003] The electronic device may include a function of generating (e.g., encoding) media data (e.g., video or image) or editing media data. The media data may include a video to which high dynamic range (HDR) is applied (hereinafter, referred to as "HDR video"). HDR is a technology that more accurately distinguishes brightness and darkness to display an image similar to that perceived by a user with the eyes.

[0004] Video editing may include decoding, processing, and encoding. Decoding may be an operation of decomposing (e.g., decompressing) a compressed video file into image data (e.g., frames) and audio data, processing may be an operation of editing the decomposed image data and audio data, and encoding is an operation of recompressing the edited image data and audio data to generate a video file. Processing may include, for example, at least one of trimming, speed editing, transforming (e.g., rotating and / or cropping), filtering, tone mapping, applying stickers, applying text, or drawing.

[0005] In video editing operations, the memory usage of electronic devices is essential. The memory capacity required for video editing can be determined by the video editing software used, the type of scenes to be edited, and the number of other applications or programs to be used simultaneously with the video editing. Typically, video editing software requires a large amount of memory, and therefore additional memory (e.g., random access memory (RAM)) may be required for a smooth video editing experience. Summary of the invention

[0006] Solution

[0007] The electronic device according to the embodiment may include a memory and at least one processor. The at least one processor may be configured to acquire a first image. The at least one processor may be configured to generate a first input frame to be mixed with the first image. Each pixel data of the first input frame may include RGB data and mask information indicating a mixing method of the pixel data. The at least one processor may be configured to assign a first alpha map including a per-pixel alpha value of the first input frame to the first input frame. The at least one processor may be configured to determine a mixing method for frame color mixing of the first input frame and the first image based on the mask information. The at least one processor may be configured to perform frame color mixing of the first input frame and the second input frame of the first image based on the determined mixing method and the first alpha map. The at least one processor may be configured to output a second image generated by frame color mixing.

[0008] A method of an electronic device for video editing according to an embodiment may include acquiring a first image. The method may include generating a first input frame to be mixed with the first image. Each pixel data of the first input frame may include RGB data and mask information indicating a mixing method of the pixel data. The method may include assigning a first alpha map including a per-pixel alpha value of the first input frame to the first input frame. The method may include determining a mixing method for frame color mixing of the first input frame and the first image based on the mask information. The method may include performing frame color mixing of the first input frame and a second input frame of the first image based on the determined mixing method and the first alpha map. The method may include outputting a second image generated by frame color mixing.

[0009] According to an embodiment, a non-transitory computer-readable recording medium may store one or more programs, and the one or more programs may include instructions, which, when executed by a processor of an electronic device, cause the electronic device to: acquire a first image; generate a first input frame to be mixed with the first image, each pixel data of the first input frame including RGB data and mask information indicating a mixing method of the pixel data; assign a first alpha map including a per-pixel alpha value of the first input frame to the first input frame; based on the mask information, determine a mixing method for frame color mixing of the first input frame and the first image; based on the determined mixing method and the first alpha map, perform frame color mixing of the first input frame and a second input frame of the first image; and output a second image generated by frame color mixing. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 is a block diagram of an electronic device in a network according to various embodiments;

[0011] Figure 2is a block diagram showing a configuration of an electronic device that performs video editing according to an embodiment;

[0012] Figure 3 shows SDR to HDR conversion according to an embodiment;

[0013] Figure 4a shows an HDR format of RGBA16F according to an embodiment;

[0014] Figure 4b An HDR format of RGBA 10101012 is shown according to an embodiment;

[0015] Figure 5a SDR video editing according to an embodiment is shown;

[0016] Figure 5b HDR video editing according to an embodiment is shown;

[0017] Figure 6 illustrates video editing in an HDR environment according to an embodiment;

[0018] Figure 7 shows HDR image frames and alpha information for video editing according to an embodiment;

[0019] Figure 8 illustrates frame color blending using alpha mapping according to an embodiment;

[0020] Figure 9a shows alpha blending reusing alpha maps according to an embodiment;

[0021] Figure 9b shows a pre-multiplied RGB mix according to an embodiment;

[0022] Fig.10 is a flow chart illustrating a video editing process according to an embodiment;

[0023] Fig.11 shows an editing object generated by a pen input function according to an embodiment;

[0024] Fig.12 An edited object image including pixel-based mask information according to an embodiment is shown;

[0025] Fig.13 shows a brushstroke including a mosaic pen according to an embodiment;

[0026] Fig.14 shows sticker blending without using mask information according to an embodiment;

[0027] Fig.15shows sticker blending using mask information according to an embodiment;

[0028] Fig.16 shows frame color blending based on alpha mapping and mask information according to an embodiment;

[0029] Fig.17 is a flow chart illustrating a video editing process using alpha mapping according to an embodiment;

[0030] Fig.18 is a flow chart illustrating a graffiti blending process using mask information according to an embodiment;

[0031] Fig.19 is a flowchart illustrating a sticker blending process using mask information according to an embodiment;

[0032] Fig. 20 illustrates memory usage according to an embodiment; and

[0033] Fig.21a and Figure 21b The storage of a graffiti image according to an embodiment is shown. DETAILED DESCRIPTION

[0034] Figure 1 is a block diagram illustrating an electronic device 101 in a network environment 100 according to various embodiments.

[0035] refer to Figure 1 , the electronic device 101 in the network environment 100 may communicate with the electronic device 102 via the first network 198 (e.g., a short-range wireless communication network), or communicate with at least one of the electronic device 104 or the server 108 via the second network 199 (e.g., a long-range wireless communication network). According to an embodiment, the electronic device 101 may communicate with the electronic device 104 via the server 108. According to an embodiment, the electronic device 101 may include a processor 120, a memory 130, an input module 150, a sound output module 155, a display module 160, an audio module 170, a sensor module 176, an interface 177, a connection terminal 178, a haptic module 179, a camera module 180, a power management module 188, a battery 189, a communication module 190, a user identification module (SIM) 196, or an antenna module 197. In some embodiments, at least one of the above components (e.g., the connection terminal 178) may be omitted from the electronic device 101, or one or more other components may be added to the electronic device 101. In some embodiments, some of the above-described components (eg, sensor module 176, camera module 180, or antenna module 197) may be implemented as a single integrated component (eg, display module 160).

[0036] The processor 120 may run, for example, software (e.g., program 140) to control at least one other component (e.g., hardware component or software component) of the electronic device 101 connected to the processor 120, and may perform various data processing or calculations. According to an embodiment, as at least part of the data processing or calculation, the processor 120 may store a command or data received from another component (e.g., sensor module 176 or communication module 190) in the volatile memory 132, process the command or data stored in the volatile memory 132, and store the resultant data in the non-volatile memory 134. According to an embodiment, the processor 120 may include a main processor 121 (e.g., a central processing unit (CPU) or an application processor (AP)) or an auxiliary processor 123 (e.g., a graphics processing unit (GPU), a neural processing unit (NPU), an image signal processor (ISP), a sensor hub processor, or a communication processor (CP)) that is independent of or combined with the main processor 121 in operation. For example, when the electronic device 101 includes a main processor 121 and an auxiliary processor 123, the auxiliary processor 123 may be adapted to consume less power than the main processor 121, or adapted to be dedicated to a specific function. The auxiliary processor 123 may be implemented separately from the main processor 121, or as part of the main processor 121.

[0037] When the main processor 121 is in an inactive (e.g., sleep) state, the auxiliary processor 123 (rather than the main processor 121) may control at least some of the functions or states related to at least one component among the components of the electronic device 101 (e.g., the display module 160, the sensor module 176, or the communication module 190), or when the main processor 121 is in an active state (e.g., running an application), the auxiliary processor 123 may control at least some of the functions or states related to at least one component among the components of the electronic device 101 (e.g., the display module 160, the sensor module 176, or the communication module 190) together with the main processor 121. According to an embodiment, the auxiliary processor 123 (e.g., an image signal processor or a communication processor) may be implemented as part of another component (e.g., a camera module 180 or a communication module 190) that is functionally related to the auxiliary processor 123. According to an embodiment, the auxiliary processor 123 (e.g., a neural processing unit) may include a hardware structure dedicated to artificial intelligence model processing. The artificial intelligence model may be generated through machine learning. For example, such learning may be performed by the electronic device 101 where the artificial intelligence is executed or via a separate server (e.g., server 108). The learning algorithm may include, but is not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, for example. The artificial intelligence model may include multiple artificial neural network layers. The artificial neural network may be a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), or a deep Q network or a combination of two or more thereof, but is not limited thereto. Additionally or optionally, the artificial intelligence model may include a software structure in addition to a hardware structure.

[0038] The memory 130 may store various data used by at least one component of the electronic device 101 (e.g., the processor 120 or the sensor module 176). The various data may include, for example, software (e.g., the program 140) and input data or output data for commands related thereto. The memory 130 may include a volatile memory 132 or a nonvolatile memory 134.

[0039] The program 140 may be stored as software in the memory 130 , and may include, for example, an operating system (OS) 142 , middleware 144 , or applications 146 .

[0040] The input module 150 may receive commands or data to be used by other components (e.g., the processor 120) of the electronic device 101 from outside (e.g., a user) of the electronic device 101. The input module 150 may include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus).

[0041] The sound output module 155 can output sound signals to the outside of the electronic device 101. The sound output module 155 may include, for example, a speaker or a receiver. The speaker can be used for general purposes such as playing multimedia or playing records. The receiver can be used to receive incoming calls. Depending on the embodiment, the receiver can be implemented as a separate part from the speaker, or as part of the speaker.

[0042] The display module 160 may visually provide information to the outside of the electronic device 101 (e.g., a user). The display device 160 may include, for example, a display, a holographic device, or a projector, and a control circuit for controlling a corresponding one of the display, the holographic device, and the projector. According to an embodiment, the display module 160 may include a touch sensor adapted to detect a touch or a pressure sensor adapted to measure the strength of a force caused by a touch.

[0043] The audio module 170 may convert sound into an electrical signal, or vice versa. According to an embodiment, the audio module 170 may obtain sound via the input module 150, or output sound via the sound output module 155 or an earphone of an external electronic device (e.g., electronic device 102) directly (e.g., wired) or wirelessly connected to the electronic device 101.

[0044] The sensor module 176 may detect an operating state (e.g., power or temperature) of the electronic device 101 or an environmental state (e.g., a user's state) outside the electronic device 101, and then generate an electrical signal or data value corresponding to the detected state. According to an embodiment, the sensor module 176 may include, for example, a gesture sensor, a gyro sensor, an atmospheric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an infrared (IR) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illumination sensor.

[0045] The interface 177 may support one or more specific protocols to be used to connect the electronic device 101 directly (e.g., wired) or wirelessly with an external electronic device (e.g., the electronic device 102). According to an embodiment, the interface 177 may include, for example, a high-definition multimedia interface (HDMI), a universal serial bus (USB) interface, a secure digital (SD) card interface, or an audio interface.

[0046] The connection end 178 may include a connector, wherein the electronic device 101 can be physically connected to an external electronic device (e.g., the electronic device 102) via the connector. According to an embodiment, the connection end 178 may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0047] The haptic module 179 may convert the electric signal into mechanical stimulation (eg, vibration or motion) or electric stimulation that can be recognized by the user via his sense of touch or kinesthetic sense. According to an embodiment, the haptic module 179 may include, for example, a motor, a piezoelectric element, or an electric stimulator.

[0048] The camera module 180 may capture still images or moving images. According to an embodiment, the camera module 180 may include one or more lenses, an image sensor, an image signal processor, or a flash.

[0049] The power management module 188 may manage power supply to the electronic device 101. According to an embodiment, the power management module 188 may be implemented as, for example, at least a part of a power management integrated circuit (PMIC).

[0050] The battery 189 may power at least one component of the electronic device 101. According to an embodiment, the battery 189 may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0051] The communication module 190 may support establishing a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device 101 and an external electronic device (e.g., electronic device 102, electronic device 104, or server 108), and perform communication via the established communication channel. The communication module 190 may include one or more communication processors capable of operating independently from the processor 120 (e.g., an application processor (AP)) and supporting direct (e.g., wired) communication or wireless communication. According to an embodiment, the communication module 190 may include a wireless communication module 192 (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module 194 (e.g., a local area network (LAN) communication module or a power line communication (PLC) module). A corresponding one of these communication modules can communicate with an external electronic device via a first network 198 (e.g., a short-range communication network such as Bluetooth, Wireless Fidelity (Wi-Fi) Direct, or Infrared Data Association (IrDA)) or a second network 199 (e.g., a long-range communication network such as a traditional cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or a wide area network (WAN))). These various types of communication modules may be implemented as a single component (e.g., a single chip), or may be implemented as multiple components (e.g., multiple chips) separated from each other. The wireless communication module 192 may identify and authenticate the electronic device 101 in a communication network (such as the first network 198 or the second network 199) using user information (e.g., an International Mobile Subscriber Identity (IMSI)) stored in the user identification module 196.

[0052] The wireless communication module 192 can support 5G networks after 4G networks and next-generation communication technologies (e.g., new radio (NR) access technologies). NR access technologies can support enhanced mobile broadband (eMBB), massive machine type communications (mMTC), or ultra-reliable low-latency communications (URLLC). The wireless communication module 192 can support high-frequency bands (e.g., millimeter wave bands) to achieve, for example, high data transmission rates. The wireless communication module 192 can support various technologies for ensuring performance on high-frequency bands, such as, for example, beamforming, massive multiple-input multiple-output (massive MIMO), full-dimensional MIMO (FD-MIMO), array antennas, analog beamforming, or massive antennas. The wireless communication module 192 can support various requirements specified in the electronic device 101, an external electronic device (e.g., electronic device 104), or a network system (e.g., a second network 199). According to an embodiment, the wireless communication module 192 may support a peak data rate for implementing eMBB (e.g., 20 Gbps or greater), loss coverage for implementing mMTC (e.g., 164 dB or less), or U-plane delay for implementing URLLC (e.g., 0.5 ms or less for each of the downlink (DL) and uplink (UL), or 1 ms or less for a round trip).

[0053] The antenna module 197 may transmit or receive a signal or power to or from the outside of the electronic device 101 (e.g., an external electronic device). According to an embodiment, the antenna module 197 may include an antenna including a radiation element formed of a conductive material or a conductive pattern formed in or on a substrate (e.g., a printed circuit board (PCB)). According to an embodiment, the antenna module 197 may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication scheme used in a communication network (such as the first network 198 or the second network 199) may be selected from the plurality of antennas by, for example, the communication module 190 (e.g., the wireless communication module 192). A signal or power may then be transmitted or received between the communication module 190 and the external electronic device via the selected at least one antenna. According to an embodiment, another component (e.g., a radio frequency integrated circuit (RFIC)) other than the radiation element may be additionally formed as a part of the antenna module 197.

[0054] According to various embodiments, the antenna module 197 may form a millimeter wave antenna module. According to an embodiment, the millimeter wave antenna module may include a printed circuit board, a radio frequency integrated circuit (RFIC), and a plurality of antennas (e.g., array antennas), wherein the RFIC is disposed on a first surface (e.g., bottom surface) of the printed circuit board, or adjacent to the first surface and capable of supporting a designated high frequency band (e.g., millimeter wave band), and the plurality of antennas are disposed on a second surface (e.g., top surface or side surface) of the printed circuit board, or adjacent to the second surface and capable of transmitting or receiving signals of the designated high frequency band.

[0055] At least some of the above components may be connected to each other via an inter-peripheral communication scheme (e.g., a bus, general purpose input output (GPIO), a serial peripheral interface (SPI), or a mobile industry processor interface (MIPI)) and communicatively transmit signals (e.g., commands or data) therebetween.

[0056] According to an embodiment, a command or data may be sent or received between the electronic device 101 and the external electronic device 104 via the server 108 connected to the second network 199. Each of the electronic device 102 or the electronic device 104 may be a device of the same type as the electronic device 101, or a device of a different type from the electronic device 101. According to an embodiment, all or some operations to be executed in the electronic device 101 may be executed in one or more of the external electronic device 102, the external electronic device 104, or the server 108. For example, if the electronic device 101 should automatically execute a function or service or should execute a function or service in response to a request from a user or another device, the electronic device 101 may request the one or more external electronic devices to execute at least part of the function or service instead of executing the function or service, or the electronic device 101 may request the one or more external electronic devices to execute at least part of the function or service in addition to executing the function or service. The one or more external electronic devices receiving the request may execute at least part of the requested function or service, or execute another function or another service related to the request, and transmit the result of the execution to the electronic device 101. The electronic device 101 may provide the result as at least a partial reply to the request with or without further processing the result. To this end, cloud computing technology, distributed computing technology, mobile edge computing (MEC) technology, or client-server computing technology, for example, may be used. The electronic device 101 may use, for example, distributed computing or mobile edge computing to provide ultra-low latency services. In another embodiment, the external electronic device 104 may include an Internet of Things (IoT) device. The server 108 may be an intelligent server using machine learning and / or neural networks. According to an embodiment, the external electronic device 104 or the server 108 may be included in the second network 199. The electronic device 101 may be applied to intelligent services (e.g., smart homes, smart cities, smart cars, or healthcare) based on 5G communication technology or IoT-related technologies.

[0057] Figure 2 is a block diagram illustrating a configuration of an electronic device that performs video editing according to an embodiment.

[0058] refer to Figure 2, the electronic device 200 (e.g., the electronic device 101) may include a processor 210 (e.g., the processor 120), a display 220 (e.g., the display module 160), and a memory 230 (e.g., the memory 130). The processor 210 may include, for example, a central processing unit (CPU) and / or a graphics processing unit (GPU).

[0059] The processor 210 may perform video editing on media data (e.g., images or videos) read from the memory 230 or received from an external electronic device (not shown). In an embodiment, video editing may include decoding, video processing, and / or encoding. Decoding may decompress (e.g., decode) a compressed video file into image data (e.g., frames, image frames, or video frames) and audio data. Video processing may edit the decomposed image data and audio data. Video processing may include, for example, trimming, speed editing, transformation (e.g., rotation and / or cropping), filters, tone mapping, stickers, text, or drawing. Encoding may compress the image data and audio data edited by video processing to generate a new video file.

[0060] The processor 210 may configure (e.g., allocate) a plurality of frame buffers in the memory 230 for video editing. The processor 210 may store the frames configured by decoding in the plurality of frame buffers respectively, and may read out and edit the frames stored in the frame buffers one by one. The edited frames may be stored in the corresponding frame buffers again. The processor 210 may configure (e.g., allocate) a memory area in the memory 230 for storing information related to video editing. In an embodiment, the information related to video editing may include an editing object and / or alpha information (e.g., alpha value) for alpha blending. In an embodiment, the editing object may be generated based on user input during video editing and stored in the memory area.

[0061] In an embodiment, alpha blending may refer to a process of generating partial or full transparency when combining one image with another image (e.g., a background image) in video editing. Alpha blending may assign a specified transparency (e.g., a transparent attribute) to pixels of an image and combine the pixels of the image with pixels of the background image. The transparency of each pixel may be determined by a corresponding alpha value, and the alpha value may be included in the pixel data of the image or specified separately.

[0062] In video editing, modes for processing edited content (e.g., edited frames) may include previewing or storing (e.g., exporting). In video editing in preview mode, the processor 210 may sequentially read out edited frames from the frame buffer and output the frames to the display 220. In an embodiment, the display 220 may include an external display device included in the electronic device 200 or connected via a communication circuit (e.g., communication module 190). In video editing in export mode, the processor 210 may sequentially read out edited frames from the frame buffer and encode the frames to generate new media data (e.g., a video file).

[0063] In an embodiment of the present disclosure, the media data may include image data (e.g., HDR images and / or HDR videos) to which HDR (e.g., HDR10 or HDR10+) is applied. The processor 210 may be configured to edit and store HDR images in an HDR environment so as to save memory and improve performance without losing video quality. In video editing in an HDR environment, the memory may be overused to store editing information (e.g., editing objects) generated according to the pixel format representing the HDR image during complex video editing, or some editing information may be lost. In an embodiment of the present disclosure, the processor 210 may reuse some memory areas allocated during the video editing process (e.g., memory areas for storing alpha information for alpha blending) for the next edit (e.g., blending) to reduce buffer read operations, thereby improving editing performance and increasing memory usage efficiency. In an embodiment of the present disclosure, the processor 210 may use some bits of the pixel format representing the HDR image as mask information for pixel-based editing functions (e.g., blending methods), thereby improving editing performance without increasing the required memory capacity.

[0064] Video editing may be performed based on a high dynamic range (HDR) color space or a standard dynamic range (SDR) color space. According to an embodiment, the electronic device 200 may perform video editing in an SDR environment, and may perform an operation of converting an SDR-based image (e.g., an SDR image and / or an SDR video) for which video editing has been completed into an HDR-based image (e.g., an HDR image and / or an HDR video) to obtain an HDR-based edited image.

[0065] Figure 3 SDR to HDR conversion according to an embodiment is shown.

[0066] refer to Figure 3, the SDR image 310 may be converted into an HDR image 330 by a conversion module 320. The conversion module 320 may be configured as, for example, a software module included in the processor 210 (e.g., executed by the processor 210). The conversion module 320 may include de-gamma (e.g., gamma decoding) and scaling, color gamut conversion, and re-gamma (e.g., gamma encoding).

[0067] In order to mix image frames having different color spaces, the image frames may be converted to have the same color space before mixing. An image frame edited in an SDR environment (eg, SDR image 310 ) may be converted into an HDR image by conversion module 320 .

[0068] The RGB color space, which is one of the color spaces, may refer to a method of representing one pixel as red (red, R), green (green, G), blue (blue, B), and alpha (alpha, A). Here, A may refer to an alpha value indicating the transparency of a pixel. In an embodiment, one pixel of an SDR image (e.g., SDR image 310) may be represented as a color type RGBA8888 (8 bits per color channel). In an embodiment, one pixel of an HDR image (e.g., HDR image 330) may be represented as an RGBA 16Float (RGBA16F) type (e.g., P016 format) or an RGBA 1010102 type (e.g., HDR10 / 10+ format).

[0069] Figure 4a An HDR format of RGBA16F according to an embodiment is shown.

[0070] refer to Figure 4a , the SDR pixel data 402 of the SDR image (e.g., the SDR image 310) may include 8 bits representing R, 8 bits representing G, 8 bits representing B, and 8 bits representing A (e.g., alpha α). For example, the SDR pixel data 402 may be referred to as an RGBA8888 type. The SDR pixel data 402 may be converted into HDR pixel data 404 of the RGBA16F type by an SDR-HDR conversion (e.g., the conversion module 320). The HDR pixel data 404 may include 16 bits representing R, 16 bits representing G, 16 bits representing B, and 16 bits representing A. In the HDR pixel data 404, since R, G, B, and A are all 16 bits, the HDR image (e.g., the HDR image 330) may occupy 8 bytes per pixel.

[0071] Figure 4b An HDR format of RGBA 10101012 is shown according to an embodiment.

[0072] refer to Figure 4b, the SDR pixel data 412 including R, G, B, and A, each represented as 8 bits, can be converted into HDR pixel data 414 of RGBA 1010102 type by SDR-HDR conversion (e.g., conversion module 320). The HDR pixel data 414 may include 10 bits representing R, 10 bits representing G, 10 bits representing B, and 2 bits representing A.

[0073] The image frame acquired by decoding may include color information (e.g., R, G, and B) in an opaque form. The conversion module 320 supporting HDR10 may configure 1 word (4 bytes) of HDR pixel data 414 by representing R, G, and B, each representing a color, as 10 bits and adding 2 bits of padding to represent A. The HDR pixel data 414 may occupy a memory area of ​​the same size (e.g., 4 bytes) as the memory area required for the color representation of the SDR pixel data 412. An HDR image (e.g., HDR image 330) including the HDR pixel data 414 may occupy 4 bytes per pixel.

[0074] For example, in HDR editing using the RGBA16F type, an 8-byte memory area for representing each of R, G, B, and A as 16 bits may be required to store one pixel data (e.g., HDR pixel data 404). Therefore, HDR editing using the RGBA16F type may require twice the memory usage of SDR pixel data (e.g., SDR pixel data 402) using RGBA8888. In addition, when editing an editing object for video editing and synthesizing (e.g., mixing) the editing object with the original video (e.g., each frame of the video), additional memory capacity may be required to store editing information (e.g., editing objects and / or alpha information), which may cause performance degradation of the electronic device 200 during video editing.

[0075] For example, in HDR editing using the RGBA 1010102 type, a 4-byte memory area for R, G, B, and A may be required to store one pixel data (e.g., HDR pixel data 414). In this case, when the color space of HDR10 is represented by 32 bits, a precision difference may occur between RGB information (R, G, and B) and alpha information (A). Alpha blending may be a very important operation in video editing. However, when 2 bits of alpha information are used, which is less than 8 bits of SDR pixel data 412, editing information (e.g., some of the 8-bit alpha information) may be lost when performing alpha blending or animation, and thus natural alpha blending may not be performed.

[0076] Figure 5a SDR video editing according to an embodiment is shown.

[0077] refer to Figure 5a , the decoder 502 may decode the compressed media data (e.g., a video file) to output an HDR image 504 (e.g., an HDR image frame in an HDR10 / 10+ format). The HDR-SDR conversion module 506 may convert the HDR image 504 into an SDR image 508 (e.g., an RGBA 8888 format). The SDR rendering module 510 may perform video editing (e.g., mixing and / or rendering) on ​​the SDR image 508 in an SDR environment and may output an edited SDR image (not shown).

[0078] The SDR-HDR conversion module 512 may convert the edited SDR image into an edited HDR image (e.g., an HDR image frame in HDR10 / 10+ format) (not shown). The RGB-YUV conversion module 514 may convert the edited HDR image into a YUV image in YUV format (e.g., a YUV image frame). The YUV image may represent each pixel as a luminance component Y and two chrominance components (e.g., U represents a blue projection, and V represents a red projection). The encoder 516 may encode the YUV image to generate edited media data (e.g., an edited video file).

[0079] The SDR rendering module 510 may be configured to perform rendering (e.g., blending) in an SDR color space, and thus may require the HDR-SDR conversion module 506 to convert the HDR image 504 into an SDR image 508. As a result, the quality of the HDR image 504 may be lost. Furthermore, the SDR-HDR conversion module 512 may additionally be used to convert the edited SDR image into an HDR format.

[0080] In an embodiment, at least one of the decoder 502, the HDR-SDR conversion module 506, the SDR rendering module 510, the SDR-HDR conversion module 512, the RGB-YUV conversion module 514, or the encoder 516 may be configured as a software module executed by a processor (e.g., the processor 210).

[0081] Figure 5b HDR video editing according to an embodiment is shown.

[0082] refer to Figure 5b, the decoder 520 can decode the compressed media data (e.g., a video file) to output an HDR image 522 (e.g., an HDR image frame in an HDR10 / 10+ format). The HDR rendering module 524 can perform video editing (e.g., mixing and / or rendering) on ​​the HDR image 522 in an HDR environment, and can output an edited HDR image (e.g., an HDR image frame in an HDR10 / 10+ format) (not shown). The RGB-YUV conversion module 526 can convert the edited HDR image into a YUV image (e.g., a YUV image frame). The encoder 528 can encode the YUV image to generate edited media data (e.g., an edited video file).

[0083] In an embodiment, at least one of the decoder 520 , the HDR rendering module 524 , the RGB-YUV conversion module 526 , or the encoder 528 may be configured as a software module executed by a processor (eg, the processor 210 ).

[0084] Embodiments of the present disclosure may support HDR rendering (e.g., blending) that enables the use of editing functions of an SDR environment while maintaining the quality of the original HDR image (e.g., HDR image 522). Embodiments of the present disclosure may reduce the increase in memory usage during HDR rendering compared to SDR rendering.

[0085] In an embodiment of the present disclosure, when performing complex video editing, the electronic device 200 may reuse a memory area storing alpha information for alpha blending, thereby resolving inaccuracies in alpha information and reducing memory usage for playback and / or storage.

[0086] In an embodiment of the present disclosure, the electronic device 200 may reuse a memory area storing alpha information, thereby providing mask information for a pixel-based editing function (eg, pixel-based blending of graffiti and / or stickers) by using the obtained extra bits.

[0087] In an embodiment of the present disclosure, the electronic device 200 may use an RGBA 1010102 type to express HDR color information, but perform alpha blending when necessary by reusing alpha information stored in a separate memory area (e.g., alpha mapping) instead of including alpha information in pixel data of the RGBA 1010102 type (e.g., HDR pixel data 708), thereby eliminating inaccuracies caused by using 2-bit alpha information (A) during video editing and reducing memory usage.

[0088] In an embodiment of the present disclosure, the electronic device 200 may store alpha information (e.g., alpha map) used for alpha blending in a designated memory area (e.g., the first area), and when alpha blending is required, alpha information not included in the pixel data (e.g., HDR pixel data 708) may be used, thereby reducing memory read operations.

[0089] In an embodiment of the present disclosure, the electronic device 200 may configure mask information (e.g., mask information 706) for applying a pixel-based editing effect (e.g., pixel-based blending of graffiti or stickers) in video editing in the free 2 bits of pixel data obtained by using alpha information in a designated memory area (e.g., the first area).

[0090] Figure 6 Video editing in an HDR environment is shown according to an embodiment.

[0091] refer to Figure 6 , the processor 210 of the electronic device 200 may include at least one of an editing module 610, an SDR-HDR conversion module 615, an alpha (α) map management module 620, a mask analysis module 625, or a frame color mixing module 630. In an embodiment, at least one of the editing module 610, the SDR-HDR conversion module 615, the alpha (α) map management module 620, the mask analysis module 625, or the frame color mixing module 630 may correspond to an HDR rendering module (e.g., the HDR rendering module 524). In an embodiment, at least one of the editing module 610, the SDR-HDR conversion module 615, the alpha (α) map management module 620, the mask analysis module 625, or the frame color mixing module 630 may be configured as a software module executed by the processor 210. For example, the operation of at least one of the editing module 610, the SDR-HDR conversion module 615, the alpha (α) map management module 620, the mask analysis module 625, or the frame color mixing module 630 may be understood as the operation of the processor 210.

[0092] For HDR video editing, an input HDR image 605 (e.g., at least one frame) may be selected by a user. In an embodiment, the input HDR image 605 may be selected from a memory (e.g., Figure 2The input HDR image 605 may be read from the memory 230 or an external memory (not shown), recorded by a camera (not shown), or received from an external electronic device (not shown) to the electronic device 210. The input HDR image 605 may be stored in a designated memory area (e.g., a frame buffer) for video editing. For example, a designated number of frames of the input HDR image 605 may be stored in a designated number of frame buffers, respectively. The input HDR image 605 is generated by decoding and therefore may not include alpha information. For example, when the input HDR image 605 (e.g., an RGB frame) is first stored in one of the frame buffers, the storage space (e.g., 2 bits) for per-pixel alpha information may be empty.

[0093] The editing module 610 may generate one or more objects to be used for editing (hereinafter referred to as editing objects) through the editing function provided for video editing. For example, at least one editing object may include a graffiti image (e.g., a brush stroke) or a sticker image. In an embodiment, the editing module 610 may generate the editing object through user input, or may read the editing object from a memory (e.g., memory 230). In an embodiment, the editing module 610 may change (e.g., edit) the editing object through user input.

[0094] At least some of the editing objects to be used for video editing may include SDR images. The SDR-HDR conversion module 615 may convert editing objects (e.g., SDR images) in SDR format into RGB frames in HDR format. The SDR-HDR conversion module 615 may convert editing objects into RGB frames in HDR format that are editable and synthesizable in an HDR environment. Each RGB frame output from the SDR-HDR conversion module 615 may include, for example, 10-bit pixel data (e.g., pixel data 708) per color channel of an HDR color space, and each pixel data may not include alpha information (e.g., alpha value 710). The RGB frame may be stored in a designated memory area (e.g., a second area) of the memory 230 for video editing.

[0095] In an embodiment, the editing object may include at least one of a sticker, a caption, a doodle, a transition, a tone map, or a filter. In an embodiment, at least one of the editing objects may include a texture image that the user wants to add or synthesize with the original image (e.g., the input HDR image 605).

[0096] In an embodiment, the editing module 610 may generate mask information corresponding to the editing object. In an embodiment, when an RGB frame including the editing object is stored in a designated memory area (e.g., a second area), the editing module 610 may store mask information (e.g., 2 bits) indicating a pixel-based editing function (e.g., a blending method) instead of alpha information of the RGB frame in a storage space (e.g., 2 bits) allocated for per-pixel alpha information in the memory area. The mask information may indicate an editing function (e.g., a filter and / or a mask) to be applied to each pixel.

[0097] Will refer to Figures 11 to 15 Embodiments related to video editing using mask information are described.

[0098] The alpha map management module 620 may store and / or manage alpha maps suitable for use in a memory (e.g., Figure 2 One or more alpha maps for each frame input for video editing (e.g., input HDR image 605 or RGB frame including at least one editing object) in memory 230 of the video editor may be stored, and an alpha map to be applied to a current frame to be edited (e.g., input HDR image 605 or RGB frame including at least one editing object) may be determined. In an embodiment, the alpha map may include alpha information (e.g., per-pixel alpha value) applicable to a plurality of pixels forming an RGB frame. In an embodiment, the per-pixel alpha value included in the alpha map may include 8 bits.

[0099] The alpha map management module 620 can manage one or more alpha maps that can be used for alpha blending of an original image to be edited (e.g., the input HDR image 605) and an RGB frame including at least one editing object, and can determine the generation and reuse of the alpha map. In an embodiment, the alpha map management module 620 can send the alpha map to be applied to the current frame to be edited (e.g., the input HDR image 605 or the RGB frame including at least one editing object) to the frame color blending module 630.

[0100] In an embodiment, the operation of the alpha map management module 620 sending the alpha map to the frame color mixing module 630 may include an operation of providing information (e.g., a pointer) indicating an address of a memory area (e.g., a first area) storing the alpha map to the frame color mixing module 630. In an embodiment, the alpha map management module 620 may store at least one alpha map in a designated memory area (e.g., the first area) allocated for video editing, and when it is determined that at least one alpha map is needed according to the video editing operation of the editing module 610, a pointer indicating the address of the at least one alpha map may be sent to the frame color mixing module 630.

[0101] When alpha blending of an input frame (e.g., the input HDR image 605 and an RGB frame including at least one editing object) is required, the frame color blending module 630 may perform alpha blending by using at least one alpha map provided from the alpha map management module 620. In an embodiment, the frame color blending module 630 may acquire at least one alpha map corresponding to the input frame from a memory area (e.g., a first area) corresponding to an address of a pointer provided from the alpha map management module 620. Alpha information updated by the alpha blending may be stored (e.g., rewritten) again in the memory area (e.g., the first area) by the frame color blending module 630.

[0102] When video editing of other input frames is required, the alpha map management module 620 can provide the alpha map (e.g., a pointer indicating the address of the first region storing the alpha map) back to the frame color blending module 630. The frame color blending module 630 can use the 8-bit per pixel alpha map provided from the alpha map management module 620 to perform improved alpha blending compared to the case of using 2-bit per pixel alpha information for the RGBA 1010102 type without significantly increasing memory usage.

[0103] The mask analysis module 625 may identify mask information (e.g., 2 bits) stored with the RGB frame including the editing object. The mask information may indicate a pixel-based editing function (e.g., a blending method) for the RGB frame. The mask analysis module 625 may forward the mask information to the frame color blending module 630, and the frame color blending module 630 may determine a per-pixel blending method based on the mask information.

[0104] The frame color blending module 630 may perform blending and / or rendering based on the alpha map read from the memory area of ​​the address corresponding to the pointer provided from the alpha map management module 620 and the information about the pixel-based editing function provided by the mask analysis module 625 (e.g., 2-bit mask information) together with the input frame (e.g., the input HDR image 605 or the RGB frame including at least one editing object), thereby generating an output HDR image 635 (e.g., a frame synthesized from the input frame). The output HDR image 635 may be generated without compromising the video quality of the input HDR image 605.

[0105] Figure 7 HDR image frames and alpha information for video editing are shown according to an embodiment.

[0106] refer to Figure 7, pixel data 702 representing one pixel of an SDR image (e.g., a 32-bit RGB frame) including at least one editing object may be converted by the SDR-HDR conversion module 615 into pixel data 708 (e.g., RGB pixel data) including 10-bit RGB data 704 per color channel. The editing module 610 may generate 2-bit mask information 706 indicating a pixel-based editing function of the pixel data 708, and may include the mask information 706 in the pixel data 708. The pixel data 708 may include the 2-bit mask information 706 instead of alpha information (e.g., a 2-bit alpha value).

[0107] The alpha map management module 620 can determine alpha information (e.g., alpha map) to be applied to video editing of RGB frames. The alpha map may include per-pixel alpha values ​​(e.g., 8 bits) corresponding to the RGB frame (e.g., alpha value 710 to be used for alpha blending of pixel data 708). In an embodiment, when alpha blending of an input frame (e.g., RGB frame) is required according to an editing function performed by the editing module 610, the alpha map management module 620 may allocate (e.g., store) the alpha map to a designated memory area (e.g., a first area) so that the alpha map can be used for alpha blending of the RGB frame. In the present disclosure, the term "alpha map" should be understood as a form of alpha information including per-pixel alpha values, and may also be configured as a table, a list, or various forms not mentioned in the present disclosure.

[0108] The RGB pixel data 708 (e.g., RGB data 704 and mask information 706) occupies the same 32 bits as the SDR pixel data 702, and when alpha blending of the RGB frame is required, the alpha map management module 620 can forward the alpha map (e.g., alpha value 710) to the frame color blending module 630. The alpha map can be updated by alpha blending of the RGB frame. In an embodiment, the alpha map management module 620 can forward a pointer indicating a memory area (e.g., a first area) storing the alpha map to the frame color blending module 630. After the alpha blending of the RGB frame is completed, the alpha map management module 620 can retain the alpha map without removing the alpha map from the first area, and the alpha map (e.g., updated alpha value) can be reused for alpha blending of another RGB frame (not shown).

[0109] The mask information 706 may indicate a specified editing function (e.g., at least one of mosaic, alpha blending, no alpha blending, source-dependent alpha blending, target-dependent alpha blending, or linear blending) to be applied to the RGB data 704. In an embodiment, for at least one pixel data of a frame including at least one brush stroke (e.g., the RGB data 704), the editing module 610 may configure the mask information 706 to indicate a value of any one of mosaic, a pen with alpha blending, or a pen without alpha blending. In an embodiment, for at least one pixel data of a frame including a sticker image (e.g., the RGB data 704), the editing module 610 may configure the mask information 706 to indicate a value of any one of source-dependent alpha blending (following the alpha of the source), target-dependent alpha blending (following the alpha of the target), or linear blending.

[0110] Figure 8 Frame color blending using alpha mapping is shown according to an embodiment.

[0111] refer to Figure 8 , the frame color mixing module 630 may receive a first input RGB frame 810 and a second input RGB frame 820. At least one of the first input RGB frame 810 or the second input RGB frame 820 may include an input HDR image (e.g., 10-bit RGB data 704 per color channel), an RGB frame corresponding to at least one editing object (e.g., 10-bit RGB data 704 per color channel), or an RGB frame generated by mixing and video edited (e.g., 10-bit RGB data 704 per color channel). At least one of the first input RGB frame 810 or the second input RGB frame 820 may not include alpha information (e.g., per-pixel alpha value). At least one of the first input RGB frame 810 or the second input RGB frame 820 may include mask information (e.g., 2 bits per pixel) instead of alpha information (e.g., per-pixel alpha value).

[0112] The alpha map management module 620 may determine that alpha blending of the first input RGB frame 810 and the second input RGB frame 820 is required through the control of the editing module 610. For alpha blending, the alpha map management module 620 may map a first input alpha map 815 (e.g., 8-bit alpha value per pixel) to the first input RGB frame 810, and may map a second input alpha map 825 (e.g., 8-bit alpha value per pixel) to the second input RGB frame 820. In an embodiment, the alpha map management module 620 may store one or more alpha maps that may be used for alpha blending, or may generate one or more alpha maps that may be used for alpha blending when needed. In an embodiment, the alpha map management module 620 may generate at least one of the first input alpha map 815 or the second input alpha map 825, or may obtain one from a stored alpha map.

[0113] The frame color blending module 630 may alpha blend the first input RGB frame 810 and the first input alpha map 815 with the second input RGB frame 820 and the second input alpha map 825 to generate an output RGB frame 830 and an output alpha map 835. The alpha map management module 620 may obtain the output alpha map 835 and may store the output alpha map 835 to be reused for alpha blending of a subsequent input frame (not shown).

[0114] When alpha blending of the first input RGB frame 810 and the second input RGB frame 820 is not required, the alpha map management module 620 may not allocate an alpha map.

[0115] Figure 9a Alpha blending reusing an alpha map according to an embodiment is shown.

[0116] refer to Figure 9a , the frame color blending module 630 may perform a first alpha blending 925 on the first frame data 910 and the second frame data 920. The first frame data 910 may include an RGB frame 912 (e.g., 10-bit RGB data per color channel) and a first alpha map 914 allocated by the alpha map management module 620, and the second frame data 920 may include an RGB frame 922 (e.g., 10-bit RGB data per color channel) and a second alpha map 924. In an embodiment, the RGB data 912 and the RGB data 922 may be configured in an RGBA1010102 format, but a 2-bit alpha channel ("A") may be empty (e.g., "00") or may include mask information. In an embodiment, the second alpha map 924 may include alpha information (e.g., per-pixel alpha value) allocated by the alpha map management module 620 or generated in a previous blend.

[0117] The frame color mixing module 630 may generate third frame data 930 including RGB data 932 (e.g., 10-bit RGB data per color channel) and a third alpha map 934 through the first alpha mixing 925. In an embodiment, the alpha map management module 620 may obtain the third alpha map 934 generated by the first alpha mixing 925, and may store the third alpha map 934 generated by the frame color mixing module 630 instead of the first alpha map 914 for the next alpha mixing (e.g., alpha mixing 945). In an embodiment, the RGB data 932 and the RGB data 942 may be configured in an RGBA1010102 format, but a 2-bit alpha channel (“A”) may be empty (e.g., “00”), or may include mask information.

[0118] The frame color mixing module 630 may perform a second alpha mixing 945 on the third frame data 930 and the fourth frame data 940. For example, the fourth frame data 940 may include an RGB frame 942 (e.g., 10-bit RGB data per color channel) containing at least one editing object and a fourth alpha map 944 allocated by the alpha map management module 620. The frame color mixing module 630 may generate a fifth frame data 950 including RGB data 952 (e.g., 10-bit RGB data per color channel) and a fifth alpha map 954 through the second alpha mixing 945. In an embodiment, the alpha map management module 620 may obtain the fifth alpha map 934 generated by the second alpha mixing 945 instead of the third alpha map 914, and may store the fifth alpha map 954 for the next alpha mixing (not shown). In an embodiment, the RGB data 952 may be configured in an RGBA1010102 format, but a 2-bit alpha channel ("A") may be empty (e.g., "00"), or may include mask information.

[0119] As described above, the frame color mixing module 630 may perform video editing (e.g., alpha blending 945) again on the result (e.g., third frame data 930) of the video editing (e.g., alpha blending 925). The alpha map management module 620 may continuously retain the third alpha map 934 generated as a result of the alpha blending 925, and may deallocate the fifth alpha map 954 when the video editing is completed. The alpha map management module 620 may retain alpha information (e.g., third alpha map 934) that may be reused for the next video editing (e.g., alpha blending 945), thereby preventing N alpha maps from being generated during N video editings (e.g., N alpha blendings).

[0120] Only in the case where alpha blending is required, the frame color blending module 630 can allocate and use the alpha map (e.g., the first alpha map 914 and the second alpha map 924) from the alpha map management module 620 instead of using an input frame including 32-bit RGBA data and 8-bit alpha information, thereby maintaining similar memory as when using 32-bit SDR images (e.g., Figure 2 The amount of memory 230) used.

[0121] In an embodiment, when the editing module 610 performs foreground editing, the alpha map management module 620 may allocate alpha maps (e.g., first alpha map 914 and second alpha map 924) to input frames including foreground image data (e.g., first frame data 912 and second frame data 922). The alpha map management module 620 may not allocate alpha maps to input frames including background image data.

[0122] Figure 9b A pre-multiplied RGB mix is ​​shown according to an embodiment.

[0123] refer to Figure 9b , the frame color mixing module 630 may perform a first pre-multiplied RGB mixing 975 on the first frame data 960 and the second frame data 970. The first frame data 960 may include an RGB frame without alpha information (e.g., 10-bit RGB data per color channel). The second frame data 970 may include an RGB frame 972 (e.g., 10-bit RGB data per color channel) and a first alpha map 974 allocated by the alpha map management module 620. In an embodiment, the first frame data 960 may include an input HDR image (e.g., the input HDR image 605) or one RGB frame including an image of an edited object. In an embodiment, the first frame data 960 may include one RGB frame representing a background image of an edited object, and the alpha map management module 620 may not allocate alpha information to the first frame data 960. In an embodiment, the first frame data 950 or the RGB frame 972 may be configured in an RGBA1010102 format, but a 2-bit alpha channel ("A") may be empty (e.g., "00"), or may include mask information.

[0124] In an embodiment, the first frame data 960 may include an input HDR image and may be generated by the decoder 520 that decodes the original media data (e.g., a video file) and stored in one of the plurality of frame buffers forming a ring buffer. The alpha map management module 620 may not allocate alpha information for the first frame data 960, and the memory usage of the first frame data 960 may be similar to the memory usage in SDR video editing.

[0125] The frame color mixing module 630 may generate the third frame data 980 by performing the first pre-multiplied RGB mixing 975 on the first frame data 960 and the second frame data 970. Due to the pre-multiplied RGB mixing, the third frame data 980 may not include alpha information, and the RGB data of the third frame data 980 may be pre-multiplied by the alpha information (e.g., per-pixel alpha value) of the first alpha map 974 assigned to the second frame data 970 by the alpha map management module 620. In an embodiment, the third frame data 980 or the RGB frame 992 may be configured in an RGBA1010102 format, but the 2-bit alpha channel ("A") may be empty (e.g., "00"), or may include mask information.

[0126] The frame color mixing module 630 may perform a second pre-multiplied RGB mixing 995 on the third frame data 980 and the fourth frame data 990. As described above, the third frame data 980 may be generated by the first pre-multiplied RGB mixing 975 and may not include alpha information. The fourth frame data 990 may include an RGB frame 992 (e.g., 10-bit RGB data per color channel) and a second alpha map 994 allocated by the alpha map management module 620.

[0127] like Figure 9a As shown, when alpha information is needed in subsequent video editing (e.g., second alpha blending 945 or subsequent blending (not shown)), the frame color blending module 630 can perform alpha blending (e.g., first alpha blending 925 or second alpha blending 945) to generate output frame data (e.g., third frame data 930 or fifth frame data 950) including alpha information (e.g., third alpha map 934 or fifth alpha map 954). However, as Figure 9b As shown, when alpha information is not needed in subsequent video editing (e.g., the second pre-multiplied RGB blend 995 or a subsequent blend (not shown)), the frame color blending module 630 can perform pre-multiplied RGB blending (e.g., the first pre-multiplied RGB blend 975 or the second pre-multiplied RGB blend 995) to generate output frame data (e.g., the third frame data 980) that does not include separate alpha information (e.g., pixel data including pre-multiplied alpha values).

[0128] Fig.10 is a flow chart illustrating a video editing process according to an embodiment.

[0129] refer to Fig.10In operation 1005, the electronic device 200 (e.g., the processor 210) may obtain a first image to be edited (e.g., the input HDR image 605). In an embodiment, the first image may be read from a memory (e.g., the memory 230 or an external memory (not shown)), recorded by a camera (not shown), or received from an external electronic device (not shown) to the electronic device 210.

[0130] In operation 1010, the electronic device 200 (e.g., the processor 210) may generate a first input frame (e.g., an SDR image) to be mixed with a first image. In an embodiment, the first input frame may include one or more objects to be edited by an editing function provided for video editing (hereinafter, referred to as editing objects). For example, the one or more editing objects may include a graffiti image (e.g., a brush stroke) or a sticker image. In an embodiment, the first input frame may include 8-bit pixel data per color channel (e.g., pixel data 702).

[0131] In an embodiment, each pixel data of the first input frame may include RGB data and mask information indicating a mixing method of the pixel data, instead of an alpha value. A frame format including mask information instead of an alpha value for each pixel may be referred to as RGBM. In an embodiment, the electronic device 200 (e.g., the processor 210) may generate mask information (e.g., 2-bit mask information 706) for each pixel of the first input frame, and include the mask information in the first input frame when generating the first input frame.

[0132] In an embodiment, the mask information may indicate a blending method based on an editing function (e.g., graffiti or sticker) corresponding to the first input frame, such as at least one of mosaic, alpha blending, or no alpha blending, or may be configured as a value indicating at least one of source-dependent alpha blending, target-dependent alpha blending, or linear blending. In an embodiment, the electronic device 200 may generate 30-bit RGB data and 2-bit mask information for each pixel of the first input frame.

[0133] In an embodiment, when the first input frame including the mask information has an SDR format, the electronic device 200 (e.g., the processor 210) may convert the first input frame into an HDR format. The first input frame converted into the HDR format may include RGB data including 10 bits per color channel (e.g., RGB data 704). The electronic device 200 (e.g., the processor 210) may include the RGB data 704 and the mask information 706 in each pixel data (e.g., pixel data 708) of the first input frame converted into the HDR format.

[0134] In operation 1015, the electronic device 200 (e.g., the processor 210) may allocate an alpha map (e.g., a first alpha map) including a per-pixel alpha value for the first input frame. In an embodiment, the electronic device 200 (e.g., the processor 210) may determine whether alpha blending of the first input frame is required according to an editing function according to a user input. In an embodiment, when an editing object included in the first input frame includes a graffiti image or a sticker image, the electronic device 200 (e.g., the processor 210) may determine that alpha blending of the first input frame is required. In an embodiment, when the first input frame includes a background image, the electronic device 200 (e.g., the processor 210) may determine that alpha blending is required. When it is determined that alpha blending is not required, operation 1015 may be omitted, and the electronic device 200 (e.g., the processor 210) may not allocate the first alpha map. However, when alpha blending is required, the electronic device 200 (e.g., the processor 210) may allocate the first alpha map in operation 1015.

[0135] In an embodiment, the electronic device 200 (eg, the processor 210 ) may identify the assigned first alpha map without modifying each pixel data (eg, RGB data and mask information) of the first input frame.

[0136] In operation 1020, the electronic device 200 (e.g., the processor 210) may determine a blending method to be applied to the first input frame based on the mask information 706 included in each pixel data (e.g., the pixel data 708) of the first input frame. In an embodiment, the electronic device 200 (e.g., the processor 210) may determine a blending method indicating a frame color blending type (e.g., alpha blending, graffiti blending, or sticker blending) based on an analysis of the mask information 706.

[0137] In operation 1025, the electronic device 200 (e.g., the processor 210) may perform frame color mixing on the first input frame based on the analysis of the mask information 706. In the frame color mixing, the electronic device 200 (e.g., the processor 210) may mix the first input frame with the first image or the frame generated in the previous mixing. In an embodiment, the first input frame may be a source frame of the frame color mixing, and the first image or the frame generated in the previous mixing may be a target frame (e.g., the second input frame) of the frame color mixing.

[0138] In an embodiment, the frame color blending may include at least one of pre-multiplied RGB blending (e.g., pre-multiplied RGB blending 975 or 995), alpha blending (e.g., alpha blending 925 or 945), graffiti blending, or sticker blending. In an embodiment, the frame color blending may include alpha blending, and the electronic device 200 (e.g., the processor 210) may store the second alpha map (e.g., alpha map 934 or 954) generated by the alpha blending for use in subsequent video editing. In an embodiment, the electronic device 200 (e.g., the processor 210) may update (e.g., replace or overwrite) the first alpha map allocated in operation 1030 to the first alpha map instead of allocating a new memory area to store the second alpha map.

[0139] In an embodiment, the frame color mixing of operation 1025 may include one or more mixings.

[0140] In operation 1030, the electronic device 200 (eg, the processor 210) may generate a second image (eg, an output HDR image) including an output frame generated by completing frame color mixing. In an embodiment, the second image may include a frame generated by frame color mixing in operation 1025.

[0141] Fig.11 An editing object generated by a pen input function according to an embodiment is shown.

[0142] refer to Fig.11 , the editing module 610 may generate a first stroke 1110 corresponding to the pen input of the user. In an embodiment, each pixel data of the first stroke 1110 may include RGB = (0,0,0) and alpha = 0.4. The editing module 610 may receive a second stroke 1120 from the user. In an embodiment, each pixel data of the second stroke 1120 may include RGB = (0,0,0) and alpha = 0.4.

[0143] In an embodiment, the first stroke 1110 may be input through a normal pen function provided by the editing module 610, and the second stroke 1120 may be input through a mosaic pen function provided by the editing module 610. The editing module 610 may store a third stroke 1130 in which the first stroke 1110 and the second stroke 1120 are combined to reduce memory usage, instead of storing the first stroke 1110 input through the normal pen function and the second stroke 1120 input through the mosaic pen function.

[0144] Fig.12 An editing object image including pixel-based mask information according to an embodiment is shown.

[0145] refer to Fig.12 , the editing module 610 may generate a first stroke 1210 corresponding to the user's pen input through a general pen function. In an embodiment, each pixel data of the first stroke 1210 may include RGB = (0,0,0) and a 2-bit mask = [pen with alpha]. In an embodiment, the editing module 610 may not include alpha information (e.g., alpha = 0.4) in each pixel data of the first stroke 1210. In an embodiment, the editing module 610 may request the alpha map management module 620 to assign an alpha map corresponding to alpha = 0.4. The alpha map management module 620 may assign an alpha map to an input frame including the first stroke 1210 based on the request and / or the 2-bit mask = [pen with alpha], and may report the assigned alpha map to the frame color mixing module 630.

[0146] The editing module 610 may receive the second stroke 1220 from the user through the mosaic pen function. In an embodiment, each pixel data of the second stroke 1220 may include RGB = (0,0,0) and a 2-bit mask = [mosaic]. In an embodiment, the editing module 610 may not include alpha information (e.g., alpha = 0.4) in each pixel data of the second stroke 1210.

[0147] The editing module 610 may store the third stroke 1230 in which the first stroke 1210 and the second stroke 1220 are combined. The editing module 610 may include mask information (eg, [Mosaic], [Pen with Alpha], or [Pen without Alpha]) in each pixel data of the third stroke 1230.

[0148] The mask analysis module 625 may acquire mask information (e.g., [mosaic], [pen with alpha], or [pen without alpha]) included in each pixel data of the third stroke 1230. The mask information may indicate whether each pixel data is from a normal stroke (e.g., the first stroke 1210) or a mosaic stroke (e.g., the second stroke 1220). The mask analysis module 625 may report the acquired 2-bit mask information to the frame color mixing module 630. The frame color mixing module 630 may perform frame color mixing (e.g., scribble mixing) on ​​each pixel data of the third stroke 1330 based on the mask information.

[0149] Fig.13 A brush stroke including a mosaic pen is shown according to an embodiment.

[0150] Fig.13An image 1300 (e.g., an input frame) including at least one pen stroke 1305 is shown. First pixel data 1310 of the image 1300 may include RGB data (e.g., 10 bits per color channel) and mask information (e.g., 2 bits) indicating a mosaic pen (e.g., [mosaic]). Second pixel data 1320 of the image 1300 may include RGB data (e.g., 10 bits per color channel) and mask information (e.g., 2 bits) indicating an alpha-based pen (e.g., [pen with alpha]). Third pixel data 1330 of the image 1300 may include RGB data (e.g., 10 bits per color channel) and mask information (e.g., 2 bits) indicating a non-alpha-based pen (e.g., [pen without alpha]).

[0151] The mask analysis module 625 may acquire mask information included in each pixel data (e.g., the first pixel data 1310, the second pixel data 1320, or the third pixel data 1330) of the input image 1300, and may report the acquired mask information to the frame color mixing module 630. The frame color mixing module 630 may perform mixing (e.g., scribble mixing) on ​​the input image 1300 based on the mask information.

[0152] Fig.14 Sticker blending without using mask information is shown, according to an embodiment.

[0153] refer to Fig.14 , the frame color blending module 630 may blend the sticker image 1410 (e.g., input frame) generated by the editing module 610 on a pixel-by-pixel basis. Each pixel data of the sticker image 1410 may include a 2-bit alpha value GL_SRC_ALPHA 1415 per pixel. Based on the alpha value GL_SRC_ALPHA 1415 of each pixel data of the sticker image 1410, the frame color blending module 630 may determine the alpha information of each pixel data of the target image 1420 (e.g., input HDR image 605) as GL_ONE_MINUS_SRC_ALPHA 1425 (e.g., 1-GL_SRC_ALPHA). The frame color blending module 630 may perform sticker blending (e.g., linear blending, source-dependent blending, or target-dependent blending) on ​​all complete pixel data of the sticker image 1410 and the target image 1420 by using GL_SRC_ALPHA 1415 and GL_ONE_MINUS_SRC_ALPHA 1425 together.

[0154] Fig.15 Sticker blending using mask information is shown, according to an embodiment.

[0155] refer to Fig.15, the frame color blending module 630 may blend the sticker image 1510 generated by the editing module 610 on a pixel-by-pixel basis. Each pixel data of the sticker image 1510 may include 2-bit mask information for distinguishing a blending method for each pixel, instead of alpha information (e.g., a 2-bit alpha value per pixel). For example, the first pixel data 1510a corresponding to the background may include mask information indicating target-dependent alpha blending [Follow target's alpha] 1515. For example, the second pixel data 1510b corresponding to the boundary between the editing object (e.g., sticker) and the background may include mask information indicating linear blending [Linear blending] 1525. For example, the third pixel data 1510c corresponding to the editing object may include mask information indicating source-dependent alpha blending [Follow source's alpha] 1505.

[0156] The mask analysis module 625 may obtain mask information about each pixel data (e.g., pixel data 1510a, 1510b, or 1510c). Based on the mask information, the frame color blending module 630 may determine a blending method (e.g., source-dependent alpha blending, target-dependent alpha blending, or linear blending) for each pixel data of the sticker image 1510. The frame color blending module 630 may perform sticker blending (e.g., source-dependent alpha blending, target-dependent alpha blending, or linear blending) on ​​each pixel data of the sticker image 1510 and the target image 1520 based on the mask information provided from the mask analysis module 625 and the alpha map assigned to each pixel when necessary.

[0157] Fig.16 Frame color blending based on alpha map and mask information is shown according to an embodiment.

[0158] refer to Fig.16 The frame color blending module 630 may include a blending decider 1610, one or more blenders (eg, at least one of a pre-multiplied RGB blender 1620, an alpha blender 1632, a scribble blender 1634, or a sticker blender 1636).

[0159] The blending decider 1610 of the frame color blending module 630 may determine a blending type (e.g., at least one of pre-multiplied RGB blending, alpha blending, scribble blending, or sticker blending) to be applied to the input frame 1602 (e.g., at least one of the input HDR image 605 or the RGB frame including at least one editing object) based on the mask information M sent from the mask analysis module 625 and the alpha map sent from the alpha map management module 620. The blending decider 1610 may forward the input frame 1602 to one of the pre-multiplied RGB mixer 1620, the alpha mixer 1632, the scribble mixer 1634, or the sticker mixer 1636 according to the determined result.

[0160] In an embodiment, at least one input frame 1602 may include each pixel data including mask information, and the blending decider 1610 may determine whether a pixel-based editing function is required based on the mask information sent from the mask analysis module 625. In an embodiment, when the mask information included in the input frame 1602 indicates a pen-related mask (e.g., [mosaic, pen with alpha, or pen without alpha]), the blending decider 1610 may select the graffiti blender 1634. In an embodiment, when the mask information included in the input frame 1602 indicates a sticker-related mask (e.g., [alpha following source, alpha following target, or linear blend]), the blending decider 1610 may select the sticker blender 1636.

[0161] The blending decision 1610 may identify the blending type for blending each pixel data based on the information (e.g., mask information and alpha map) obtained from the mask analysis module 625 and the alpha map management module 620, and may forward each pixel data of the input frame 1602 to a corresponding mixer (e.g., pre-multiplied RGB mixer 1620, alpha mixer 1632, scribble mixer 1634, or sticker mixer 1636). In an embodiment, the blending decision 1610 may forward each pixel data of the input frame 1602 including a scribble image or a sticker image to the scribble mixer 1634 or the sticker mixer 1636, respectively.

[0162] The blending decider 1610 may forward the input frame 1602 including an image other than a graffiti image or a sticker image to the pre-multiplied RGB mixer 1620 or the alpha mixer 1632. In an embodiment, when at least one alpha map assigned to the input frame 1602 is assigned by the alpha map management module 620 and the output of the blending requires additional blending, the blending decider 1610 may forward the input frame 1602 to the alpha mixer 1632. In an embodiment, when the output of the blending requires additional blending, the blending decider 1610 may forward the input frame 1602 to the pre-multiplied RGB mixer 1620.

[0163] When the mixed output (e.g., output RGB frame 1604) no longer requires alpha blending, a pre-multiplied RGB mixer 1620 can be used. The pre-multiplied RGB mixer 1620 can perform blending by pre-multiplying each pixel data of the input frame 1602 with alpha information (e.g., each alpha value of the alpha map assigned by the alpha map management module 620). The mixed output (e.g., frame data 980) from the pre-multiplied RGB mixer 1620 can include only RGB data without alpha information. Using the pre-multiplied RGB mixer 1620 makes it possible to reduce the processing time required for alpha blending (e.g., alpha mixer 1632) and generate RGB data multiplied by alpha information.

[0164] Table 1 below shows the basic logic of pre-multiplied mixing according to an embodiment.

[0165] Table 1

[0166]

[0167] Here, DestinationRGB may represent each pixel data of the blended output generated by the pre-multiplied RGB blender 1620, SourceRGB may represent each pixel data of an input frame (e.g., one of the input frames 1602), and SourceAlpha may represent a per-pixel alpha value assigned to the input frame. glBlendFunc(GL_ONE, GL_ONE) may represent a logical function for the pre-multiplied blend. DestinationAlpha represents the alpha value of the blended output, which may be 1.0.

[0168] Referring to Table 1, the pre-multiplied RGB mixer 1610 may multiply input RGB data (eg, SourceRGB) by alpha information (eg, SourceAlpha) to generate output RGB data (eg, DestinationRGB), and the output alpha information may always be configured to be 1.0. The alpha map management module 620 does not need to obtain alpha information from the pre-multiplied RGB mixer 1610.

[0169] Table 2 below shows the basic logic of alpha blending performed by the alpha blender 1632.

[0170] Table 2

[0171]

[0172] Here, FinalRGB may represent each pixel data of the blended output generated by the alpha blender 1632. FinalAlpha may represent alpha information (e.g., per-pixel alpha value) generated by the alpha blender 1632, 1stInputRGB may represent pixel data of a first input frame in the input frames, and 2ndInputRGB may represent pixel data of a second input frame. In an embodiment, the first input frame may be a source frame, and the second input frame may be a target frame. "alpha" may represent an alpha value input from a user to the editing module 610, and 1stInputAlpha and 2ndInputAlpha may represent per-pixel alpha values ​​obtained from alpha maps assigned to the first input frame and the second input frame, respectively.

[0173] The alpha mixer 1632 may generate alpha information (e.g., FinalAlpha) about an output frame based on an alpha map (e.g., 1stInputAlpha and 2ndInputAlpha) of an input frame. The alpha mixer 1632 may output mixed output RGB data (e.g., FinalRGB) and corresponding alpha information (e.g., FinalAlpha), and the alpha information (e.g., FinalAlpha) may be used for subsequent video editing (e.g., alpha blending).

[0174] Table 3 below shows the basic logic of graffiti mixing according to an embodiment.

[0175] Table 3

[0176]

[0177] Here, inputColor may represent each pixel data of an input frame (e.g., a first input frame in input frame 1602) including a scribble image, and inputAlpha may represent a per-pixel alpha value of an alpha map assigned to the first input frame by the alpha map management module 620. "mask" may represent mask information obtained from the last 2 bits (e.g., inputColor.a) corresponding to alpha information in each pixel data of the first input frame.

[0178] For example, when the "mask" of the first pixel data indicates a mosaic pen (e.g., "MOSAIC"), the graffiti mixer 1634 may perform a drawMosaic function representing a graffiti mixing of a mosaic pen type on backgroundTexture, a pixel data indicating a background image (e.g., a second input frame in the input frame 1602). For example, when the "mask" of the second pixel data indicates an alpha-based pen (e.g., "PEN_WITH_ALPHA"), the graffiti mixer 1634 may perform a drawPen function representing a graffiti mixing based on the second pixel data (e.g., inputRGB) of inputColor and inputAlpha. Here, inputAlpha may represent a per-pixel alpha value of an alpha map assigned by the alpha map management module 620. For example, when the "mask" of the third pixel data indicates a non-alpha-based pen (e.g., "PEN_WITHOUT_ALPHA"), the graffiti mixer 1634 may perform a drawPen function based on the third pixel data (e.g., inputRGB) of inputColor and a specified alpha value (e.g., 1.0).

[0179] Table 4 below shows the basic logic of sticker mixing according to an embodiment.

[0180] Table 4

[0181]

[0182] Here, inputColor may represent each pixel data of an input frame (e.g., a first input frame in input frames 1602) including a sticker image (e.g., a source image), and inputAlpha may represent a per-pixel alpha value of an alpha map assigned to the first input frame by the alpha map management module 620. blendColor may represent each pixel data of a target image (e.g., a second input frame in input frames 1602) to which the sticker image is added, and blendAlpha may represent a per-pixel alpha value of an alpha map assigned to the second input frame by the alpha map management module 620. "mask" may represent mask information obtained from the last 2 bits (e.g., inputColor.a) corresponding to the alpha information in each pixel data of the first input frame.

[0183] For example, when the "mask" of the first pixel data indicates source-dependent alpha blending (e.g., SOURCE_BLEND), the sticker mixer 1636 may calculate the output pixel data (e.g., outColor) based on the first pixel data (e.g., inputColor.rgb), inputAlpha, and the pixel data of the second input frame (e.g., blendColor.rgb) and blendAlpha. The alpha value (outAlpha) of the output pixel data may be determined as inputAlpha. For example, when the "mask" of the second pixel data indicates destination-dependent alpha blending (e.g., DESTINATION_BLEND), the sticker mixer 1636 may calculate the output pixel data (e.g., outColor) based on the second pixel data (e.g., inputColor.rgb), inputAlpha, and the pixel data of the second input frame (e.g., blendColor.rgb) and blendAlpha. The alpha value (outAlpha) of the output pixel data may be determined by blendAlpha. For example, when the "mask" of the third pixel data indicates linear blending (e.g., LINEAR_BLEND), the sticker mixer 1636 can calculate the output pixel data (e.g., outColor) based on the third pixel data (e.g., inputColor.rgb), inputAlpha, and the pixel data of the second input frame (e.g., blendColor.rgb). The alpha value (outAlpha) of the output pixel data can be calculated based on inputAlpha and blendAlpha.

[0184] In an embodiment, the sticker mixer 1636 may select which of the alpha information about the source image (e.g., inputAlpha) or the alpha information about the target image (e.g., blendAlpha) is output as outAlpha based on mask information (e.g., "mask"), or the output alpha information (e.g., outAlpha) may be calculated by applying linear blending. The alpha map management module 620 may replace the alpha information about the source image (e.g., the alpha map assigned to the first input frame) with the output alpha information (e.g., outAlpha).

[0185] Fig.17 is a flowchart illustrating a video editing process using alpha mapping according to an embodiment. According to an embodiment, the processor 210 of the electronic device 200 may perform at least one of the following operations. According to an embodiment, at least one of the following operations may be omitted, modified, or changed.

[0186] refer to Fig.17 , in operation 1705, the electronic device 200 (e.g., the processor 210) may receive an HDR image (e.g., at least one frame). In operation 1710, the electronic device 200 (e.g., the processor 210) may select an editing function. In an embodiment, the editing function may include graffiti and / or stickers. In operation 1715, the electronic device 200 (e.g., the processor 210) may generate an image (e.g., an input frame) including an editing object according to the selected editing function. In an embodiment, the editing object may be generated by an editing function other than graffiti or stickers. In an embodiment, the image including the editing object may include RGB data (e.g., 10-bit RGB data 704 per pixel) and mask information (e.g., mask information 706).

[0187] In operation 1720, the electronic device 200 (e.g., the processor 210) may determine whether to retain the output alpha information after blending. In an embodiment, when the output image needs to be blended or displayed again after blending (e.g., blending 1735 or blending 1760), the electronic device 200 (e.g., the processor 210) may determine to retain the output alpha information. In an embodiment, when video editing is completed and the output image does not need to be stored after blending (e.g., blending 1735 or blending 1760), the electronic device 200 (e.g., the processor 210) may determine that the output alpha information does not need to be retained. When the output alpha information needs to be retained, the electronic device 200 (e.g., the processor 210) may perform operation 1750 to perform alpha blending. When the output alpha information does not need to be retained, the electronic device 200 (e.g., the processor 210) may perform operation 1725 to perform pre-multiplied RGB blending.

[0188] At operation 1750, the electronic device 200 (e.g., the processor 210) may determine whether an alpha map is allocated for alpha blending. In an embodiment, the electronic device 200 (e.g., the processor 210) may determine whether at least one alpha map is stored in a designated memory area (e.g., the first area) for alpha blending. When at least one alpha map is allocated (e.g., stored in the first area), the electronic device 200 (e.g., the processor 210) may perform operation 1760. When no alpha map is allocated (e.g., no alpha map is stored in the first area), the electronic device 200 (e.g., the processor 210) may perform operation 1755.

[0189] In operation 1755, the electronic device 200 (eg, the processor 210) may assign an alpha map to the input frame including the editing object. In operation 1760, the electronic device 200 (eg, the processor 210) may perform alpha blending on the input frame including the editing object and the HDR image based on the per-pixel alpha value of the assigned alpha map.

[0190] At operation 1725, the electronic device 200 (e.g., the processor 210) may determine whether an alpha map for alpha blending is allocated. In an embodiment, the electronic device 200 (e.g., the processor 210) may determine whether at least one alpha map is stored in a designated memory area (e.g., the first area) for alpha blending. When at least one alpha map is allocated (e.g., stored in the first area), the electronic device 200 (e.g., the processor 210) may perform operation 1730. When no alpha map is allocated (e.g., no alpha map is stored in the first area), the electronic device 200 (e.g., the processor 210) may perform operation 1735.

[0191] In operation 1730, the electronic device 200 (eg, the processor 210) may release (eg, delete from the first region) the allocated alpha map. In operation 1735, the electronic device 200 (eg, the processor 210) may perform pre-multiplied RGB mixing on the input frame including the editing object and the HDR image.

[0192] After operation 1760 or operation 1735 , the electronic device 200 (eg, the processor 210 ) may display an output image (eg, an output frame) generated as a result of blending (eg, alpha blending or pre-multiplied RGB blending).

[0193] At operation 1745, the electronic device 200 (e.g., the processor 210) may determine whether the video editing of the HDR image is completed. In an embodiment, the electronic device 200 (e.g., the processor 210) may determine whether the video editing of each frame of the HDR image is completed. When the editing is completed, the electronic device 200 (e.g., the processor 210) may terminate the process. When additional video editing is required, the electronic device 200 (e.g., the processor 210) may return to operation 1710.

[0194] According to the foregoing process, the electronic device 200 (e.g., the processor 210) can repeatedly allocate, reuse, and deallocate the alpha map, thereby maintaining a memory area (e.g., the second area) storing the alpha map. The electronic device 200 (e.g., the processor 210) can retain the alpha map with 8 bits per pixel in the second area, thereby reducing memory usage (e.g., memory read operations) when performing repeated blending.

[0195] Fig.18 is a flowchart illustrating a graffiti blending process using mask information according to an embodiment. According to an embodiment, the processor 210 of the electronic device 200 may perform at least one of the following operations. According to an embodiment, at least one of the following operations may be omitted, modified, or changed.

[0196] refer to Fig.18 In operation 1805, the electronic device 200 (e.g., the processor 210) may receive an HDR image (e.g., at least one frame). In operation 1810, the electronic device 200 (e.g., the processor 210) may generate an image (e.g., an input frame) including a brush stroke (e.g., the brush stroke 1230) based on a user input according to a graffiti function. In an embodiment, each pixel data of the input frame may include RGB data (e.g., 10-bit RGB data 704 per pixel) and mask information (e.g., mask information 706). In operation 1815, the electronic device 200 (e.g., the processor 210) may acquire the pixel data of the input frame.

[0197] At operation 1820, the electronic device 200 (e.g., the processor 210) may acquire mask information (e.g., 2 bits) included in the first pixel data of the input frame. At operation 1825, the electronic device 200 (e.g., the processor 210) may determine whether the mask information indicates a mosaic pen (e.g., [mosaic]), and when the mask information indicates the mosaic pen, operation 1830 may be performed. When the mask information does not indicate the mosaic pen, the electronic device 200 (e.g., the processor 210) may perform operation 1835.

[0198] At operation 1830 , the electronic device 200 (eg, the processor 210 ) may determine that first pixel data of the input frame belongs to a mosaic pen type, and may perform scribble mixing on the first pixel data of the input frame and corresponding pixel data of the HDR image.

[0199] At operation 1835, the electronic device 200 (e.g., the processor 210) may determine whether the mask information indicates an alpha-based pen (e.g., [pen with alpha]), and when the mask information indicates an alpha-based pen, operation 1845 may be performed. When the mask information does not indicate an alpha-based pen (e.g., a non-alpha-based pen [pen without alpha]), the electronic device 200 (e.g., the processor 210) may perform operation 1840.

[0200] At operation 1845 , the electronic device 200 (eg, the processor 210 ) may determine that first pixel data of the input frame belongs to an alpha-based pen type, and may perform scribble blending on the first pixel data of the input frame and corresponding pixel data of the HDR image.

[0201] In operation 1840 , the electronic device 200 (eg, the processor 210 ) may determine that first pixel data of the input frame belongs to a non-alpha-based pen type, and may perform scribble blending on the first pixel data of the input frame and corresponding pixel data of the HDR image.

[0202] The output image and the output alpha information generated as a result of scribble blending after operation 1830, operation 1850, or operation 1840 may be a target image for next video editing.

[0203] At operation 1850, the electronic device 200 (e.g., the processor 210) may determine whether the rendering (e.g., blending) of all pixels of the input frame is completed. When the rendering of all pixels is completed, the electronic device 200 (e.g., the processor 210) may terminate the process. When there are one or more remaining pixels that have not been rendered, the electronic device 200 (e.g., the processor 210) may perform operation 1815.

[0204] Fig.19 is a flowchart illustrating a sticker mixing process using mask information according to an embodiment. According to an embodiment, the processor 210 of the electronic device 200 may perform at least one of the following operations. According to an embodiment, at least one of the following operations may be omitted, modified, or changed.

[0205] refer to Fig.19In operation 1905, the electronic device 200 (e.g., the processor 210) may receive an HDR image (e.g., at least one frame). In operation 1910, the electronic device 200 (e.g., the processor 210) may generate an image (e.g., an input frame) including stickers based on user input according to a sticker function. In an embodiment, each pixel data of the input frame may include RGB data (e.g., 10-bit RGB data 704 per pixel) and mask information (e.g., mask information 706). In operation 1915, the electronic device 200 (e.g., the processor 210) may obtain pixel data of the input frame.

[0206] In operation 1920, the electronic device 200 (e.g., the processor 210) may identify (or acquire) mask information (e.g., 2 bits) included in the first pixel data of the input frame. In operation 1925, the electronic device 200 (e.g., the processor 210) may determine whether the mask information indicates linear mixing, and when the mask information indicates linear mixing, operation 1930 may be performed. When the mask information does not indicate linear mixing, the electronic device 200 (e.g., the processor 210) may perform operation 1950.

[0207] At operation 1930, the electronic device 200 (e.g., the processor 210) may configure a blending strength for the first pixel data of the input frame for linear blending. At operation 1935, the electronic device 200 (e.g., the processor 210) may obtain an alpha value corresponding to the first pixel data from an alpha map assigned to the input frame. At operation 1940, the electronic device 200 (e.g., the processor 210) may perform sticker blending on the first pixel data of the input frame and the corresponding pixel data of the HDR image based on the alpha value.

[0208] In operation 1950, the electronic device 200 (e.g., the processor 210) may determine whether the mask information indicates source-dependent alpha blending (alpha following the source), and when the mask information indicates source-dependent alpha blending, operation 1955 may be performed. In operation 1955, the electronic device 200 (e.g., the processor 210) may acquire an alpha value corresponding to the first pixel data from an alpha map allocated to the input frame so as to perform source-dependent alpha blending. When the mask information does not indicate source-dependent alpha blending (e.g., indicates target-dependent alpha blending (alpha following the target)), the electronic device 200 (e.g., the processor 210) may perform operation 1940 without acquiring alpha information.

[0209] After operation 1935 or operation 1955, the electronic device 200 (e.g., the processor 210) may perform sticker blending on the first pixel data of the input frame and the corresponding pixel data of the HDR image based on the alpha value in operation 1940. Output alpha information generated as a result of sticker blending including source-dependent alpha blending may be determined based on the alpha value corresponding to the first pixel data of the input frame.

[0210] When the mask information does not indicate source-dependent alpha blending in operation 1950, the electronic device 200 (e.g., the processor 210) may perform sticker blending on the first pixel data of the input frame and the corresponding pixel data of the HDR image in operation 1940. Output alpha information generated as a result of sticker blending including target-dependent alpha blending may be determined as an alpha value corresponding to the corresponding pixel data of the HDR image. After operation 1940, the output image generated as a result of sticker blending and the output alpha information may be a target image for the next video editing.

[0211] Embodiments of the present disclosure can provide HDR-based editing functions without significantly changing the module configuration used in an SDR-based editing environment, and can also save the memory capacity required to store alpha information about the HDR image format. In addition, it can prevent the editing function from being restricted due to memory capacity limitations in electronic devices with limited memory.

[0212] Embodiments of the present disclosure may configure each pixel data of an image in a 40-bit format (e.g., 32-bit pixel data 708 and 8-bit alpha value 710) instead of a 64-bit format (e.g., RGBA16F) to provide HDR-based editing functionality, thereby improving the graphics rendering performance of the image frame.

[0213] Embodiments of the present disclosure can provide mask information for supporting pixel-based editing functions by using two free bits of pixel data obtained through allocation of alpha mapping. The provision of mask information can enable appropriate use of memory in a graffiti mixture where multiple strokes such as pen input occur.

[0214] Fig. 20 Memory usage according to an embodiment is shown.

[0215] refer to Fig. 20, the decoder output 2002 (e.g., in YUV format) may be a mixed input frame and may be stored in a ring buffer 2004 including a plurality of (e.g., four) frame buffers. The input frame read from the ring buffer 2004 may be mixed with a hue image (e.g., a hue frame) in a first mixer 2006a. The output frame from the first mixer 2006a may be mixed with a filter image (e.g., a filter frame) in a second mixer 2006b. The third mixer 2006c may mix the output frame from the second mixer 2006b with a doodle image (e.g., a doodle frame). The fourth mixer 2006d may mix the output frame from the third mixer 2006c with a transition image (e.g., a transition frame). The fifth mixer 2006e may mix the output frame from the fourth mixer 2006d with a sticker image (e.g., four sticker frames). The sixth mixer 2006f may mix the output frame from the fifth mixer 2006e with a Ken Burns background image (eg, four Ken Burns background frames). The output from the sixth mixer 2006f may be a final output frame 2008.

[0216] In an embodiment, the electronic device 200 may perform pre-multiplied RGB blending on the input frame read from the ring buffer 2004 and the Ken Burns background frame without assigning an alpha map thereto. In an embodiment, the electronic device 200 may perform alpha blending on the tint frame, filter frame, scribble frame, transition frame, and sticker frame with a target frame (e.g., an input frame read from the ring buffer 2004 or an output frame from a previous blend) by assigning at least one alpha map thereto.

[0217] In an embodiment, the input image may include full high density (FHD) HDR video.

[0218] For example, the required memory usage may correspond to 8 frames for the ring buffer, 1 frame for the tone image, 1 frame for the filter image, 1 frame for the graffiti image, 4 frames for the Ken Burns background image, 1 frame for the transition image, and 4 frames for the sticker image.

[0219] When the RGBA1F format (8 bytes) is used for each frame, the required memory usage may be as follows:

[0220]

[0221] When the RGBM1010102 format including mask information and a 1-byte alpha value based on alpha mapping are used according to an embodiment of the present disclosure, the required memory usage may be as follows:

[0222] = 174MB[182,476,800]

[0223] In the above example, by applying the embodiments of the present disclosure, 45% of memory can be saved.

[0224] Fig.21a and Figure 21b The storage of a graffiti image according to an embodiment is shown.

[0225] refer to Fig.21a , the electronic device 200 may receive a plurality of pen strokes, and the plurality of pen strokes may have different pen types (eg, a pen, a highlighter, or a pencil). When the plurality of pen strokes are stored as separate frames 2102, 2104, and 2106, a memory capacity of three frames may be required.

[0226] refer to Figure 21b , the electronic device 200 can synthesize multiple brush strokes 2102, 2104, and 2106 to store a frame 2108, and can include mask information specifying the pen type (e.g., a mosaic pen type) of each brush stroke in each pixel data of the frame. The electronic device 200 can store multiple brush strokes as a single frame 2108 instead of storing them as separate frames 2102, 2104, and 2106, thereby saving memory usage. In addition, the electronic device 200 can perform blending by distinguishing the pen type of each pixel data of the frame 2108, thereby improving blending performance. As described above, embodiments of the present disclosure can retain the pen type of each pixel in blending while reducing memory usage.

[0227] The electronic device 200 according to the embodiment may include a memory 230 and at least one processor 210. The at least one processor may be configured to acquire a first image. The at least one processor may be configured to generate a first input frame to be mixed with the first image. Each pixel data of the first input frame may include RGB data and mask information indicating a mixing method of the pixel data. The at least one processor may be configured to assign a first alpha map including a per-pixel alpha value of the first input frame to the first input frame. The at least one processor may be configured to determine a mixing method for frame color mixing of the first input frame and the first image based on the mask information. The at least one processor may be configured to perform frame color mixing of the first input frame and the second input frame of the first image based on the determined mixing method and the first alpha map. The at least one processor may be configured to output a second image generated by frame color mixing.

[0228] In an embodiment, the at least one processor may be configured to generate RGB data and a second alpha map based on the first input frame and the second input frame by frame color blending, and store the second alpha map in place of the first alpha map.

[0229] In an embodiment, at least one processor may be configured to, when alpha blending of the first input frame is not required, determine whether there is a third alpha map generated in a previous blending, when the third alpha map exists, de-allocate the third alpha map, and perform pre-multiplied RGB blending on the first input frame and the second input frame.

[0230] In an embodiment, at least one processor may be configured to, when alpha blending of a first input frame is required, determine whether there is a fourth alpha map allocated in a previous blending, assign the fourth alpha map to the first input frame when the fourth alpha map exists, and assign the first alpha map to the first input frame when alpha blending of the first input frame is required and the fourth alpha map does not exist.

[0231] In an embodiment, the at least one processor may be configured to maintain allocation of a memory area storing the first alpha map for reuse in next blending.

[0232] In an embodiment, each pixel data of the first input frame may include 10-bit RGB data and 2-bit mask information per color channel, but does not include alpha information.

[0233] In an embodiment, at least one processor may be configured to recognize that a first input frame includes a stroke image, recognize mask information about first pixel data of the first input frame, perform mosaic pen type scribble blending on the first pixel data and second pixel data of a second input frame when the mask information indicates a mosaic pen type, perform scribble blending on the first pixel data and second pixel data of the second input frame based on an alpha value of an alpha map assigned to the first input data when the mask information indicates an alpha-based pen type, and perform scribble blending on the first pixel data and second pixel data of the second input frame based on a specified alpha value when the mask information indicates a non-alpha-based pen type.

[0234] In an embodiment, at least one processor may be configured to receive a first stroke, receive a second stroke, generate a first input frame including a stroke image by combining the first stroke and the second stroke, and include mask information instead of alpha information in each pixel data of the first input frame.

[0235] In an embodiment, at least one processor may be configured to identify that a first input frame includes a sticker image; identify mask information about first pixel data of the first input frame; when the mask information indicates linear blending, perform linear blending on the first pixel data and second pixel data of the second input frame based on a first alpha value of a first alpha map assigned to the first input data and a second alpha value corresponding to a second alpha map of the second input frame; when the mask information indicates source-dependent alpha blending, perform source-dependent alpha blending on the first pixel data and the second pixel data of the second input frame based on the first alpha value and the second alpha value; and when the mask information indicates target-dependent alpha blending, perform target-dependent alpha blending on the first pixel data and the second pixel data of the second input frame based on the first alpha value and the second alpha value.

[0236] In an embodiment, at least one processor may be configured to receive a sticker image, generate a first input frame including the sticker image, and include mask information instead of alpha information in each pixel data of the first input frame.

[0237] The method of the electronic device 200 for video editing according to the embodiment may include acquiring a first image (1005). The method may include generating a first input frame to be mixed with the first image (1010 and 1015). Each pixel data of the first input frame may include RGB data and mask information indicating a mixing method of the pixel data. The method may include assigning a first alpha map including a per-pixel alpha value of the first input frame to the first input frame (1030). The method may include determining a mixing method for frame color mixing of the first input frame and the first image based on the mask information (1035). The method may include performing frame color mixing of the first input frame and a second input frame of the first image based on the determined mixing method and the first alpha map (1040). The method may include outputting a second image generated by the frame color mixing (1045).

[0238] In an embodiment, performing frame color blending may include generating RGB data and a second alpha map based on the first input frame and the second input frame by frame color blending, and storing the second alpha map in place of the first alpha map.

[0239] In an embodiment, performing frame color blending may include determining whether there is a third alpha map generated in a previous blending when alpha blending of a first input frame is not required; when the third alpha map exists, de-allocating the third alpha map; and performing pre-multiplied RGB blending on the first input frame and the second input frame.

[0240] In an embodiment, the method may further include, when alpha blending of the first input frame is required, determining whether there is a fourth alpha map allocated in the previous blending; and when the fourth alpha map exists, allocating the fourth alpha map to the first input frame. Allocating the first alpha map may include allocating the first alpha map to the first input frame when alpha blending of the first input frame is required and there is no fourth alpha map.

[0241] In an embodiment, the method may further include maintaining allocation of a memory area storing the first alpha map for reuse in next blending.

[0242] In an embodiment, each pixel data of the first input frame may include 10-bit RGB data and 2-bit mask information per color channel, but does not include alpha information.

[0243] In an embodiment, performing frame color blending may include identifying that a first input frame includes a stroke image; when mask information about first pixel data of the first input frame indicates a mosaic pen type, performing mosaic pen type scribble blending on the first pixel data and second pixel data of the second input frame; when the mask information indicates an alpha-based pen type, performing scribble blending on the first pixel data and the second pixel data of the second input frame based on an alpha value of an alpha map assigned to the first input data; and when the mask information indicates a non-alpha-based pen type, performing scribble blending on the first pixel data and the second pixel data of the second input frame based on a specified alpha value.

[0244] In an embodiment, the generation of the first frame may include receiving a first stroke, receiving a second stroke, generating a first input frame including a stroke image by combining the first stroke and the second stroke, and including mask information instead of alpha information in each pixel data of the first input frame.

[0245] In an embodiment, performing frame color blending may include identifying that a first input frame includes a sticker image; identifying mask information about first pixel data of the first input frame; when the mask information indicates linear blending, performing linear blending on the first pixel data and second pixel data of the second input frame based on a first alpha value of a first alpha map assigned to the first input data and a second alpha value corresponding to a second alpha map of the second input frame; when the mask information indicates source-dependent alpha blending, performing source-dependent alpha blending on the first pixel data and the second pixel data of the second input frame based on the first alpha value and the second alpha value; and when the mask information indicates target-dependent alpha blending, performing target-dependent alpha blending on the first pixel data and the second pixel data of the second input frame based on the first alpha value and the second alpha value.

[0246] In an embodiment, the generating of the first input frame may include receiving a sticker image, generating the first input frame including the sticker image, and including mask information instead of alpha information in each pixel data of the first input frame.

[0247] The electronic device according to various embodiments may be one of various types of electronic devices. The electronic device may include, for example, a portable communication device (e.g., a smart phone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a household appliance. According to an embodiment of the present disclosure, the electronic device is not limited to those electronic devices described above.

[0248] It should be understood that the various embodiments of the present disclosure and the terms used therein are not intended to limit the technical features set forth herein to specific embodiments, but include various changes, equivalent forms or alternative forms for the corresponding embodiments. For the description of the accompanying drawings, similar reference numerals may be used to refer to similar or related elements. It will be understood that the nouns in the singular form corresponding to the term may include one or more things unless the relevant context clearly indicates otherwise. As used herein, each of the phrases such as "A or B", "at least one of A and B", "at least one of A or B", "A, B or C", "at least one of A, B and C" and "at least one of A, B or C" may include any one or all possible combinations of the items listed together with the corresponding one of the multiple phrases. As used herein, terms such as "1st" and "2nd" or "first" and "second" may be used to simply distinguish the corresponding component from another component, and do not limit the component in other aspects (e.g., importance or order). It will be understood that if an element (e.g., a first element) is referred to as being “coupled with another element (e.g., the second element)”, “coupled to another element (e.g., the second element)”, “connected with another element (e.g., the second element)”, or “connected to another element (e.g., the second element)” with or without the terms “operably” or “communicatively” being used, it means that the element may be directly (e.g., wired) connected to the other element, wirelessly connected to the other element, or connected to the other element via a third element.

[0249] As used in connection with various embodiments of the present disclosure, the term "module" may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with other terms (e.g., "logic," "logic block," "portion," or "circuit"). A module may be a single integrated component adapted to perform one or more functions or a minimum unit or portion of the single integrated component. For example, according to an embodiment, a module may be implemented in the form of an application specific integrated circuit (ASIC).

[0250] The various embodiments described herein may be implemented as software (e.g., program 140) including one or more instructions stored in a storage medium (e.g., memory 390, internal memory 136, or external memory 138) that can be read by a machine (e.g., electronic device 202, 204, or 101). For example, under the control of a processor, a processor (e.g., processor 310 or 120) of the machine (e.g., electronic device 202, 204, or 101) may call at least one of the one or more instructions stored in the storage medium and execute the at least one instruction with or without the use of one or more other components. This enables the machine to operate to perform at least one function according to the at least one instruction called. The one or more instructions may include code generated by a compiler or code that can be executed by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Among them, the term "non-transitory" only means that the storage medium is a tangible device and does not include signals (e.g., electromagnetic waves), but the term does not distinguish between data being semi-permanently stored in the storage medium and data being temporarily stored in the storage medium.

[0251] According to an embodiment, the method according to various embodiments of the present disclosure may be included and provided in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be released in the form of a machine-readable storage medium (e.g., a compact disk read-only memory (CD-ROM)), or may be downloaded via an application store (e.g., Play Store TM ) The computer program product may be published (e.g., downloaded or uploaded) online, or the computer program product may be distributed (e.g., downloaded or uploaded) directly between two user devices (e.g., smart phones). If published online, at least part of the computer program product may be temporarily generated, or at least part of the computer program product may be at least temporarily stored in a machine-readable storage medium (such as a memory of a manufacturer's server, a server of an application store, or a forwarding server).

[0252] According to various embodiments, each of the above-mentioned components (e.g., a module or a program) may include a single entity or multiple entities, and some of the multiple entities may be separately arranged in different components. According to various embodiments, one or more of the above-mentioned components may be omitted, or one or more other components may be added. Alternatively or additionally, multiple components (e.g., a module or a program) may be integrated into a single component. In this case, according to various embodiments, the integrated component may still perform the one or more functions of each of the multiple components in the same or similar manner as a corresponding one of the multiple components performing one or more functions before integration. According to various embodiments, the operations performed by a module, a program or another component may be performed sequentially, in parallel, repeatedly or in a heuristic manner, or one or more of the operations may be run or omitted in a different order, or one or more other operations may be added.

Claims

1. An electronic device (200) for video editing, the electronic device comprising: Memory (230); and at least one processor (210), connected to the memory, Wherein, the at least one processor is configured to: acquiring a first image; generating a first input frame to be mixed with the first image, each pixel data of the first input frame including red, green and blue RGB data and mask information indicating a mixing method for the pixel data; assigning a first alpha map comprising per-pixel alpha values ​​for a first input frame to the first input frame; determining the mixing method for frame color mixing of the first input frame and the first image based on the mask information; performing frame color blending of the first input frame and a second input frame including the first image based on the determined blending method and the first alpha map; and A second image generated by mixing the frame colors is output.

2. The electronic device according to claim 1, wherein: The at least one processor is configured to: generating RGB data and a second alpha map based on the first input frame and the second input frame by the frame color blending; and The second alpha map is stored in place of the first alpha map.

3. The electronic device according to claim 1 or 2, wherein: The at least one processor is configured to: When alpha blending of the first input frame is not required, performing a check whether there is a third alpha map generated in a previous blending; when the third alpha map exists, deallocating the third alpha map; as well as A pre-multiplied RGB blend is performed on the first input frame and the second input frame.

4. The electronic device according to any one of claims 1 to 3, wherein: The at least one processor is configured to: when the alpha blending of the first input frame is required, determining whether there is a fourth alpha map allocated in a previous blending; assigning the fourth alpha map to the first input frame when the fourth alpha map is present; as well as When the alpha blending of the first input frame is required and a fourth alpha map is not present, the first alpha map is assigned to the first input frame.

5. The electronic device according to any one of claims 1 to 4, wherein: The at least one processor is configured to maintain allocation of a memory area storing the first alpha map for reuse in next blending.

6. The electronic device according to any one of claims 1 to 5, wherein: Each pixel data of the first input frame includes 10 bits of the RGB data and 2 bits of the mask information per color channel, but does not include alpha information.

7. The electronic device according to any one of claims 1 to 6, wherein: The at least one processor is configured to: identifying that the first input frame includes a stroke image; identifying the mask information regarding first pixel data of the first input frame; When the mask information indicates a mosaic pen type, performing mosaic pen type graffiti mixing on the first pixel data and the second pixel data of the second input frame; When the mask information indicates an alpha-based pen type, performing scribble blending on the first pixel data and the second pixel data of the second input frame based on an alpha value of an alpha map assigned to the first input data; as well as When the mask information indicates a non-alpha-based pen type, scribble blending is performed on the first pixel data and the second pixel data of the second input frame based on a designated alpha value.

8. The electronic device according to claim 7, wherein: The at least one processor is configured to: Receive the first stroke; Receive the second stroke; generating the first input frame including the stroke image by combining the first stroke and the second stroke; and The mask information is included in each pixel data of the first input frame instead of alpha information.

9. The electronic device according to any one of claims 1 to 7, wherein: The at least one processor is configured to: identifying that the first input frame includes a sticker image; identifying mask information about first pixel data of the first input frame; When the mask information indicates linear blending, performing the linear blending on the first pixel data and the second pixel data of the second input frame based on a first alpha value of a first alpha map assigned to the first input data and a second alpha value of a second alpha map corresponding to the second input frame; When the mask information indicates source-dependent alpha blending, performing the source-dependent alpha blending on the first pixel data and the second pixel data of the second input frame based on the first alpha value and the second alpha value; as well as When the mask information indicates target-dependent alpha blending, the target-dependent alpha blending is performed on the first pixel data and the second pixel data of the second input frame based on the first alpha value and the second alpha value.

10. The electronic device according to claim 9, wherein: The at least one processor is configured to: receiving the sticker image; generating the first input frame including the sticker image; and The mask information is included in each pixel data of the first input frame instead of alpha information.

11. A method of an electronic device (200) for video editing, the method comprising: Get the first image (1005); Generate a first input frame to be mixed with the first image (1010), each pixel data of the first input frame includes red, green and blue RGB data and mask information indicating a mixing method for the pixel data; assigning a first alpha map comprising per-pixel alpha values ​​for a first input frame to the first input frame (1015); determining the mixing method for frame color mixing of the first input frame and the first image based on the mask information (1020); performing frame color blending of the first input frame and a second input frame including the first image based on the determined blending method and the first alpha map (1025); and A second image generated by the frame color mixing is output (1030).

12. The method according to claim 11, wherein: The performing frame color mixing comprises: generating RGB data and a second alpha map based on the first input frame and the second input frame by the frame color blending; and The second alpha map is stored in place of the first alpha map.

13. The method according to claim 11 or 12, wherein: The performing frame color mixing further comprises: When alpha blending of the first input frame is not required, performing a check whether there is a third alpha map generated in a previous blending; when the third alpha map exists, deallocating the third alpha map; performing a pre-multiplied RGB blend on the first input frame and the second input frame; When the alpha blending of the first input frame is required, determining whether there is a fourth alpha map allocated in a previous blending; and assigning the fourth alpha map to the first input frame when the fourth alpha map is present, and The assigning of the first alpha map includes assigning the first alpha map to the first input frame when the alpha blending of the first input frame is required and a fourth alpha map does not exist.

14. The method according to any one of claims 11 to 13, further comprising: The memory area storing the first alpha map is kept allocated for reuse in the next blending.

15. The method according to any one of claims 11 to 14, wherein: Each pixel data of the first input frame includes 10 bits of the RGB data and 2 bits of the mask information per color channel, but does not include alpha information.

Images