Using special effects for HDR video

Abstract

A computing system for using a special effect for a high dynamic range (HDR) video by receiving a video frame of the HDR video, the video frame comprising a plurality of pixels, each color component of each pixel having one or more luminance values, an electro-optical transfer function, a tone mapping algorithm, and an electro-optical transfer function are sequentially applied to each pixel in a video frame of an HDR video. Subsequently, video frames of a standard dynamic range (SDR) video having transformed luminance values for each color component are generated. Then, one or more special effects are used for video frames of the SDR video, thereby generating an edited SDR video, and an output is generated based on the edited SDR video.

Description

Background Technology

[0001] Video technology continues to evolve, especially with the advent of High Dynamic Range (HDR) video, which has significantly improved visual quality. Compared to Standard Dynamic Range (SDR) video, HDR video offers a wider range of brightness and color depth. For example, HDR is characterized by brighter whites, deeper blacks, and a potentially larger number of visible colors, resulting in more vivid and realistic images. This increased color depth and expanded dynamic range give HDR a significant edge in providing a more immersive visual experience, especially compared to SDR video, which operates within a more limited color gamut and a narrower range of brightness levels.

[0002] With the increasing prevalence of HDR-enabled cameras and displays, especially in the mobile device sector, modern smartphones, tablets, and cameras now universally support HDR video recording and playback, bringing a professional-grade viewing experience to the consumer market.

[0003] However, the surge in HDR video content has also presented challenges when attempting to apply effects designed specifically for SDR video. Applications and software originally developed for SDR video are often not optimized for the increased color depth and dynamic range of HDR content. When such applications are used with HDR video, the mismatch between the two formats can lead to severe color distortion. This distortion may manifest as oversaturation or desaturation in certain color spaces, resulting in an unnatural or undesirable video appearance. Summary of the Invention

[0004] In view of the above, the present invention provides a computational system for applying special effects to high dynamic range (HDR) video. The computational system includes processing circuitry and a memory storing instructions, which, when executed, cause the processing circuitry to: receive a video frame of HDR video, the video frame comprising a plurality of pixels, each pixel having one or more luminance values ​​for each of a plurality of color components; and sequentially apply an HDR-to-SDR pipeline, including an electro-optical transfer function, a tone mapping algorithm, and an opto-transfer function, to each pixel in the HDR video frame. Subsequently, a video frame of standard dynamic range (SDR) video with transformed luminance values ​​for each of the plurality of color components is generated, and then one or more special effects are applied to the SDR video frame to generate an edited SDR video, and an output is generated based on the edited SDR video.

[0005] This summary aims to introduce some concepts in a simplified form, which will be further described in the detailed embodiments below. This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter. Furthermore, the claimed subject matter is not limited to the implementation of solutions to any or all the shortcomings mentioned in any part of this disclosure. Attached Figure Description

[0006] Figure 1 A schematic diagram of a computing system according to an example of this disclosure is shown.

[0007] Figure 2 It shows that it can be made by Figure 1 An example of how the effects module in a computing system is applied to video effects.

[0008] Figure 3 It shows Figure 1 A detailed schematic diagram of the tone mapping algorithm operation in the computing system.

[0009] Figure 4 This is a flowchart of a method for applying special effects to HDR video according to an example embodiment of the present disclosure.

[0010] Figure 5 This is a flowchart of a tone mapping method according to an exemplary embodiment of the present disclosure.

[0011] Figure 6 The example computing environment of this disclosure is shown. Detailed Implementation

[0012] Figure 1 A schematic diagram of an example computing system 100 for applying special effects to high dynamic range (HDR) video 110 is shown. The example computing system 100 can be implemented using various types of computing devices, including mobile devices, smartphones, personal computers, laptops, computing servers, etc. The example computing system 100 includes processing circuitry 102 and a memory 104. The memory 104 stores instructions that, during execution, cause the processing circuitry 102 to perform various processes described herein to generate video frames 132 of edited standard dynamic range (SDR) video 130 and / or video frames 146 of edited HDR video 144 with added special effects. These video frames can be output to be drawn on a display 148 and / or encoded by a video encoder 154 to generate and output encoded video 156. The display 148 can be a display device within the computing system 100 or an external device communicatively coupled to the computing system 100.

[0013] The computing system 100 also includes a camera 106 configured to acquire HDR video 110 and then transfer it to memory 104 for further processing via an HDR-to-SDR pipeline 114. The video data of the HDR video 110 may be initially processed by an image signal processor before being transferred to memory 104. Alternatively, the video data of the HDR video 110 may also be transferred directly to memory 104 in real time via a high-speed communication interface such as, for example, Universal Serial Bus (USB), Thunderbolt, or High Definition Multimedia Interface (HDMI). The high-speed communication interface may be wireless, for example, via Wi-Fi, Bluetooth, Wireless HDMI, or cellular networks.

[0014] Alternatively, HDR video 110 can be imported from various external sources via video importer 108 and then transferred to memory 104. For example, video importer 108 can be implemented as acquisition hardware configured to capture HDR video 110 from an external camera and transfer the video data to memory 104.

[0015] The HDR to SDR pipeline 114 processes video frames 112 of HDR video 110 to generate video frames 126 of SDR video 124. Each video frame 112 includes multiple pixels, each pixel having one or more luminance values ​​for each color component. For example, the input HDR video 110 may have a wide color gamut Rec.2020 color space with a color depth of 10 bits. The output SDR video 124 may have a narrow color gamut Rec.709 color space with a color depth of 8 bits. Each pixel of each video frame 112 is processed by multiple functions 116, 118, 120, and 122 of the HDR to SDR pipeline 114 to generate video frames 126 of SDR video 124.

[0016] In this example, the electro-optical transfer function 116, the photo-optical transfer function 118, the tone mapping algorithm 120, and the photoelectric transfer function 122 are sequentially applied to each pixel in video frame 112 of HDR video 110 to generate video frame 126 of SDR video 124 with transformed brightness values ​​for each of the multiple color components. In some embodiments, the photo-optical transfer function 118 can be omitted, so that only the electro-optical transfer function 116, the tone mapping algorithm 120, and the photoelectric transfer function 122 are applied to each pixel in video frame 112 of HDR video 110. The functions 116, 118, 120, and 122 of the HDR to SDR pipeline 114 can be applied to each pixel in video frame 112 in real time as video frame 112 is received from camera 106, so that video frame 126 of the generated SDR video 124 is output for rendering on display 148 without perceptible delay.

[0017] Electro-optic transfer function 116 specifies how the luminance level encoded in each pixel of video frame 112 of HDR video 110 is mapped to the actual light intensity on target display 148. For example, a mixed log-gamma (HLG) transfer function or a perceptual quantizer (PQ) can be selected as electro-optic transfer function 116. Optical transfer function 118 specifies the mathematical relationship between real-world luminance or scene-dependent light and display-dependent light or luminance presented on display 148.

[0018] The tone mapping algorithm 120 compresses the range of brightness values ​​in video frames 112 of the HDR video 110 to conform to the limitations of the display 148, while preserving the visual details and contrast of the original HDR video 110 to the greatest extent possible within the physical limitations of the display 148. An operational example of the tone mapping algorithm 120 will be provided in [reference needed]. Figure 3 To provide a more detailed description.

[0019] The photoelectric transfer function 122 further applies a non-linear curve to compress the brightness value of each pixel in each video frame 112 while preserving details in shadows. Therefore, both the brightest and darkest parts of each video frame 112 retain visual detail.

[0020] After each pixel in each video frame 112 of the HDR video 110 is processed by the HDR to SDR pipeline 114, video frames 126 of the SDR video 124 are generated and input into the effects module 128. The effects module 128 is configured to apply effects to at least a portion of the video frames 126 of the SDR video 124, recombine the modified video frames into an edited SDR video 130, and generate an edited SDR video 130 in which at least some video frames 132 include the effects applied by the effects module 128. These effects can range from relatively simple effects such as color correction, cropping, and trimming to more complex effects such as filters, stickers, text, motion tracking, audio modification, or transitions. (Reference) Figure 2 An example of a visual effect that can be applied to video frame 126 of SDR video 124 is described.

[0021] The edited SDR video 130 is then used to generate an output. For example, the edited SDR video 130 can be output to be drawn on a display 148, encoded by a video encoder 154 to generate and output a formatted encoded video 156 for storage or sharing, and / or further processed in the SDR to HDR pipeline 134 to generate an edited HDR video 144. The SDR to HDR pipeline 134 includes inverse versions of functions 116, 118, 120, and 122 of the HDR to SDR pipeline 114. In this example, the inverse photoelectric transfer function 136, the inverse tone mapping algorithm 138, the inverse photoelectric transfer function 140, and the inverse electro-optical transfer function 142 in the SDR to HDR pipeline 134 are the inverses of the photoelectric transfer function 122, the tone mapping algorithm 120, the photoelectric transfer function 118, and the electro-optical transfer function 116 of the HDR to SDR pipeline 114, respectively. Therefore, when the optical transfer function 118 is omitted in the HDR to SDR pipeline 114, the backlight optical transfer function 140 is also omitted in the SDR to HDR pipeline 134.

[0022] The inverse functions 136, 138, 140, and 142 of the SDR-to-HDR pipeline 134 are applied to each pixel in video frame 146 of the edited SDR video 144 to generate video frame 146 of the edited HDR video 144, which includes added effects applied by the effects module 128. The edited HDR video 144 can be output for rendering on display 148, and / or encoded by video encoder 154 to generate and output encoded video 156 formatted for storage or sharing.

[0023] The computing system 100 can also execute a preview generator 150, which is configured to generate a preview 152 for drawing on a display 148 based on the edited HDR video 144 and / or the edited SDR video 130 before the user-authorized video encoder 154 generates the encoded video 156 for sharing. For example, the preview 152 can be drawn on the display 148 based on the edited SDR video 130, then user input can be received to generate the edited HDR video 144, and in response to receiving user input to generate the edited HDR video 144, the SDR to HDR pipeline 134 can be applied to generate video frames 146 of the edited HDR video 144.

[0024] refer to Figure 2This describes an example of visual effects that effects module 128 can apply to video frames of SDR video 124. In this example, the visual effect applied to SDR video 124 is a motion tracking effect, which applies a microphone sticker next to the user's face each time the user speaks. The user commands the chat application "Add a microphone speaker next to my face whenever I speak." In response, the chat application causes effects module 128 to apply the microphone sticker next to the user's face in the original HDR video.

[0025] refer to Figure 3 It further describes in detail Figure 1 An example of operation 200 of tone mapping algorithm 120 in computing system 100. In this description, operation 200 is performed on... Figure 1 and Figure 2 The given pixel 202 in video frame 112 of the original HDR video 110 is transformed. The given pixel 202 has three color components transformed by tone mapping algorithm 120: red component 202a, green component 202b, and blue component 202c. The given pixel 202 is first processed by maximum brightness determination module 204 to determine the maximum brightness value among the color components 202a, 202b, and 202c of the given pixel 202. Perceived brightness determination module 206 uses the weights of the Rec.709 standard for high-definition television (HDTV) or the Rec.2020 standard for ultra-high-definition television (UHDTV) to determine the perceived brightness value of the given pixel 202. Maximum brightness adjustment module 208 adjusts the maximum brightness value of the given pixel 202 based on the perceived brightness value.

[0026] The tone mapping adjustment module 210 determines an adjustment factor for a given pixel 202 based on the adjusted maximum luminance value and the peak luminance of the display 148. The adjustment factor can be determined using a tone mapping function. Examples of tone mapping functions that can be used to determine the adjustment factor include linear functions, logarithmic functions, exponential functions, Reinhard's formula, and cinematic tone mapping operators, such as the Hable tone mapping operator or the Academy Color Coding System (ACES). Cinematic tone mapping operators are configured to compress the luminance value of each pixel while maintaining a "film-like" or "cinematic" aesthetic, mimicking how traditional film processes light and color. The color component scaling module 212 scales each color component 202a, 202b, 202c of the given pixel 202 based on the adjustment factor to generate an adjusted pixel 214 with a scaled red component 214a, a scaled green component 214b, and a scaled blue component 214c. For example, the adjusted pixel 214 can be further processed in a subsequent downstream processing module by applying a photoelectric transfer function.

[0027] Figure 4A flowchart of an example method 300 for applying special effects to HDR video is shown. Example method 300 includes receiving HDR video in step 302. HDR video can be received from various sources in a variety of ways, such as from a camera configured to capture HDR video or a video importer configured to import HDR video from various external sources. Example method 300 includes processing the received HDR video using an HDR-to-SDR pipeline in step 304 to generate SDR video based on the received HDR video.

[0028] Step 304 includes step 306 of applying the electro-optical transfer function to the HDR video. For example, the HLG transfer function or PQ can be selected as the electro-optical transfer function.

[0029] Step 304 may optionally include: in step 308, applying the optical transfer function to the HDR video. The optical transfer function specifies a mathematical relationship between real-world brightness or scene reference light and display reference light or brightness presented on the display.

[0030] Step 304 includes: In step 310, applying a tone mapping algorithm to the HDR video. The tone mapping algorithm compresses the range of luminance values ​​in the video frames of the SDR video to conform to the display limitations of the monitor, while preserving the visual details and contrast of the original HDR video to the greatest extent possible within the physical limitations of the monitor.

[0031] Step 304 includes applying the phototransfer function to the HDR video as in step 312 to generate SDR video. The phototransfer function further applies a non-linear curve to compress the luminance value of each pixel in each video frame while preserving details in shadows.

[0032] Example method 300 includes applying effects to SDR video in step 314. Step 314 includes applying effects to at least a portion of the video frames of the SDR video, reassembling the video frames into an edited SDR video, and generating an edited SDR video having at least a portion of the video frames including the applied effects.

[0033] After applying effects to the SDR video in step 312, in step 328, example method 300 may include generating output based on the edited SDR video. The edited SDR video may be output for drawing on a display, and / or encoded by a video encoder to generate and output encoded video that is formatted for storage or sharing.

[0034] Alternatively, after applying the effect to the SDR video in step 314, example method 300 may include step 316, which involves processing the edited SDR video using an SDR-to-HDR pipeline to generate an edited HDR video based on the edited SDR video. Step 316 includes applying the inverse phototransfer function to the edited SDR video in step 318. The inverse phototransfer function is the inverse of the phototransfer function in step 312.

[0035] Step 316 further includes applying an inverse tone mapping algorithm to the edited SDR video in step 320. The inverse tone mapping algorithm is the inverse of the tone mapping algorithm in step 310. Step 316 includes applying an inverse electro-optical transfer function to the edited SDR video in step 324. The inverse electro-optical transfer function is the inverse of the electro-optical transfer function in step 306. Between steps 320 and 324, step 316 may include applying an inverse optical transfer function to the edited SDR video in step 322. The inverse optical transfer function is the inverse of the optical transfer function in step 308.

[0036] Following the SDR-to-HDR pipeline processing in step 316, example method 300 may include step 326, which outputs the edited HDR video. The edited HDR video may be output for rendering on a display, and / or encoded by a video encoder to generate and output encoded video that is formatted for storage or sharing.

[0037] Figure 5 A flowchart of an example method 400 for applying a tone mapping algorithm to a video frame of a received HDR video is shown. Example method 400 includes a step 402 of receiving a given pixel as input to perform tone mapping. The given pixel in the video frame of the original HDR video has three color components that are transformed by the tone mapping algorithm: a red component, a green component, and a blue component.

[0038] Example method 400 also includes a step 404 for determining the maximum luminance value among the red, green, and blue components of a given pixel. Example method 400 also includes a step 406 for determining the perceived luminance value of a given pixel. The perceived luminance value of a given pixel can be determined using weights from the Rec.709 standard for HDTV or the Rec.2020 standard for UHDTV.

[0039] Example method 400 also includes a step 408 for adjusting the maximum brightness value of a given pixel based on perceived brightness. Example method 400 further includes a step 410 for determining an adjustment factor for a given pixel based on the adjusted maximum brightness value and the peak brightness of the display. The adjustment factor can be determined using a tone mapping function. Examples of applicable tone mapping functions include linear functions, logarithmic functions, exponential functions, Reinhard's formula, and cinematic tone mapping operators.

[0040] Example method 400 also includes step 412 of scaling each color component of a given pixel based on an adjustment factor to generate an adjusted pixel having scaled red, scaled green, and scaled blue components. Example method 400 also includes step 414 of outputting the adjusted pixel, which may be further processed, for example, by applying a phototransfer function in a downstream processing step.

[0041] As described in this article, by converting HDR video to SDR and then applying effects, users can retain the high-quality effects originally designed for SDR content while ensuring compatibility with the wider color and dynamic range associated with HDR video. This approach allows applications and software originally developed for SDR video processing to be effectively used with HDR content without introducing visual artifacts or distortions, such as oversaturation or desaturation in certain color spaces. Therefore, the system enhances the viewing experience of released HDR videos by adding effects while maintaining the integrity of the original HDR video's visual quality. Furthermore, the quality of videos created with HDR-enabled cameras can be improved to meet high standards of visual fidelity and consistency across various devices and platforms.

[0042] In some embodiments, the methods and processes described herein can be attached to a computing system of one or more computing devices. Specifically, such methods and processes can be implemented as computer applications or services, application interfaces (APIs), libraries, and / or other computer program products.

[0043] Figure 6 A non-limiting embodiment of a computing system 500 capable of performing one or more of the methods and processes described above is schematically illustrated. The computing system 500 is shown in a simplified form. The computing system 500 can embody the above-described... Figure 1 The computing system 100 shown. Components of the computing system 500 may be included in one or more personal computers, server computers, tablet computers, home entertainment computers, network computing devices, video game devices, mobile computing devices, mobile communication devices (e.g., smartphones) and / or other computing devices, as well as wearable computing devices (e.g., smartwatches and head-mounted augmented reality devices).

[0044] The computing system 500 includes processing circuitry 502, volatile memory 504, and non-volatile storage device 506. The computing system 500 may optionally include a display subsystem 508, an input subsystem 510, a communication subsystem 512, and / or... Figure 6Other components not shown.

[0045] Processing circuitry typically includes one or more logic processors, which are physical devices configured to execute instructions. For example, a logic processor may be configured to execute instructions that are part of one or more applications, programs, routines, libraries, objects, components, data structures, or other logical structures. These instructions can be used to perform tasks, implement data types, transition the state of one or more components, implement technical effects, or otherwise obtain desired results.

[0046] The logic processor may include one or more physical processors configured to execute software instructions. Additionally or alternatively, the logic processor may include one or more hardware logic circuits or firmware devices configured to execute hardware-implemented logic or firmware instructions. The processor of processing circuit 502 may be single-core or multi-core, and the instructions executed thereon may be configured for sequential, parallel, and / or distributed processing. The various components of the processing circuit may optionally be distributed across two or more separate devices, which may be located remotely and / or configured for coordinated processing. For example, aspects of the computing system disclosed herein may be virtualized and executed by remotely accessible networked computing devices configured in a cloud computing configuration. In this case, it should be understood that these virtualized aspects will run on different physical logic processors on various different machines. These different physical logic processors on different machines will be understood as being collectively referred to as processing circuit 502.

[0047] The non-volatile storage device 506 includes one or more physical devices configured to store instructions executable by processing circuitry to implement the methods and processes described herein. During the execution of such methods and processes, the state of the non-volatile storage device 506 may change, for example, storing different data.

[0048] Non-volatile storage device 506 may include removable and / or built-in physical devices. Non-volatile storage device 506 may include optical memory, semiconductor memory, and / or magnetic memory, or other high-capacity storage device technologies. Non-volatile storage device 506 may include non-volatile, dynamic, static, read / write, read-only, sequential access, location-addressed, file-addressed, and / or content-addressed devices. It should be understood that non-volatile storage device 506 is configured to retain instructions even when power to non-volatile storage device 506 is cut off.

[0049] Volatile memory 504 may include physical devices, including random access memory. Processing circuitry 502 typically uses volatile memory 504 to temporarily store information during software instruction processing. It should be understood that when volatile memory 504 is powered off, it typically will not continue storing instructions.

[0050] Various aspects of the processing circuitry 502, the volatile memory 504, and the non-volatile storage device 506 can be integrated into one or more hardware logic components. For example, these hardware logic components may include field-programmable gate arrays (FPGAs), programmable and application-specific integrated circuits (PASICs / ASICs), programmable and application-specific standard products (PSSPs / ASSPs), system-on-a-chip (SoCs), and complex programmable logic devices (CPLDs).

[0051] The terms "module," "program," and "engine" can be used to describe an aspect of computing system 500, typically implemented in software by a processor, to perform a specific function using a portion of volatile memory. This function involves transformative processing specifically configured for the processor to perform that function. Therefore, a module, program, or engine can be instantiated by processing circuitry 502 using a portion of volatile memory 504 to execute instructions stored in non-volatile storage device 506. It is understood that different modules, programs, and / or engines can be instantiated from the same application, service, code block, object, library, routine, API, function, etc. Similarly, the same module, program, and / or engine can be instantiated from different applications, services, code blocks, objects, routines, APIs, functions, etc. The terms "module," "program," and "engine" can encompass single or grouped executable files, data files, libraries, drivers, scripts, database records, etc.

[0052] When a display subsystem 508 is included, it can be used to present a visual representation of the data stored in the non-volatile storage device 506. The visual representation may take the form of a graphical user interface (GUI). Because the methods and processes described herein change the data stored in the non-volatile storage device, thereby changing the state of the non-volatile storage device, the state of the display subsystem 508 may also change accordingly to visually represent the changes in the underlying data. The display subsystem 508 may include one or more display devices employing virtually any type of technology. Such display devices may be combined with the processing circuitry 502, the volatile memory 504, and / or the non-volatile storage device 506 in a shared housing, or such display devices may be peripheral display devices.

[0053] When the input subsystem 510 is included, the input subsystem 510 may include one or more user input devices, such as a keyboard, mouse, touch screen, camera or microphone, or interface with such devices.

[0054] When a communication subsystem 512 is included, the communication subsystem 512 can be configured to communicatively couple the various computing devices described herein to each other and to other devices. The communication subsystem 512 may include wired and / or wireless communication devices compatible with one or more different communication protocols. As a non-limiting example, the communication subsystem may be configured to communicate via wired or wireless local area networks or wide area networks, broadband cellular networks, etc. In some embodiments, the communication subsystem may allow the computing system 500 to send messages to and / or receive messages from other devices via a network such as the Internet.

[0055] The following paragraphs provide a supplementary description of the subject matter of this disclosure. In one aspect, a computational system is provided for applying special effects to high dynamic range (HDR) video. This computational system includes processing circuitry and a memory storing instructions that, when executed, cause the processing circuitry to perform the following operations: receiving a video frame of HDR video, the video frame comprising a plurality of pixels, each pixel having one or more luminance values ​​for each of a plurality of color components; sequentially applying an HDR-to-SDR pipeline, including an electro-optical transfer function, a tone mapping algorithm, and a photoelectric transfer function, to each pixel of the HDR video video frame, thereby generating a video frame of standard dynamic range (SDR) video having transformed luminance values ​​for each of the plurality of color components; applying one or more special effects to the video frame of the SDR video, thereby generating an edited SDR video; and generating an output based on the edited SDR video. In this aspect, additionally or alternatively, in the HDR-to-SDR pipeline, the photoelectric transfer function may be further applied to each pixel of the HDR video video frame so that the electro-optical transfer function, the photoelectric transfer function, the tone mapping algorithm, and the photoelectric transfer function are applied to each pixel of the video frame in sequence. In this regard, additionally or alternatively, a tone mapping algorithm may be performed to determine the maximum luminance value in the color components of a given pixel; determine the perceived luminance value of the given pixel; and adjust the maximum luminance value of the given pixel based on the perceived luminance value. In this regard, additionally or alternatively, the perceived luminance value of the given pixel may be determined using the weights of the Rec.709 or Rec.2020 standards for the color components of the given pixel. In this regard, additionally or alternatively, the tone mapping algorithm may be further performed to determine an adjustment factor for the given pixel based on the adjusted maximum luminance value and the peak luminance of the display; and scale each color component of the given pixel based on the adjustment factor. In this regard, additionally or alternatively, the adjustment factor may be determined using a tone mapping function selected from the group including linear functions, logarithmic functions, exponential functions, Reinhard formulas, and cinematic tone mapping operators. In this regard, additionally or alternatively, the processing circuitry may be further configured to apply an SDR-HDR pipeline, including an inverse photoelectric transfer function, an inverse tone mapping algorithm, and an inverse electro-optical transfer function, to each pixel in a video frame of an edited SDR video, thereby generating a video frame of an edited HDR video; and outputting the video frame of the edited HDR video for rendering on a display. In this regard, additionally or alternatively, the system may also include a camera that, upon receiving a video frame from the camera, applies the electro-optical transfer function, tone mapping algorithm, and photoelectric transfer function in real time to each pixel in the video frame, thereby outputting a video frame of the SDR video for rendering on a display without perceptible delay. In this regard, additionally or alternatively, the electro-optical transfer function may be a mixed log-gamma (HLG).In this regard, additionally or alternatively, video frames of the edited SDR video can be input into the preview generator to generate a preview for drawing on the monitor.

[0056] On the other hand, a computational method for applying special effects to high dynamic range (HDR) video is provided. This method includes: receiving a video frame of an HDR video, the video frame comprising a plurality of pixels, each pixel having one or more luminance values ​​for each of a plurality of color components; applying an HDR-to-SDR pipeline, including an electro-optical transfer function, a tone mapping algorithm, and a photoelectric transfer function, sequentially to each pixel of the HDR video video frame to generate a standard dynamic range (SDR) video frame having converted luminance values ​​for each of the plurality of color components; applying one or more special effects to the SDR video frame to generate an edited SDR video; and generating an output based on the edited SDR video. In this regard, additionally or alternatively, in the HDR-to-SDR pipeline, a photoelectric transfer function may be further applied to each pixel of the HDR video video frame so that the electro-optical transfer function, photoelectric transfer function, tone mapping algorithm, and photoelectric transfer function are applied sequentially to each pixel of the video frame. In this regard, additionally or alternatively, a tone mapping algorithm may be performed to determine the maximum luminance value in the color components of a given pixel; determine the perceived luminance value of the given pixel; and adjust the maximum luminance value of the given pixel based on the perceived luminance value. In this regard, additionally or alternatively, the perceived luminance value of the given pixel may be determined using the weights of the Rec.709 or Rec.2020 standards for the color components of the given pixel. In this regard, additionally or alternatively, a tone mapping algorithm may be further performed to determine an adjustment factor for the given pixel based on the adjusted maximum luminance value and the peak luminance of the display; and scale each color component of the given pixel based on the adjustment factor. In this regard, additionally or alternatively, the adjustment factor may be determined using a tone mapping function selected from the group including linear functions, logarithmic functions, exponential functions, Reinhard formulas, and cinematic tone mapping operators. In this respect, additionally or alternatively, the method may further include: applying an SDR-HDR pipeline, including an inverse photoelectric transfer function, an inverse tone mapping algorithm, and an inverse electro-optical transfer function, to each pixel in a video frame of an edited SDR video, thereby generating a video frame of an edited HDR video; and outputting the video frame of the edited HDR video for rendering on a display. In this respect, additionally or alternatively, the electro-optical transfer function may be a mixed log-gamma (HLG).

[0057] On the other hand, a computational system for applying special effects to high dynamic range (HDR) video is provided. This system includes processing circuitry and a memory storing instructions. When executed, the instructions cause the processing circuitry to receive video frames of HDR video, each video frame comprising multiple pixels, each pixel having one or more luminance values ​​for each of a plurality of color components. An HDR-to-SDR pipeline, including an electro-optical transfer function, a tone mapping algorithm, and a photoelectric transfer function, is sequentially applied to each pixel in the HDR video frame, thereby generating a standard dynamic range (SDR) video with special effects for each of the multiple color components. The system comprises: video frames with varying brightness values; applying one or more effects to the video frames of the SDR video to generate an edited SDR video; generating a preview for rendering on a display based on the edited SDR video; receiving user input to generate an edited HDR video; in response to receiving user input to generate the edited HDR video, applying an SDR-HDR pipeline including an inverse photoelectric transfer function, an inverse tone mapping algorithm, and an inverse electro-optical transfer function to each pixel in the video frames of the edited SDR video to generate video frames of the edited HDR video; and outputting the video frames of the edited HDR video for rendering on a display. Additionally or alternatively, the system may also include a camera that, upon receiving video frames from the camera, applies the electro-optical transfer function, tone mapping algorithm, and photoelectric transfer function in real time to each pixel in the video frames to output video frames of the HDR video for rendering on a display without perceptible delay.

[0058] It should be understood that the configurations and / or methods described herein are exemplary in nature, and these specific embodiments or examples should not be considered limiting, as many possible variations exist. The specific routines or methods described herein may represent one or more of any number of processing strategies. Therefore, the various operations shown and / or described may be performed in the shown and / or described order, other orders, in parallel, or omitted. Similarly, the order of the above processing may also be changed.

[0059] The subject matter of this disclosure includes all novel and non-obvious combinations and sub-combinations of the various processes, systems and configurations disclosed herein, as well as any and all equivalents thereof.

Claims

1. A computing system for applying special effects to a high dynamic range (HDR) video, the computing system comprising: processing circuitry and a memory for storing instructions that, when executed, cause the processing circuitry to: receive a video frame of the HDR video, the video frame comprising a plurality of pixels, each pixel having one or more luminance values for each of a plurality of color components; apply, in sequence, an HDR-to-standard dynamic range (SDR) pipeline comprising an electro-optical transfer function, a tone mapping algorithm, and an optical-electrical transfer function to each pixel in the video frame of the HDR video, thereby generating a video frame of an SDR video having transformed luminance values for each of the plurality of color components; apply one or more special effects to the video frame of the SDR video, thereby generating an edited SDR video; and generate an output based on the edited SDR video.

2. The computing system of claim 1, wherein, in the HDR-SDR pipeline, an optical- optical transfer function is further applied to each pixel in the video frame of the HDR video, such that the electro-optical transfer function, the optical-optical transfer function, the tone mapping algorithm, and the optical-electrical transfer function are applied to each pixel in the video frame in this order.

3. The computing system of claim 1, wherein the tone mapping algorithm is executed to: determine a maximum luminance value of the color components of a given pixel; determine a perceived luminance value of the given pixel; and adjust the maximum luminance value of the given pixel based on the perceived luminance value.

4. The computing system of claim 3, wherein the perceived luminance value of the given pixel is determined by using Rec. 709 standard weights or Rec. 2020 standard weights of the color components of the given pixel.

5. The computing system of claim 1, wherein the tone mapping algorithm is executed to: determine an adjustment factor for a given pixel based on the adjusted maximum luminance value and a peak luminance of a display; and scale each color component of the given pixel based on the adjustment factor.

6. The computing system of claim 5, wherein the adjustment factor is determined using a tone mapping function selected from a group comprising: a linear function, a logarithmic function, an exponential function, a Reinhard formula, and a filmic tone mapping operator.

7. The computing system of claim 1, wherein the processing circuitry is further configured to: apply an SDR-to-HDR pipeline comprising an inverse optical-electrical transfer function, an inverse tone mapping algorithm, and an inverse electro-optical transfer function to each pixel in a video frame of the edited SDR video, thereby generating a video frame of an edited HDR video; and output the video frame of the edited HDR video for rendering on a display.

8. The computing system of claim 7, further comprising a camera, wherein ​ ​ The electro-optical transfer function, the tone mapping algorithm, and the photo- optical transfer function are applied to each pixel in the video frame in real-time as the video frame is received from the camera, such that the video frame of the SDR video is output for rendering on the display without perceptible delay.

9. The computing system of claim 1, wherein the electro-optical transfer function is a hybrid logarithmic gamma (HLG).

10. The computing system of claim 1, wherein the video frame of the edited SDR video is input into a preview generator to generate a preview for rendering on the display.

11. A computing method for applying special effects to a high dynamic range (HDR) video, the computing method comprising: receiving a video frame of the HDR video, the video frame comprising a plurality of pixels, each pixel having one or more luminance values for each of a plurality of color components; applying, in sequence, an HDR-to-standard dynamic range (SDR) pipeline comprising an electro-optical transfer function, a tone mapping algorithm, and a photo-optical transfer function to each pixel in the video frame of the HDR video, thereby generating a video frame of an SDR video having luminance transformed values for each of the plurality of color components; applying one or more special effects to the video frame of the SDR video, thereby generating an edited SDR video; and generating an output based on the edited SDR video.

12. The computing method of claim 11, further comprising: applying an SDR-to-HDR pipeline comprising an inverse photo-optical transfer function, an inverse tone mapping algorithm, and an inverse electro-optical transfer function to each pixel in a video frame of the edited SDR video, thereby generating a video frame of an edited HDR video; and outputting the video frame of the edited HDR video for rendering on a display.

13. A computing system for applying special effects to a high dynamic range (HDR) video, the computing system comprising: processing circuitry and a memory for storing instructions that, when executed, cause the processing circuitry to: receive a video frame of the HDR video, the video frame comprising a plurality of pixels, each pixel having one or more luminance values for each of a plurality of color components; apply, in sequence, an HDR-to-standard dynamic range (SDR) pipeline comprising an electro-optical transfer function, a tone mapping algorithm, and a photo-optical transfer function to each pixel in the video frame of the HDR video, thereby generating a video frame of an SDR video having transformed luminance values for each of the plurality of color components; apply one or more special effects to the video frame of the SDR video, thereby generating an edited SDR video; and generate a preview for rendering on a display based on the edited SDR video; receive a user input to generate an edited HDR video; in response to receiving the user input to generate an edited HDR video, applying an SDR-HDR pipeline including an inverse electro-optical transfer function, an inverse tone mapping algorithm, and an inverse optical-electrical transfer function to each pixel in video frames of the edited SDR video, thereby generating video frames of an edited HDR video; and outputting the video frames of the edited HDR video for rendering on the display.

14. The computing system of claim 13, further comprising a camera, wherein in real-time as the video frames are received from the camera, the electro-optical transfer function, the tone mapping algorithm, and the optical-electrical transfer function are applied to each pixel in the video frames, such that the video frames of the HDR video are outputted for rendering on the display without a perceptible delay.

Images