Enhancing image data with appearance controls

Abstract

Embodiments of the present disclosure relate to enhancing image data with appearance controls. A digital image sequence having a plurality of image frames can be enhanced. An appearance graph can be determined from the digital image sequence. The appearance graph includes primary layer nodes. Each primary layer node can represent a unique visual style. A primary layer image sequence matching the visual style represented by the primary layer node can be computed for each primary layer node. An enhanced image sequence can be generated by blending at least two primary layer image sequences defined by the appearance graph.

Description

Divisional Application Instructions

[0001] This application is a divisional application of Chinese invention patent application No. 201980013693.9, entitled "Enhancing Image Data with Appearance Controls", which was filed internationally on January 18, 2019, entered the Chinese national phase on August 14, 2020. Cross-references to related applications

[0002] This disclosure claims priority to U.S. Provisional Application Serial No. 62 / 620,711 entitled “Enhancing Image Data with Appearance Controls”, filed January 23, 2018. Technical Field

[0003] This disclosure generally relates to the field of digitally enhancing the visual quality of image content, and more specifically to the ability to control the visual appearance of image content specifically tailored for various types of media platforms. Background Technology

[0004] The visual quality of moving images or video content can be improved by digitally processing image data in the temporal domain using sophisticated motion estimation or optical flow methods. These methods track the motion of each pixel across multiple image frames and enhance pixels to obtain better quality attributes, including noise reduction, deblurring, artifact removal, and detail enhancement. However, due to their high computational complexity, temporal processing techniques are extremely time-consuming, and the computation time can increase exponentially with increasing image resolution and the sophistication of motion tracking algorithms. The process can take hours or even days to yield any results from the computational job. If the resulting visual quality is unsatisfactory, the user adjusts certain rendering parameters and restarts the tedious computational work. Consequently, for typical post-production equipment that may be under very tight schedules and very limited production budgets, it is difficult to fine-tune image quality to satisfy the client.

[0005] Furthermore, image content is typically distributed across multiple types of displays, such as cinemas, packaged media, cable television, or streaming to tablets and mobile devices. Different display platforms can vary significantly in brightness, resolution, sharpness, color, contrast, bit rate, and whether the image is viewed directly or projected onto the display. Image data is processed to account for these differences in order to maintain a consistent artistic style or visual appearance across multiple platforms. For example, moving pictures might be timed for a 30-foot-wide frosted cinema screen with an organic “cinematic look.” When a film is upconverted for a large theatrical release (such as a 60-foot-wide IMAX® screen), the film grain preserved in the image data is further suppressed to prevent over-enlargement. When a film is released for television viewing via streaming services such as Netflix®, Amazon®, or Hulu®, the image data is typically re-enhanced for higher brightness, the Rec.709 color space instead of P3, and a higher level of data compression. Furthermore, the same movie from the same streaming service provider can appear significantly different, partly due to adaptive streaming protocols that adjust video stream quality based on available network bandwidth, and partly due to differences in the display quality of user devices. The rapid adoption of the latest UHD, HDR, WCG, and HFR display technologies will further impact the quality gap between different display platforms used to distribute the same content. Summary of the Invention

[0006] The embodiments of this disclosure relate to enhancing image data using appearance controls.

[0007] According to a first aspect of this disclosure, a method for enhancing a digital image sequence is provided, comprising: receiving a digital image sequence having a plurality of image frames; generating a plurality of master layer image sequences based on the digital image sequence, wherein each master layer image sequence is generated by a corresponding master layer processor, wherein each master layer processor includes a set of image processing methods and algorithms, such that each master layer image sequence represents a unique visual style; determining an appearance map including a plurality of master layer nodes, wherein each master layer image sequence is represented by a master layer node, such that each of the plurality of master layer nodes represents a unique visual style of the corresponding master layer image sequence; generating a second layer node by mixing at least two master layer nodes of the appearance map; and generating an enhanced image sequence corresponding to the second layer node by mixing and being mixed at least two master layer image sequences corresponding to the at least two master layer nodes used to generate the second layer node.

[0008] According to a second aspect of this disclosure, a system is provided, comprising: a processing device; and a memory device including instructions executable by the processing device to cause the processing device to: receive a digital image sequence having a plurality of image frames; generate a plurality of master layer image sequences based on the digital image sequence, wherein each master layer image sequence is generated by a corresponding master layer processor, wherein each master layer processor includes a set of image processing methods and algorithms such that each master layer image sequence represents a unique visual style; determine an appearance map comprising a plurality of master layer nodes, wherein each master layer image sequence is represented by a master layer node such that each of the plurality of master nodes represents a unique visual style of the corresponding master layer image sequence; generate a second layer node by mixing at least two master layer nodes of the appearance map; and generate an enhanced image sequence corresponding to the second layer node by mixing and being mixed at least two master layer image sequences corresponding to the at least two master layer nodes used to generate the second layer node.

[0009] According to a third aspect of this disclosure, a non-transient computer-readable medium is provided, including program code executable by a processor to cause the processor to: receive a digital image sequence having a plurality of image frames; generate a plurality of master-layer image sequences from the digital image sequence, wherein each master-layer image sequence is generated by a corresponding master-layer processor, wherein each master-layer processor includes a set of image processing methods and algorithms such that each master-layer image sequence represents a unique visual style; determine an appearance map including a plurality of master-layer nodes, wherein each master-layer image sequence is represented by a master-layer node such that each of the plurality of master-layer nodes represents a unique visual style of the corresponding master-layer image sequence; generate a second-layer node by mixing at least two master-layer nodes of the appearance map; and generate an enhanced image sequence corresponding to the second-layer node by mixing and being mixed at least two master-layer image sequences corresponding to the at least two master-layer nodes used to generate the second-layer node.

[0010] According to a fourth aspect of this disclosure, a method for enhancing a digital image sequence is provided, comprising: receiving a digital image sequence having a plurality of image frames and being image data; using a first master layer processor representing a unique visual style, generating first master layer nodes from the digital image sequence to represent the first master layer image sequence having a unique visual style; using a second master layer processor, generating second master layer nodes from the digital image sequence to represent a second master layer image sequence, the second master layer image sequence being an unchanged version of the digital image sequence; generating second layer nodes by mixing the first master layer nodes and the second master layer nodes; and generating an enhanced image sequence corresponding to the second layer nodes by mixing the first master layer image sequence and the second master layer image sequence corresponding to the first master layer nodes and the second master layer image sequence, respectively.

[0011] According to a fifth aspect of this disclosure, a system is provided, comprising: a processing device; and a memory device including instructions executable by the processing device to cause the processing device to: receive a digital image sequence having a plurality of image frames and being image data; generate a first master layer node based on the digital image sequence to represent a first master layer image sequence having a unique visual style; generate a second master layer node based on the digital image sequence to represent a second master layer image sequence, the second master layer image sequence being an unchanged version of the digital image sequence; generate a second layer node by mixing the first master layer node and the second master layer node; and generate an enhanced image sequence corresponding to the second layer node by mixing the first master layer image sequence and the second master layer image sequence corresponding to the first master layer node and the second master layer node, respectively. Attached Figure Description

[0012] Figure 1 This is a flowchart of a process for enhancing a digital image sequence according to some aspects of this disclosure.

[0013] Figure 2 This is an illustration of a system for enhancing image sequences using a mixing process, based on some aspects of this disclosure.

[0014] Figure 3 This is an illustration of an appearance map used in the process of enhancing the blending of image sequences, according to some aspects of this disclosure.

[0015] Figure 4 This is another example of an illustration of the appearance used in the process of enhancing the blending of image sequences, according to some aspects of this disclosure.

[0016] Figure 5 This is an illustration of a three-dimensional appearance used in the process of enhancing the mixing of image sequences, according to some aspects of this disclosure.

[0017] Figure 6 This is a block diagram of an image processing apparatus for enhancing image sequences according to some aspects of the present disclosure. Detailed Implementation

[0018] Some aspects and features involve enhancing one or more features of an image sequence by computing multiple prime layer image sequences with unique features. Unique features can be specific visual features related to image style. Combinations of prime layer image sequences can be blended to generate a specific visual appearance. The prime layer of an image sequence can be a digital representation of most of the content in the image sequence. The image sequence comprises multiple image frames. Each image frame comprises multiple image pixels. A typical image pixel can define an image content element at a spatiotemporal location related to adjacent image pixels within the same image frame, and can be associated with corresponding pixels across many image frames.

[0019] To achieve high performance, image enhancement algorithms can operate in the temporal domain by exploring pixel relationships across multiple image frames. Due to the time-varying nature of image sequences, such algorithms can include estimating pixel motion and tracking pixel movement across multiple image frames. Examples of motion estimation and tracking methods can include optical flow-based methods and more complex trajectory-based methods. However, the computation of algorithms utilizing motion estimation and tracking can become very complex and time-consuming. Furthermore, the performance of such algorithms can be difficult to control, and a trial-and-error approach may be necessary to obtain the desired results: if the results from one computation round are unsatisfactory, certain rendering parameters can be adjusted, and another computation round can be performed. Such a process can be expensive and time-consuming.

[0020] Based on some examples, image quality and appearance specifications can be achieved. Multiple methods can be used to process image sequences to produce multiple renditions of image sequences with unique appearances or artistic styles. At least a subset of these renditions can be mixed (e.g., by mixing) to produce a new image sequence with a novel visual appearance. The resulting image sequence can acquire a new visual appearance through the mixing of different artistic styles. If the new visual appearance is still unsatisfactory, it can be fine-tuned by carefully adjusting the contributions from each artistic style until the desired visual appearance is achieved. The renditions of the image sequence can be computed in advance so that they can be used as the basis for subsequent combination and fine-tuning operations using simpler and faster processing methods.

[0021] In one example, multiple master layer image sequences are computed based on an image sequence. Each master layer image sequence can represent a unique artistic style. Some master layer image sequences can be blended together to produce a new visual appearance. The blending process can include a blending function, and the contribution of each master layer image sequence to the new visual appearance can be quantitatively described by the appropriate weights used in the blending function. The visual appearance can be fine-tuned to produce the desired visual appearance by adjusting the weights of the individual master layer image sequences.

[0022] These illustrative examples are provided to introduce the general topics discussed here, and are not intended to limit the scope of the concepts disclosed. Various other features and examples will be described in the following sections with reference to the accompanying drawings.

[0023] Figure 1 This is a flowchart of a process for enhancing a digital image sequence according to an example of this disclosure. In block 102, the digital image sequence is received, such as by being received by a computing device. The digital image sequence may have multiple image frames that together form a visual media presentation, such as a film. In some examples, the digital image sequence may include or represent digital data. The digital data may be data intended to be used as streaming data or as data downloaded and stored from a remote source.

[0024] In box 104, an appearance graph is determined based on the digital image sequence. The appearance graph may include master-level nodes defining each master-level image sequence within the master-level image sequence. Each master-level node may represent a unique visual style. The appearance graph can be determined based on the digital image sequence by processing the digital image sequence individually. Processing the digital image sequence may include implementing image data analysis and feedback from the film director to determine master-level nodes reflecting visual characteristics or styles that may appeal to the film director.

[0025] In box 106, a sequence of master layer images matching the visual style represented by the node is computed for each master layer node.

[0026] In box 108, an enhanced image sequence is generated by blending at least two master layer image sequences defined by the appearance map. The master layer image sequences can be weighted individually before blending the image sequences together. In some examples, the enhanced image sequence can be refined to obtain a specific visual appearance by adjusting the weights used when blending the master layer image sequences.

[0027] Figure 2 This is a schematic block diagram of a system for enhancing image sequences according to some aspects of this disclosure. Image processing devices can be used to perform image processing operations according to various examples. Examples of image processing devices are shown in... Figure 6 It is described as being able to support a limited set of file formats for input and output image sequences. Figure 2In this process, the image processing device receives an input image sequence 200, which can be a moving picture or digital video content. The image sequence 200 may include multiple image frames. A format conversion process 202 is applied to the input image sequence 200 to convert its file format to one of the digital formats supported by the image processing device. Examples of supported digital formats may include common uncompressed raster formats such as TIFF, OpenEXR, DPX, PNG, and ACES. If the input image sequence 200 is encoded using a video compression method such as MPEG, JPEG, H.264, AVC, or ProRes, the file conversion process 202 may also perform video decoding. If the input image sequence 200 is encrypted, the file format conversion process 202 may also perform decryption. If the input image sequence 200 has already been packaged in a format supported by the image processing device, the file conversion process 202 may be skipped, and the input image sequence 200 may be provided for further processing.

[0028] Image sequences can also be digitized into multiple main layer sequences. For example... Figure 2 In the example shown, the image sequence is rendered into three different versions by three individual master layer processors 204, 206, and 208. The first master layer sequence is rendered from the input image sequence by master layer processor A 204 and is represented by master layer node A 210. The second master layer sequence is generated by master layer processor B 206 and is represented by master layer node B 212. The third master layer sequence is generated by master layer processor C 208 and is represented by master layer node C 214. Each master layer image sequence can represent a unique artistic style, allowing the desired visual appearance to be produced by combining some or all of the master layer image sequences. Any number of master layer nodes can be used.

[0029] Figure 2Each of the main layer processors 204, 206, and 208 can include a set of image processing methods and algorithms designed to generate main layer image sequences with a specific artistic style. In one example, one or more of the main layer processors 204, 206, and 208 can enhance image details, such as motion features, by computing accurate pixel motion trajectories across multiple image frames. The image can be decomposed into layers, and features of the image content can then be determined for each layer. The trajectory can represent the spatiotemporal location of selected image content elements over the length of the image sequence. Anchor trajectories can be computed in each layer based on feature connections. Feature connections can include links connecting image pixels in layers within spatiotemporal locations. Anchor trajectories can be based on at least one selected feature connection. The relationship between each pixel and the anchor trajectory can be analyzed, and a new trajectory can be created for each pixel based on the obtained relationship with the anchor trajectory. The resulting trajectories can also be optimized through an iterative improvement process. By establishing highly accurate trajectories for each pixel of the image sequence, it is possible to produce main layer sequences with a super-resolution artistic style, where fine image details are enhanced and some lost image details are restored.

[0030] Alternatively or concurrently, one or more of the main layer processors 204, 206, and 208 may include a noise reduction method that preserves image edges. Spatial and temporal computational processes can be deployed in the main layer processors to ensure that image edges are maintained. The resulting main layer image sequence can represent a very clean edge-preserving artistic style.

[0031] Alternatively or additionally, one or more of the main layer processors 204, 206, 208 may include using optical flow methods for motion estimation to remove random noise. The resulting main layer image sequence may represent a clean-cut artistic style, which may have no or substantially no noise perceived in the image, such as film grain or camera electronic noise.

[0032] Alternatively or concurrently, one or more of the main layer processors 204, 206, and 208 may include enhancing image quality by using temporal filtering of motion. The resulting main layer image sequence may represent an artistic style that maintains a balance between noise suppression and detail enhancement.

[0033] Alternatively or concurrently, one or more of the main layer processors 204, 206, 208 may include using frame interpolation to enhance the temporal resolution of the image sequence. As a result of frame interpolation, the resulting main layer image sequence may have additional image frames, representing an improved motion-smoothing art style with an increased frame rate.

[0034] Alternatively or alternatively, one or more of the main layer processors 204, 206, and 208 may include dynamic range enhancement. The resulting main layer image sequence may represent high dynamic range or HDR artistic style. Alternatively or alternatively, one or more of the main layer processors 204, 206, and 208 may include a method for expanding the color gamut. The resulting main layer image sequence may represent wide color gamut or WCG artistic style.

[0035] Alternatively or alternatively, one or more of the master layer processors 204, 206, and 208 may include the addition of artificially generated film grain. The resulting master layer image sequence may represent the visual style of an old film look. Alternatively or alternatively, the original image sequence may be treated as a master layer image sequence without any processing, and one or more master layer processors 204, 206, and 208 may be all-pass filters that do not alter the image. Other types of image processing methods may be used in any of the master layer processors 204, 206, and 208, as long as these methods help produce the desired artistic style.

[0036] An appearance map can be constructed to provide a sequence of images with a new visual appearance, which can then be used for further fine-tuning. An appearance map can include multiple master-layer nodes, each representing a master-layer image sequence with a unique artistic style. Since at least two master-layer image sequences are used for blending, an appearance map can have at least two master-layer nodes. Second-layer nodes can be generated by blending multiple master-layer nodes to indicate the generation of a new sequence of images with a novel visual appearance. Second-layer nodes can also be used to form third-layer nodes by blending with one or more other second-layer nodes and possibly with one or more master-layer nodes, and so on. Third-layer nodes can be used for fine-tuning for a specific display device or to address bandwidth limitations. Typically, higher-layer nodes are generated by blending multiple lower-layer nodes, indicating the generation of a higher-layer image sequence with a newer appearance—a blend of the artistic style of the lower-layer image sequences and improvements that overcome display system limitations.

[0037] exist Figure 2In the diagram, there is an appearance diagram 230 representing a portion of the blending process. The three master layer image sequences generated from the previous steps are represented by master layer nodes A 210, B 212, and C 214, or by the abbreviated form of nodes A 210, B 212, and C 214. The blending function can be used to combine nodes B 212 and C 214 to create a new second-layer node D 216. A link 218 exists between node A 210 and node B 212, indicating that these two nodes can be blended together. A link 220 also exists between node A 210 and node C 214, further indicating that these two nodes can also be blended together. Similarly, a link 222 exists between nodes B 212 and C 214. Figure 2 In the example, nodes B and C are used to create a new image sequence represented by second-layer node D 216 with a new appearance.

[0038] In some examples, the blending function is implemented as a weighted arithmetic mean. Figure 2 In the above, the pixel from node D 216 can be calculated using the following formula based on the corresponding pixels from nodes B 212 and C 214: in These are the weights assigned to the second master layer image sequence represented by node B 212. It is the weight assigned to the third master layer image sequence represented by node C 214. It is the spatiotemporal location of the new image sequence Pixel value at that location, The i-th main layer sequence is at the same spatiotemporal position The pixel value at that location. Other examples for blending functions are also possible, including the median or geometric mean.

[0039] By adjusting the weights and Once a satisfactory appearance is achieved, the new image sequence can be further fine-tuned. To maintain brightness and color, the fine-tuning process can meet the following conditions: , , .

[0040] File conversion process 224 can be used to package a new image sequence, which is the output image sequence 226, into a specific output file format used by certain applications.

[0041] exist Figure 3Another example of the blending process and another example of the appearance illustration are depicted. Three master layer image sequences are generated, and the process of creating these master layer image sequences is omitted to focus the discussion on the appearance illustration. Master layer sequence A 302 is represented by master layer node A 310, master layer sequence B 304 is represented by master layer node B 312, and master layer sequence C 306 is represented by master layer node C 314. Similar to Figure 2 After fine-tuning the weights assigned to nodes B 312 and C 314, nodes B 312 and C 314 are blended together to create a second-layer node D 316. The second-layer image sequence D 316 is further blended with node A 310 to create a third-layer image sequence represented by node E 318. The third-layer image sequence 318 has a new appearance with contributions from nodes B 312, C 314, and A 310. During the second blending process, the weights assigned to nodes D 316 and A 310 are fine-tuned respectively. , This creates the desired new appearance. After two blending processes, the node E 318 sequence becomes the output image sequence 326. For simplicity, the file format conversion step has been omitted.

[0042] exist Figure 4 Another example of the blending process and another example of the appearance illustration are depicted. Four master layer image sequences are generated. Master layer sequence A 402 is represented by master layer node A 410, master layer sequence B 404 is represented by master layer node B 412, master layer sequence C 406 is represented by master layer node C 414, and master layer sequence D 408 is represented by master layer node D 416. Three master layer nodes (node ​​B 412, node C 414, and node D 416) are blended together to create a second layer node E418. The fine-tuning process may involve adjusting three weights: the weights assigned to node B 412, the weights assigned to node B 412, and the weights assigned to node D 416. Assigned to node C414 and assigned to node D416 After finally determining the second-layer node E 418, it is further blended with node A 410 to create a third-layer image sequence represented by node F 420. The third-layer image sequence represented by node F 420 has a new appearance, which incorporates contributions from nodes B 412, C 414, D 416, and A 410. During the second blending process, the weights assigned to nodes F 420 and A 410 are adjusted accordingly. , Fine-tune the process to create the desired new look. Node F 420 sequence becomes output image sequence 426.

[0043] The appearance map disclosed in the above examples can be extended to cover blending processes with any number of main layer sequences and any number of blending layers. In general, when blending M lower-layer image sequences... When generating new image sequences represented by nodes at higher levels, the general form of the mixing formula can be described as follows: .

[0044] When using luminance preservation and color preservation, the weights can satisfy the following conditions: , ,for

[0045] The appearance diagram can be presented graphically in alternative formats. In one example, the appearance diagram is drawn as a 3D model, where nodes from the same layer are placed on the same plane, while nodes from higher layers are placed on a higher plane. An example of a 3D appearance diagram is shown in... Figure 5 The description in the middle, Figure 5 Described with Figure 4 The same blending functionality. When blending functionality involves a large number of nodes and a large number of blending layers, the 3D representation can be more visually appealing.

[0046] exist Figure 5 In the example, the main layer sequence A 502 is represented by main layer node A 510, the main layer sequence B 504 is represented by main layer node B 512, the main layer sequence C 506 is represented by main layer node C 514, and the main layer sequence D 508 is represented by main layer node D 516. Nodes 510, 512, 514, and 516 are on the same plane. The second layer node E formed by node B 512 and node D 516, and the associated weights depicted, are on different planes. This results in the output image sequence 526, and the third layer node F 520 formed by node E 518, node A 510, and node C, and the associated weights depicted, are even on different planes.

[0047] although Figure 5 In this example, nodes at different layers are labeled with text; however, in other examples, nodes at different layers may be labeled with different colors, shapes, sizes, or other methods. Despite the various forms of graphical description, an appearance graph with a selected set of weights can still define the process of generating a sequence of enhanced images with a defined appearance, and the results are reproducible. The appearance graph can also be implemented as a graphical user interface for interactive control of the blending process.

[0048] For the blending function to succeed, the main layer image sequence can have the same pixel registration as the input image sequence. For example... Figure 2 As shown, image methods and algorithms implemented in any main-layer processor can be translation-invariant to avoid image sharpness and detail degradation.

[0049] An example of the blending process is that one master layer image sequence represents a clean plate style, with less detail than the digital image data, while another master layer image sequence is ultra-detailed image data generated from the digital image data, with more detail than the digital image data. The clean plate image data and ultra-detailed image data are blended to produce enhanced digital image data that is augmented compared to the received digital image data. Blending clean plate image data with ultra-detailed image data to produce enhanced digital image data can include: blending the clean plate image data and ultra-detailed image data with the original image data to produce enhanced digital image data. This blending can be performed simultaneously with a viewer viewing the enhanced digital image data on a display. The blending of clean plate image data with ultra-detailed image data can be performed within a streaming bandwidth threshold.

[0050] Some applications may involve regions of an image whose visual appearance differs from the rest of the image. Blending two or more image sequences at any layer of the appearance map can be performed globally or locally. In one example, the blending function applies the same weights globally to the image pixels in each image frame. In another example, the blending function applies weights locally to different parts of the image frame using different weights. In yet another example, the blending function applies a different set of weights to different scenes in the image sequence or even different image frames. Masks can be created to define local regions where different or optimal local blending operations can be applied.

[0051] In some aspects, digital image data is analyzed to identify at least two distinct objects or regions. Each object or region can be blended differently to produce blended enhanced digital image data for each object or region. The enhanced digital image data for each object or region can be combined to produce enhanced digital image data. Blending differently for each of at least two distinct objects or regions to produce blended enhanced digital image data for each object or region may include: using blending input received from a film director presenting a characteristic cinematic type. One or more blending settings for objects or regions can be stored. Furthermore, the blending settings can be applied to corresponding objects or regions in another characteristic cinematic presentation produced by the film director.

[0052] One example could be a close-up scene of a face, where certain skin-tone areas of the face could use a texture slightly different from the rest of the scene. One solution is to utilize a different set of weights to blend the skin-tone areas. Local masks are created to track skin-tone areas across the scene. One set of weights is applied inside the mask, while another set is applied outside. Masks for skin-tone areas can be generated using face recognition and tracking methods. Another example includes a landscape scene. The sky can have a cleaner appearance than the hills and trees. A local mask is created using a color separation algorithm to isolate the sky, and a different set of weights is applied to the isolated sky area. This local appearance control can be generalized to include any number of local masks, each associated with a different set of weights. Soft local masks can also be used, where the weight values ​​can gradually change from inside the mask to the mask's edges.

[0053] Some other applications may involve appearances that change with the scene, and even with each frame. An example of such an application could be a daytime scene followed by a nighttime scene. The daytime scene might require a sharper appearance, while the nighttime scene might contain more noise and involve additional noise cleanup. One solution is to choose a set of weights for the daytime scene and another set for the nighttime scene. The changes to these weights can be controlled by defining time windows for the start and end frames.

[0054] Some examples of digital augmentation of input image sequences and output of augmented image sequences can be implemented in a hardware computing device connected to an image display system via an external connection. The computations performed in the hardware computing device can be implemented using a general-purpose computer transformed into a dedicated computer configured to utilize a high-performance graphics processing unit (GPU) card to execute various methods according to the various examples of this disclosure. Such a configuration allows the computing power of the GPU to be utilized to perform computational tasks in software such as OpenGL GLSL, Cg, or CUDA. For example, a computer program product can be set on one or more hardware devices such as those described above, or on other devices such as a computer system or server. The computer program product may include executable code stored on computer-readable storage, which, when executed, causes the hardware device to perform various methods according to the various examples of this disclosure. Using a GPU can reduce the development cost for embedded devices and provide flexibility for future feature upgrades. Augmented image sequence data generated by the hardware computing device or software can be directly displayed on the image display system via an external connection.

[0055] Figure 6This is a block diagram of an example computing system including an image processing device 600 that can be used to process images according to various aspects of this disclosure. The image processing device 600 includes a processor 602, a memory 604, and a bus 606. The memory 604 includes a computer-readable storage device that stores code thereon. The processor 602 can execute the code stored in the memory 604 via communication via the bus 606 to cause the image processing device 600 to perform actions. The image processing device 600 may include an input / output (I / O) interface 608 for communicating with other components such as a display device 614. The image processing device 600 can be any device capable of processing data and executing code (i.e., a set of instructions for performing operations). Examples of the image processing device 600 include database servers, web servers, desktop personal computers, laptop personal computers, handheld computing devices, and mobile devices.

[0056] Examples of processor 602 include microprocessors, application-specific integrated circuits (ASICs), state machines, or other suitable processors. Processor 602 may include one processor or any number of processors. Processor 602 can access code stored in memory 604 via bus 606. Memory 604 may be any non-transient computer-readable medium configured to tangibly represent code and may include electronic, magnetic, or optical devices. Examples of memory 604 include random access memory (RAM), read-only memory (ROM), floppy disks, optical disks, digital video devices, magnetic disks, ASICs, configured processors, or other storage devices.

[0057] Instructions may be stored in memory 604 as executable code. Instructions may include processor-specific instructions generated by a compiler or interpreter from code written in any suitable computer programming language. Instructions may include applications such as image processing engine 610, which, when executed by processor 602, can cause image processing device 600 to perform actions, such as combining... Figures 1-5 The described actions. The image processing device 600 may also include a GPU. The memory 604 may also include a data storage 612 in which content and data can be stored. Some aspects can be implemented in a projector system such as a projector system that includes the image processing device 600.

[0058] Examples of some aspects of this disclosure are described below.

[0059] Example 1 is a method for enhancing a digital image sequence, the method comprising: receiving a digital image sequence having multiple image frames; determining an appearance map comprising multiple master layer nodes based on the digital image sequence, each of the multiple master layer nodes representing a unique visual style; computing a master layer image sequence for each master layer node that matches the visual style represented by the master layer node; and generating an enhanced image sequence by mixing at least two master layer image sequences defined by the appearance map.

[0060] Example 2 is based on the method of Example 1, wherein blending at least two master layer image sequences includes applying separate weights to each of the at least two master layer image sequences, and the method also includes improving the enhanced image sequences to achieve a specific visual appearance by adjusting the separate weights used during blending.

[0061] Example 3 is based on the method of Example 1, and further includes: receiving a digital image sequence as digital image data; generating clean plate image data from the digital image data, the clean plate image data having less detail than the digital image data; generating ultra-detail image data from the digital image data, the ultra-detail image data having more detail than the digital image data; and mixing the clean plate image data with the ultra-detail image data to produce enhanced digital image data.

[0062] Example 4 is based on the method of Example 3, wherein mixing the clean plate image data with the hyper-detail image data to produce enhanced digital image data includes: mixing the clean plate image data and the hyper-detail image data with digital image data to produce enhanced digital image data.

[0063] Example 5 is based on the method of Example 3, wherein the mixing of net-plate image data with hyper-detailed image data to produce enhanced digital image data is performed simultaneously with the viewer viewing the enhanced digital image data on the monitor.

[0064] Example 6 is based on the method of Example 3, and further includes: analyzing digital image data to identify at least two different objects or regions; mixing them differently for each of the at least two different objects or regions to produce mixed enhanced digital image data for each object or region; and combining the enhanced digital image data for each object or region to produce enhanced digital image data.

[0065] Example 7 is based on the method of Example 6, wherein different blending for each of at least two different objects or regions to produce blended enhanced digital image data for each object or region includes: using blending input received from a film director presented with a characteristic cinema type.

[0066] Example 8 is based on the method of Example 7, and further includes: storing one or more blending settings for at least two different objects or regions; and applying one or more blending settings to the corresponding objects or regions in another feature cinematic presentation created by the film director.

[0067] Example 9 is based on the method of Example 3, wherein the mixing of clean plate image data with ultra-detailed image data is performed within a streaming bandwidth threshold.

[0068] Example 10 is a system comprising: a processing device; and a memory device including instructions executable by the processing device to cause the processing device to perform the following operations: receiving a digital image sequence having a plurality of image frames; determining an appearance map comprising a plurality of master layer nodes based on the digital image sequence, each of the plurality of master layer nodes representing a unique visual style; computing, for each master layer node, a master layer image sequence matching the visual style represented by the master layer node; and generating an enhanced image sequence by mixing at least two master layer image sequences defined by the appearance map.

[0069] Example 11 is a system according to Example 10, wherein mixing at least two master layer image sequences includes: applying separate weights to each of the at least two master layer image sequences, wherein the memory device further includes instructions executable by a processing device to improve the enhanced image sequences to achieve a particular visual appearance by adjusting the weights used during mixing.

[0070] Example 12 is a system according to Example 10, wherein the memory device further includes instructions executable by a processing device to perform the following operations: receiving a sequence of digital images as digital image data; generating bare plate image data from the digital image data, the bare plate image data having less detail than the digital image data; generating hyper-detailed image data from the digital image data, the hyper-detailed image data having more detail than the digital image data; and mixing the bare plate image data with the hyper-detailed image data to produce enhanced digital image data.

[0071] Example 13 is a system according to Example 12, wherein mixing clean plate image data with hyper-detail image data to produce enhanced digital image data includes: mixing clean plate image data and hyper-detail image data with digital image data to produce enhanced digital image data.

[0072] Example 14 is a system based on Example 12, wherein mixing net-panel image data with hyper-detailed image data to produce enhanced digital image data is performed simultaneously with a viewer viewing the enhanced digital image data on a display.

[0073] Example 15 is a system according to Example 12, wherein the memory device further includes instructions executable by the processing device to perform the following operations: analyzing digital image data to identify at least two different objects or regions; mixing differently for each of the at least two different objects or regions to produce mixed enhanced digital image data for each object or region; and combining the enhanced digital image data for each object or region to produce enhanced digital image data.

[0074] Example 16 is a system based on Example 15, wherein different blending for each of at least two different objects or regions to produce blended enhanced digital image data for each object or region includes: using blending input received from a film director presented with a characteristic cinema type.

[0075] Example 17 is a system according to Example 16, wherein the memory device further includes instructions executable by the processing device to perform the following operations: storing one or more blending settings for at least two different objects or regions; and applying one or more blending settings to corresponding objects or regions in another feature cinematic presentation created by a film director.

[0076] Example 18 is a system based on Example 12, in which the mixing of clean plate image data with hyper-detailed image data is performed within a streaming bandwidth threshold.

[0077] Example 19 is a non-transient computer-readable medium comprising program code executable by a processor to cause the processor to perform the following operations: receiving a sequence of digital images having a plurality of image frames; determining an appearance map comprising a plurality of master layer nodes, each of the plurality of master layer nodes representing a unique visual style, based on the digital image sequence; computing a master layer image sequence for each master layer node that matches the visual style represented by the master layer node; and generating an enhanced image sequence by mixing at least two master layer image sequences defined by the appearance map.

[0078] Example 20 is a non-transient computer-readable medium according to Example 19, wherein mixing at least two master layer image sequences includes: applying separate weights to each of the at least two master layer image sequences, and the method further includes improving the enhanced image sequences to achieve a specific visual appearance by adjusting the weights used during mixing.

[0079] Example 21 is a non-transient computer-readable medium according to Example 20, and also includes program code executable by a processor to perform the following operations: receiving a sequence of digital images as digital image data; generating bare plate image data from the digital image data, the bare plate image data having less detail than the digital image data; generating hyper-detailed image data from the digital image data, the hyper-detailed image data having more detail than the digital image data; and mixing the bare plate image data with the hyper-detailed image data to produce enhanced digital image data.

[0080] Example 22 is a non-transient computer-readable medium according to Example 21, wherein mixing clean plate image data with hyper-detailed image data to produce enhanced digital image data includes: mixing clean plate image data and hyper-detailed image data with digital image data to produce enhanced digital image data.

[0081] Example 23 is a non-transient computer-readable medium according to Example 21, wherein mixing net-plate image data with hyper-detailed image data to produce enhanced digital image data is performed simultaneously with a viewer viewing the enhanced digital image data on a display.

[0082] Example 24 is a non-transient computer-readable medium according to Example 21, and also includes program code executable by a processor to perform the following operations: analyzing digital image data to identify at least two distinct objects or regions; mixing them differently for each of the at least two distinct objects or regions to produce mixed enhanced digital image data for each object or region; and combining the enhanced digital image data for each object or region to produce enhanced digital image data.

[0083] Example 25 is a non-transient computer-readable medium according to Example 24, wherein different mixing for each of at least two different objects or regions to produce mixed enhanced digital image data for each object or region includes: using mixed input received from a film director presented with a characteristic cinema type.

[0084] Example 26 is a non-transient computer-readable medium according to Example 25, further comprising: storing one or more blending settings for at least two different objects or regions; and applying the one or more blending settings to the corresponding objects or regions in another feature cinematic presentation created by a film director.

[0085] Example 27 is performed on a non-transient computer-readable medium according to Example 21, wherein the mixing of clean plate image data with hyper-detailed image data is performed within a streaming bandwidth threshold.

[0086] The foregoing is provided to illustrate, explain, and describe embodiments of this disclosure. Other modifications and adaptations to these embodiments will be readily apparent to those skilled in the art and can be made without departing from the scope of this disclosure.

Claims

1. A method for enhancing a digital image sequence, the method comprising: Receive the digital image sequence having multiple image frames; Multiple master layer image sequences are generated based on the digital image sequence, wherein each master layer image sequence is generated by a corresponding master layer processor, and each master layer processor includes a set of image processing methods and algorithms, thereby enabling each master layer image sequence to represent a unique visual style. A visual image is determined comprising multiple master layer nodes, wherein each master layer image sequence is represented by a master layer node, such that each of the multiple master layer nodes represents the unique visual style of the corresponding master layer image sequence; The second layer nodes are generated by merging at least two primary layer nodes of the appearance graph; as well as An enhanced image sequence corresponding to the second layer node is generated by mixing and being mixed to generate at least two main layer image sequences corresponding to the at least two main layer nodes of the second layer node.

2. The method of claim 1, wherein mixing the at least two main layer image sequences comprises: The method further includes applying individual weights to each of the at least two master layer image sequences, and improving the enhanced image sequences to achieve a specific visual appearance by adjusting the individual weights used during blending.

3. The method according to claim 1, further comprising: Receive the digital image sequence as digital image data; Clean board image data is generated based on the digital image data, wherein the clean board image data has less detail than the digital image data; Generate ultra-detailed image data based on the digital image data, wherein the ultra-detailed image data contains more details than the digital image data; and The clean plate image data is mixed with the ultra-detailed image data to produce enhanced digital image data.

4. The method of claim 3, wherein mixing the net-plate image data with the ultra-detailed image data to generate the enhanced digital image data comprises: The clean plate image data and the ultra-detailed image data are mixed with the digital image data to generate the enhanced digital image data.

5. The method of claim 3, wherein mixing the net-panel image data with the ultra-detailed image data to generate the enhanced digital image data is performed simultaneously with a viewer viewing the enhanced digital image data on a display.

6. The method according to claim 3, further comprising: Analyze the digital image data to identify at least two distinct objects or regions; For each of the at least two different objects or regions, they are blended differently to produce blended enhanced digital image data for each object or region; as well as The enhanced digital image data for each object or region is combined to produce the enhanced digital image data.

7. The method of claim 6, wherein different mixing for each of the at least two different objects or regions to generate the mixed enhanced digital image data for each object or region comprises: The film director is received using a mix of inputs from the featured cinema types.

8. The method of claim 7, further comprising: Store one or more mixed settings for the at least two different objects or regions; as well as The one or more hybrid settings are applied to the corresponding objects or areas in another feature cinematic presentation created by the film director.

9. The method of claim 3, wherein the mixing of the net plate image data and the ultra-detailed image data is performed within a streaming bandwidth threshold.

10. The method of claim 1, wherein one of the at least two master layer image sequences corresponds in content to the digital image sequence.

11. The method of claim 10, further comprising: A master layer image sequence corresponding to the digital image sequence in content is generated by using an all-pass filter that does not change the image content of the digital image sequence.

12. A system comprising: Processing equipment; as well as A memory device, including instructions that can be executed by the processing device to cause the processing device to: Receive a digital image sequence with multiple image frames; Multiple master layer image sequences are generated based on the digital image sequence, wherein each master layer image sequence is generated by a corresponding master layer processor, and each master layer processor includes a set of image processing methods and algorithms, thereby enabling each master layer image sequence to represent a unique visual style. A visual image is determined comprising multiple master layer nodes, wherein each master layer image sequence is represented by a master layer node, such that each of the multiple master layer nodes represents a unique visual style of the corresponding master layer image sequence; The second layer nodes are generated by merging at least two primary layer nodes of the appearance graph; as well as An enhanced image sequence corresponding to the second layer node is generated by mixing and being mixed to generate at least two main layer image sequences corresponding to the at least two main layer nodes of the second layer node.

13. The system of claim 12, wherein the memory device includes instructions executable by the processing device to cause the processing device to: blend the at least two master layer image sequences by applying separate weights to each of the at least two master layer image sequences, wherein the memory device further includes instructions executable by the processing device to improve the enhanced image sequences to achieve a specific visual appearance by adjusting the weights used during blending.

14. The system of claim 12, wherein the memory device further comprises instructions executable by the processing device to: Receive the digital image sequence as digital image data; Clean board image data is generated based on the digital image data, wherein the clean board image data has less detail than the digital image data; Generate ultra-detailed image data based on the digital image data, wherein the ultra-detailed image data contains more details than the digital image data; and The clean plate image data is mixed with the ultra-detailed image data to produce enhanced digital image data.

15. The system of claim 14, wherein the memory device includes instructions executable by the processing device to cause the processing device to: mix the clean plate image data with the hyper-detailed image data to generate the enhanced digital image data by mixing the clean plate image data and the hyper-detailed image data with the digital image data to generate the enhanced digital image data.

16. The system of claim 14, wherein the memory device includes instructions executable by the processing device to cause the processing device to: simultaneously with a viewer viewing the enhanced digital image data on a display, blend the net-panel image data with the hyper-detailed image data to produce the enhanced digital image data.

17. The system of claim 14, wherein the memory device further comprises instructions executable by the processing device to: Analyze the digital image data to identify at least two distinct objects or regions; For each of the at least two different objects or regions, they are blended differently to produce blended enhanced digital image data for each object or region; as well as The enhanced digital image data for each object or region is combined to produce the enhanced digital image data.

18. The system of claim 17, wherein the memory device includes instructions executable by the processing device to cause the processing device to: differently mix for each of the at least two different objects or regions by using mixed input received from a film director presented from a characteristic cinema type to produce the mixed enhanced digital image data for each object or region.

19. The system of claim 18, wherein the memory device further comprises instructions executable by the processing device to: Store one or more mixed settings for the at least two different objects or regions; and The one or more hybrid settings are applied to the corresponding objects or areas in another feature cinematic presentation created by the film director.

20. The system of claim 14, wherein the memory device includes instructions executable by the processing device to cause the processing device to: mix the clean plate image data with the hyper-detailed image data within a streaming bandwidth threshold.

21. The system of claim 12, wherein one of the at least two master layer image sequences corresponds in content to the digital image sequence.

22. The system of claim 21, wherein the memory device includes instructions executable to cause the processing device to generate a primary layer image sequence corresponding in content to the digital image sequence by using an all-pass filter that does not alter the image content of the digital image sequence.

23. A non-transient computer-readable medium comprising program code executable by a processor to cause the processor to: Receive a digital image sequence with multiple image frames; Multiple master layer image sequences are generated based on the digital image sequence, wherein each master layer image sequence is generated by a corresponding master layer processor, and each master layer processor includes a set of image processing methods and algorithms, thereby enabling each master layer image sequence to represent a unique visual style. A visual image is determined comprising multiple master layer nodes, wherein each master layer image sequence is represented by a master layer node, such that each of the multiple master layer nodes represents the unique visual style of the corresponding master layer image sequence; The second layer nodes are generated by merging at least two primary layer nodes of the appearance graph; as well as An enhanced image sequence corresponding to the second layer node is generated by mixing and being mixed to generate at least two main layer image sequences corresponding to the at least two main layer nodes of the second layer node.

24. The non-transient computer-readable medium of claim 23, wherein mixing the at least two main layer image sequences comprises: The method further includes applying individual weights to each of the at least two master layer image sequences, and improving the enhanced image sequences to achieve a specific visual appearance by adjusting the weights used during blending. The non-transient computer-readable medium also includes program code executable by the processor for: Receive the digital image sequence as digital image data; Clean board image data is generated based on the digital image data, wherein the clean board image data has less detail than the digital image data; Ultra-detailed image data is generated based on the digital image data, wherein the ultra-detailed image data contains more details than the digital image data. The clean panel image data is mixed with the ultra-detailed image data to produce enhanced digital image data. The process of mixing the clean-panel image data with the hyper-detailed image data to generate the enhanced digital image data includes: simultaneously with a viewer viewing the enhanced digital image data on a display and within a bandwidth threshold, mixing the clean-panel image data and the hyper-detailed image data with the digital image data to generate the enhanced digital image data.

25. The non-transient computer-readable medium of claim 23, wherein one of the at least two main image sequences corresponds in the content to the digital image sequence.

26. The non-transient computer-readable medium of claim 25, wherein the program code is executable to cause the processor to generate, by using an all-pass filter that does not alter the image content of the digital image sequence, the primary layer image sequence corresponding to the digital image sequence in the content.

27. A method for enhancing a digital image sequence, the method comprising: Receive the digital image sequence, which has multiple image frames and is image data; Using a first master layer processor that represents a unique visual style, a first master layer node is generated based on the digital image sequence to represent a first master layer image sequence having the unique visual style; Using a second master layer processor, a second master layer node is generated based on the digital image sequence to represent the second master layer image sequence, wherein the second master layer image sequence is an unchanged version of the digital image sequence; The second layer nodes are generated by mixing the first main layer nodes and the second main layer nodes; as well as An enhanced image sequence corresponding to the second layer node is generated by mixing the first master layer image sequence and the second master layer image sequence, which are respectively corresponding to the first master layer node and the second master layer node.

28. The method of claim 27, further comprising: Determine the appearance diagram including the first main layer node and the second main layer node.

29. The method of claim 27, wherein mixing the first master layer node and the second master layer node includes applying separate weights to each master layer image sequence in the first master layer image sequence and the second master layer image sequence, the method further comprising: The enhanced image sequence is improved to achieve a specific visual appearance by adjusting the individual weights used during blending.

30. The method of claim 27, wherein generating the second master layer node to represent the second master layer image sequence comprises: The second main layer processor uses an all-pass filter that does not change the image content of the digital image sequence.

31. The method of claim 27, wherein the first master layer node represents a net plate image sequence generated according to the digital image sequence, the generated net plate image sequence having less detail than the digital image sequence.

32. The method of claim 31, wherein the generated clean plate image sequence is substantially free of perceptible noise.

33. The method of claim 31, wherein the generated clean plate image sequence is substantially free of camera noise.

34. The method of claim 31, wherein the clean plate image sequence is substantially free of film grain.

35. The method of claim 27, wherein the mixing to generate the enhanced image sequence is performed simultaneously with the viewer viewing the enhanced image sequence on a display.

36. The method of claim 27, further comprising: Analyze the digital image sequence to identify at least two distinct objects or regions; For each of the at least two different objects or regions, the data is mixed differently to produce the enhanced digital image data.

37. The method of claim 27, wherein the mixing is performed within a flow bandwidth threshold.

38. A system comprising: Processing equipment; as well as A memory device, including instructions that can be executed by the processing device to cause the processing device to: Receive a digital image sequence, the digital image sequence having multiple image frames and being image data; A first master layer node is generated based on the digital image sequence to represent a first master layer image sequence with a unique visual style; A second master layer node is generated based on the digital image sequence to represent the second master layer image sequence, wherein the second master layer image sequence is an unchanged version of the digital image sequence; The second layer nodes are generated by mixing the first main layer nodes and the second main layer nodes; as well as An enhanced image sequence corresponding to the second layer node is generated by mixing the first master layer image sequence and the second master layer image sequence, which are respectively corresponding to the first master layer node and the second master layer node.

39. The system of claim 38, wherein the memory device further comprises instructions executable by the processing device to: Determine the appearance diagram including the first main layer node and the second main layer node.

40. The system of claim 38, wherein the memory device includes instructions executable by the processing device to cause the processing device to: The first master layer nodes and the second master layer nodes are blended by applying separate weights to each master layer image sequence in the first master layer image sequence and the second master layer image sequence; and The enhanced image sequence is improved to achieve a specific visual appearance by adjusting the individual weights used when mixing the first master layer node and the second master layer node.

41. The system of claim 38, wherein the memory device includes instructions executable by the processing device to cause the processing device to: generate a second master layer node to represent the second master layer image sequence using an all-pass filter that does not alter the image content of the digital image sequence.

42. The system of claim 38, wherein the first master layer node represents a generated clean plate image sequence based on the digital image sequence, the generated clean plate image sequence having less detail than the digital image sequence.

43. The system of claim 42, wherein the generated clean plate image sequence is substantially free of perceptible noise.

44. The system of claim 42, wherein the generated clean plate image sequence is substantially free of camera noise.

45. The system of claim 42, wherein the clean plate image sequence is substantially free of film grain.

46. ​​The system of claim 38, wherein the system is configured to simultaneously blend with a viewer viewing the enhanced image sequence on a display to produce the enhanced image sequence.

47. The system of claim 38, wherein the memory device further comprises instructions executable by the processing device to: Analyze the digital image sequence to identify at least two distinct objects or regions; For each of the at least two different objects or regions, the data is mixed differently to produce the enhanced image data.

48. The system of claim 38, wherein the system is configured to mix within a streaming bandwidth threshold.

Images