Real-time non-photo real rendering

Abstract

To provide a system, a method and a storage medium for shortening a calculation time for non-photo real (NPR) rendering.SOLUTION: System 300 includes processing hardware 334 and a system memory 336 that stores an NPR renderer 320. Processing hardware receives a digital asset 331, displays a preliminary image corresponding to the digital asset, executes an NPR renderer to receive edit data identifying an NPR effect, and generates an NPR image 352 using the preliminary image and the edit data 332 in an integrated rendering and compositing process in real-time with respect to receiving the edit data.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] This application is a continuation of pending provisional patent application No. 63 / 281, filed November 19, 2021. No. 537, and the name "real-time non-photorealistic rendering" The benefit of and priority to the present application is claimed, and the present application is incorporated by reference in its entirety. . [Background technology]

[0002] The game engine must be able to run at the highest quality possible at high frame rates to allow for user interaction. Graphics enthusiasts focused on achieving high-quality photorealistic rendering In contrast, non-photorealistic (NPR) rendering is traditionally built on top of Rendering refers to a variety of approaches to rendering engineering. It does not mimic real-life images, but instead creates a specific look that mimics different format media. Focus on creating a unique aesthetic look or style that has never been seen before. However, you can use an offline renderer to direct your art to appear as a Existing solutions for performing NPR rendering typically require the use of compositing software. As well as several rendering passes compiled in software, it can take hours per frame. This requires undesirably long rendering times. NPR rendering solution that enables real-time generation of NPR images at high frame rates There is a need in the art for solutions. Summary of the Invention [Problem to be solved by the invention]

[0003] The following description contains specific information for implementing the present invention. It will be appreciated that the present invention may be implemented in ways other than those specifically discussed herein. The drawings and their accompanying detailed description are directed to exemplary implementations only. Unless otherwise noted, the same or corresponding elements among the figures are indicated by the same or corresponding reference numerals. Furthermore, the drawings and figures in this application are generally not to scale and may not represent actual relative dimensions. It is not intended to correspond to relative dimensions.

[0004] As mentioned above, game engines traditionally use high frame rates to allow for user interaction. Focused on achieving the highest quality photorealistic rendering possible at a reasonable price It has been built on a graphics foundation. Non-photorealistic (NPR) rendering is a different type of rendering engineering. It refers to an approach that does not imitate real-life images, but instead uses different formats. Focus on creating a certain look that mimics a certain set or medium, or something you've never seen before. Recreate images that can be turned to art to achieve an unparalleled aesthetic look and style. An example of this is the rendering of a three-dimensional (3D) model and lighting. Create a graphics pipeline that creates a watercolor or comic book look for your To create these types of looks with a traditional rendering process, There are graphics packages on the market that do not provide all the components required to achieve these looks. The package covers all elements in a cohesive package, allowing for real-time high frame rates. It is not present in the rendering. [Means for solving the problem]

[0005] This application aims to reduce the calculation time for NPR rendering. Customized or partially customized software configured to perform real-time NPR rendering It works as a plugin for both customized game engines and traditional game engines. It uses a custom NPR renderer that allows for real-time artistic image compositing. Modify the stock renderer included with the game engine to achieve the final render NPR software plugins and tooling solutions that require low overhead for The concepts disclosed herein are applicable to long-term video processing, such as several hours per frame. Existing solutions created for offline renderers that require long rendering times Some renderings compiled in compositing software, well beyond Furthermore, in some implementations, the systems disclosed herein may be The systems and methods can be substantially or fully automated.

[0006] As defined for purposes of this application, "automated," "automated," and "automating" are The term "systems and processes that do not require the participation of a human system administrator" In some embodiments, a human editor or artist may Images synthesized by the described automated systems and methods can be analyzed without human intervention, e.g. Enter any editorial or artistic input, within 60 seconds or an additional 10 seconds Or it can be reviewed in real time, such as within less than 10 seconds. In some implementations, the methods described herein may be implemented using the hardware of the disclosed automation system. The process may be performed under the control of a software processing component. [Brief explanation of the drawings]

[0007] [Figure 1] 1 illustrates a conventional pipeline for generating three-dimensional (3D) renderings. [Figure 2] We present an exemplary real-time pipeline for generating non-photorealistic (NPR) images according to one implementation. [Figure 3] 1 shows a diagram of an exemplary system for performing real-time NPR rendering, according to one implementation. [Figure 4] 1 shows a flowchart describing an exemplary method for performing real-time NPR rendering according to one implementation. DETAILED DESCRIPTION OF THE INVENTION

[0008] Figure 1 shows a traditional pipeline for generating 3D renderings. As shown, the conventional pipeline 100 is a 3D digital content creation package. 102, traditional 3D rendering packages 104, multiple rendering passes 106, The example shown in FIG. The implementation depicts four rendering passes, but like the traditional pipeline 100 In traditional pipelines, the creative complexity of the content being generated requires multiple There may be more or less than four rendering passes between the number of rendering passes 106. Note that this may occur.

[0009] As shown in FIG. 1, multiple rendering passes 106 and compositing packages 108 It is further noted that this represents post-processing of the output of a conventional 3D rendering package 104. Multiple rendering passes 106, their compositing using a compositing package 108 required to generate rings and improve the performance of traditional 3D rendering packages 104 Due to the significant processing resources required, the generation of the final output rendering 110 is performed in a time-consuming manner, either as an image or may require multiple times per video frame.

[0010] Referring to FIG. 2, FIG. 2 shows a real-time display according to one implementation of the novel and inventive concepts of the present invention. 2 shows a real-time pipeline 200. As shown in FIG. 200, 3D Digital Content Creation Package 202, NPR Software Plug-in Extensions including 3D real-time rendering package 224 provided using application 250 3D rendering package 220 (hereinafter referred to as "NPR renderer 220"), and the final Includes output rendering 252.

[0011] In contrast to the traditional pipeline 100, the real-time pipeline 200 is 3D real-time rendering package 22 under the control of software plugin 250 4, the rendering pass 106 and the compositing pass 107 are organized as instructions executed by the Note that the cage 108 is advantageously integrated, i.e., the rendering path 106 and package 108 in the post-processing sequence in the conventional pipeline 100. The actions performed by creating the image are advantageously performed in real time, not during post-processing. 3D real-time rendering package, which runs in the system pipeline 200. 224 and NPR software plug-in 250 during rendering. Therefore, the real-time pipeline 200 provides content artists and other content creators with Authors can now create their own custom-made videos in a fraction of the time that traditional processes would require, and with substantially less effort. Allows you to create unique styles and "unrealistic" looks in just a few steps .

[0012] The conventional 3D rendering package 104 shown in FIG. The game engine is designed to create realistic images based on the physics of light transport phenomena. Note further that in some implementations, the NPR software Software Plugin 250 modifies the performance of the stock game engine to improve the appearance of NPR effects. 3D can generate "unrealistic" images with artistic decoration and modifications A real-time rendering package 224 may be provided. Alternatively, some In this implementation, NPR software plug-ins 250 are used to create custom shading effects. A game engine whose source code has been modified to perform some NPR rendering functions such as It may be advantageous or desirable to use a hybrid configuration with a

[0013] FIG. 3 illustrates an exemplary diagram for performing real-time NPR rendering, according to one implementation. 3 shows a diagram of a system 300. As shown in FIG. 3, the system 300 includes processing hardware 3 a computing platform 332 having a computer-readable non-transitory storage device 34; and optionally, a system memory 336 implemented as a storage medium, and It may include a database 327, a display 362, and an input device 363. As further shown in FIG. 3, the system memory 336 is Package 302 and the extended 3D rendering package 320 (hereinafter referred to as the "NPR Renderer" The NPR renderer 320 stores the game engine (hereinafter referred to as "320") in FIG. A graphical user interface (GUI) is provided in the form of a game engine 324. )326 provides a 3D real-time rendering package 224 and a game engine It acts as a plugin to the 324, modifies the game engine 324, and uses the NPR renderer 32 Configured to modify GUI326 to allow control of 0 tunable parameters In some implementations, the game engine The program 324 is a conventional game engine such as that included in the conventional 3D rendering package 104 of FIG. Note that the implementation may also be a game engine. The performance of the game engine 324 is based on a custom rendering engine to provide NPR effects. Modifying the stock rendering engine included with Game Engine 324 by Jin The GUI 326 is modified by an NPR software plug-in 350 configured to: The GUI provided by the stock game engine 324 may also be included. No need to render multiple passes and adjust variables in compositing software, just Authors can manipulate the appearance in real time with the enhanced 3D rendering package 320, Added by NPR software plugin 350 to allow you to define It contains editing tools and additional interface panes. However, in some implementations ,NPR software plug-in 350, some of the custom shading, Game Engine 32 with source code modified to perform some NPR rendering functions It may be advantageous or desirable to use a hybrid configuration with In some implementations, the NPR renderer 320 performs some of the rendering or compositing of the NPR image 352. A customized game with source code containing some instructions for at least one The game engine 324 may be in the form of a game engine, while in other implementations, NPR Software Plug-in 350 is a complete solution for rendering and compositing NPR images 352. The instruction may include:

[0014] In addition to the features described above, in some implementations, the NPR renderer 320 may 0, and further configured to predict parametric settings accordingly. One or more machine learning (ML) models configured to improve the performance of the model over time. 28 (hereinafter referred to as "ML model 328"). As such, the phrase "machine learning model" or "ML model" refers to a sample of data or A mathematical model that makes future predictions based on patterns learned from "training data" Various learning algorithms can be used to find the relationship between input data and output data. These correlations can be used to map future predictions for new input data. Such predictive models may be used to create mathematical models that can be used to make predictions. Logistic regression models, Bayesian models, or neural networks (NNs) Furthermore, in the context of deep learning, a "deep neural network" is a network consisting of an input layer and It can refer to a neural network that uses multiple hidden layers between the output layer, which is clear in the raw data. This may allow learning based on features that are not explicitly defined. Thus, the feature identified as NN refers to a deep neural network. In this case, NNs can be trained as classifiers and can be used for image processing, audio processing, or natural language processing. It can be used to perform

[0015] As further shown in FIG. 3, the system 300 includes a communication network 338 and a display 342 and an input device 343, and a user system 340 using the user system 340. A user 344 who uses the system 300 and a communication network 33 that connects the user system 340 and the system 300. Implemented within a use environment including a network communication link 348 that interactively connects the FIG. 3 also shows a digital asset 331, composite edited data 332 (hereinafter referred to as "edited data"). 332), and systems based on edited data 332 and digital assets 331. the NPR image 352 generated by the system 300 as the final output rendering, and Content creation data 330 is shown identifying the

[0016] The system 300 implements the real-time pipeline 200 as shown in FIG. Therefore, the 3D digital content creation package 302, NPR Renderer 320, Game Engine 324, NPR Software Plugin 350, ML Model 328), and NPR image 352 are generally used in 3D digital content creation in FIG. Synthesis Package 202, NPR Renderer 220, 3D Real-time Rendering Package 22 4, NPR software plug-in 250, and final output render 252 respectively. Therefore, the 3D content creation package 202, the NPR renderer 220, and the 3D real Time Rendering Package 224, NPR Software Plugin 250, and Final The output render 252 may be implemented in a 3D digital content creation package according to the present invention. 302, NPR Renderer 320, Game Engine 324, NPR Software Plug-in 350 , and may share any of the properties attributed to NPR images 352, and vice versa. It seems that

[0017] As mentioned above, in some implementations, the game engine 324 may use the conventional 3D rendering of FIG. The game engine may be a conventional game engine such as that included in the programming package 104, The performance of the 324 rendering engine is a custom rendering engine to provide NPR effects. Fixes the stock rendering engine included with Jin Game Engine 324 The GUI 326 is modified by the NPR software plug-in 350 configured to The GUI provided by the game engine 324 may also include multiple Render passes and adjust variables in compositing software without content control. The author manipulates and defines the appearance in real time with the extended 3D rendering package 320. Editing added by NPR software plugin 350 to enable Contains tools and additional interface panes.

[0018] Alternatively, and as mentioned above, in some implementations, the NPR software plugin 3 50 to perform some NPR rendering functions, such as custom shading. It is used in a hybrid configuration with the game engine 324, whose source code has been modified to allow That is, in some implementations, the NPR renderer 32 0 contains some instructions for rendering and / or compositing NPR images 352. Game Engine 3 in the form of a customized game engine with source code including 24, while in other implementations, the NPR software plug-in 350 may include N It may contain all the instructions for rendering and compositing the PR image 352, but Both the hybrid implementation and the NPR software plugin support real-time animation. and virtual generation, advantageously saving substantial time during project finalization, resulting in faster delivery and Allowing for more creative iterations and future use cases in interactive media Make it possible.

[0019] 2 and 3 in combination, the real-time The NPR image 352 is generated as the final output rendering 252 of the image pipeline 200. Note that NPR images 352 may correspond to a variety of different types of content. Note that examples of content NPR image 352 types include audio and video components. This includes audio-video content with accompanying audio or video without accompanying audio. Alternatively, in some implementations, the type of content contained in the NPR image 352 may be: People, fictional characters, locations, objects, and other content, e.g., virtual reality (VR), augmented reality (AR) or digital representations of identifying names such as brands and logos that make mixed reality (MR) environments popular Such content may be current or may contain personal information. While providing continuity of data such as identity, user history, qualifications, possessions, and payments, It is possible to depict a virtual world that can be experienced by any number of users on a continuous and persistent basis. Additionally, in some implementations, the content contained in the NPR image 352 may be a traditional audio file. Hybrid audio-video and fully immersive VR / AR / MR experiences, such as interactive video It is possible.

[0020] NPR Software Plug-in 250 / 350, NPR Software Plug-in 250 / 350 is a robust customizable format that can be integrated into game engine 324 Through a customized GUI326 configured to provide shader models, Expose control to artists in a user-friendly way, allowing them to create multiple styles Adjust various elements of the predefined look to achieve a temporary graphic object. Add either a graphic overlay or a final frame graphic overlay. The NPR software plug-in 250 / 350 allows you to Custom Diffuse Wrapping, Shadow Edges and Specular Edges using Functions (BDRF) texture decomposition, as well as object edge and camera view-based NPR effects. It can be configured to provide three main features:

[0021] Custom Diffuse Wrapping Features can include custom shading models This itself may involve a diffuse reflectance model based on the Kubelka-Munk theory. For example, how light affects different objects in the frame and interacts with surfaces, and how the objects You can manipulate the way it is rendered on the display. The rendering model provides artistic control via a GUI326, allowing for the finalization of shadows and lighting. Change the hardness of the lines, adjust where they fall, and create an almost matte look. This feature can be used to provide a simplified reflectance scale. The features are also applied after shading and lighting have been rendered (e.g., chromatic aberration, blur, etc.). frame), allowing real-time frame adjustments using the GUI326, and Artistic manipulation of the final image (e.g. texture overlays, style density, pigment Post-process stack integral exposure parameters that allow for provide.

[0022] Texture decomposition of shadow edge and specular edge functions allows for the detection of areas bordering the end line and , various textures that hit specific objects in the shadows, or lack thereof, depending on the light. This feature allows you to control the hardness of shadow or specular edge transitions. Additionally, this feature can be adjusted to provide a texture map for shadowing. Provides the ability to decompose or modify dot or specular edge transitions.

[0023] Object edge and camera view based NPR effect function, color is based on object albedo. View-based, but affected by scene lighting (e.g., using scene color) Furthermore, this feature can provide the internal and external contours of the object (e.g. , adjust one or more of line thickness, line presence, line taper, texture decomposition / variation Provides technically configurable controls for adding color (options, color overrides, and coloring) May contain customized bleed effects, e.g., custom volume colorizers ( i.e., surfaces and volumes) are visualized in 3D through the viewport provided by the GUI 326. Localized coloring of two-dimensional (2D) or 3D regions in space, and in-engine tools Alternatively, or in addition, the drawing may be used to Precision oil pen for overpainting, 2D FX, smear frame motion blur, artist Allows users to draw 2D lines in 3D space for FX, animation, and note-taking. Vector drive lines can be inserted into the image via the GUI 326 and locked to 3D elements, One or more 2D animations can be provided within a 3D scene.

[0024] With respect to the depiction of system 300 shown in FIG. 3, the 3D digital content creation package The page 302, NPR renderer 320, and NPR image 352 are shown in system format for conceptual clarity. Although depicted as being stored in system memory 336, it is more commonly It should be noted that library 336 may take the form of any computer-readable non-transitory storage medium. As used herein, the term "computer-readable non-transitory storage medium" means a medium that is Computing hardware such as processing hardware 334 of the operating platform 332 Excluding carrier waves or other transient signals that provide instructions to the processing hardware of the operating platform. Thus, a computer-readable non-transitory storage medium refers to any medium, e.g., a volatile The present invention can support various types of media, including volatile media and non-volatile media. Dynamic Random Access Memory (Dynamic RAM) Non-volatile memory may include optical, magnetic, or electrostatic storage. Common forms of computer-readable non-transitory storage media include, for example, , optical disk, RAM, programmable read-only memory (PROM), erasable PROM (EPROM) ), and flash memory.

[0025] FIG. 3 illustrates a 3D digital content creation package 302, an NPR renderer 320, and an NP R image 352 is a representation provided solely as an aid to conceptual clarity. Furthermore, the data are depicted as being co-located with one another in the system memory 336. More generally, system 300 may be co-located, e.g. One or more computing platforms, such as computer servers, that can This can include interactive systems, such as cloud-based systems. As a result, a linked but distributed system can be created. The processor 334 and system memory 336 are the components of the distributed processor and memory in the system 300. Therefore, the 3D digital content creation package 3 02, NPR renderer 320, and NPR image 352 are stored in the distributed memory resources of system 300. It should be understood that they may be stored remotely from each other.

[0026] The processing hardware 334 may include, for example, one or more central processing units, one or more graphics processors, a digital processing unit, one or more tensor processing units, one or more field processors Programmable Gate Array (FPGA) and application programming interface It may contain multiple hardware processing units, such as an API server. As used herein, "central processing unit" (CPU), "graphics processing unit" The terms "GPU," "Tensor Processing Unit," and "Tensor Processing Unit" are commonly used in the art. CPUs have their customary meanings in computing platforms. an arithmetic logic unit (ALU) for performing arithmetic and logical operations of form 332; 3D Digital Content Creation Package 302, Game Engine 324, NPR Software Plug-ins 350 and programs such as ML models 328 from system memory 336 The GPU contains a control unit (CU) for searching the graphics system, while the GPU is used for processing the graphics to reduce CPU processing overhead by performing other processing tasks. TPUs may be implemented for artificial intelligence (AI) applications such as machine learning modeling. It is an application specific integrated circuit (ASIC) specifically configured for

[0027] In some implementations, the computing platform 332 may be, for example, an internet to one or more web servers accessible over a packet-switched network such as Alternatively, the computing platform 332 may Supports wide area networks (WANs), local area networks (LANs), or or other type of limited distribution or private network However, in some implementations, the system 3 00 can be implemented virtually anywhere, such as in a data center. For example, in some implementations: The system 300 may be implemented in software or as a virtual machine. In some implementations, the communications network 338 may be, for example, a 10 GigE network or High-speed networks suitable for high-performance computing (HPC), such as InfiniBand networks It may also be a network.

[0028] The user system 340 is shown in FIG. 3 as a desktop computer. However, the display is provided merely as an example. More generally, the user system 0 provides a user interface and supports connection to a communications network 338 and data sufficient to implement the functionality attributed to user system 340 herein. Any suitable mobile or stationary computing device implementing data processing functionality. For example, in another embodiment, the user system 340 For example, a laptop computer, a tablet computer, or a smartphone Additionally, in some implementations, the user system 340 may take the form of a system The system can be a "dumb terminal" peripheral workstation for the 300.

[0029] In various embodiments, the enhanced 3D rendering package 320 is used to generate the The resulting NPR image 352 may be stored in system memory 336 or in non-volatile storage. Note that the file may be copied to the . In some embodiments, as shown in FIG. 3, the NPR image 352 is provided by a system 300, or may be displayed on the display 362, or may be transmitted over a communication network, for example. 338 via network communication link 348 to 342.

[0030] Regarding the display 342 of the user system 340, the display 342 It may be physically integrated with the system 340 or communicatively connected to the user system 340. For example, the user system 340 may be As a smartphone, laptop, or tablet computer If implemented, the display 342 is typically integrated with the user system 340. In contrast, if the user system 340 is implemented as a desktop computer, The display 342 may be a separate monitor from the user system 340, forming a computer tower. The user system 340 can be a "dumb terminal" of the system 300. In an implementation that is a "perfect" peripheral workstation, the user system 340 and the The play 342 may be controlled by the processing hardware 334 of the system 300. Note that the display 342 of the user system 340, as well as the system The display 362 of 300 may be a liquid crystal display (LCD), a light emitting diode display (LED), or the like. LED, Organic Light Emitting Diode Display (OLED), Quantum Dot Display (QD) , or any other suitable display screen that performs the physical conversion of signals into light. It may be implemented.

[0031] Also, although the input device 343 of the user system 340 is depicted as a keyboard, Note that the representation is merely an example. More generally, the system 300 Input device 343, as well as input device 363, may be a touch screen, to name a few. , touchscreen in combination with a pen or stylus, trackpad, mouse, or Alternatively, it may take the form of a voice command input (eg, a microphone).

[0032] The functionality of a system 300 implementing the real-time pipeline 200 of FIGS. 2 and 3 is ,An exemplary method for performing real-time NPR rendering, according to one embodiment This is further explained with reference to FIG. 4, which shows a flowchart 470 illustrating the Regarding the operations described, specific details and features may obscure the discussion of inventive features in this application. Note that the IF-IF data is omitted from flowchart 470 to avoid confusion.

[0033] Referring to FIG. 4 in combination with FIG. 3, a flowchart 470 illustrates the digital asset 331 The digital asset 331 may be, for example, a photo or video. Alternatively, the digital asset 331 may be a digital image such as a frame of a movie. It may be a video sequence, or a 2D or 3D digital model. The digital asset 331 is received at operation 471 as part of the content creation data 330. In those implementations, the digital asset 331 may be a 3D digital content. can be extracted from content creation data 331 by content creation package 302. , and is transferred by the 3D digital content creation package 302 to the NPR renderer 320. Alternatively, in some implementations, the content creation data 330 may simply be It is possible to identify digital assets 331. In these implementations, 3D digital In response to receiving the content creation data 330, the content creation package 302 Digital assets 331 can be obtained from the digital asset database 337, and NPR Len Digital assets can be provided as input to the Dara 320. Digital assets 331 can be , by the NPR renderer 320 executed by the processing hardware 334 of the system 300. For example, the NPR software plug-in 350 may process The hardware 334 executes and controls the game engine 324 to perform operations 471 The digital asset 331 can be received at

[0034] Flowchart 470 further includes displaying a preliminary image corresponding to digital asset 331 (operation 4 72). Operation 472 is performed by NPR renderer 320 and includes The processing hardware 334 executes the process and uses a GUI to display the system 300. A preliminary image is displayed via a ray 362 or a display 342 of the user system 340. It is possible.

[0035] Flowchart 470 further includes receiving edit data 332 identifying NPR effects (operation 47). The edited data 332 is processed by the processing hardware 334 of the system 300. 3. For example, the NPR renderer 320 may receive the NPR data in operation 473. As shown in FIG. 1, the edited data 332 is input from the user system 340 using the input device 343. GUI 32 in response to input provided to user system 340 by user 344 6. Flowchart 470 identifies a "single" NPR effect The reception of edited data 332 is described, but the term "single" is used for conceptual clarity. In some implementations, the edited data 332 may include more than a single NPR effect (i.e., two or more Note that it is possible to distinguish between the NPR effect and the

[0036] For example, the NPR effect identified by editing data 332 is the effect of light on the video frame. How it affects the various objects in the room, interacts with the surface, and It may also include how the image is rendered on the display. The NPR effect identified by 2 changes the hardness of shadows and lighting end lines, It may also include artistic controls to adjust where it falls. Additionally or alternatively, edit data 3 The NPR effect, identified by 32, is a result of the way object outlines appear on a display. In one use case, the edit data 332 can include a description of what should be displayed. The identified NPR effects affect not only the lighting and rendering system but also the stylization of objects. (e.g., scene depth-driven operations, object proximity and overlap operations).

[0037] In some implementations, the NPR effects identified by the edit data 332 may be applied to objects that are hit by the NPR effects. This allows for the application of different textures to areas that are bordered by the end line and are in shadow depending on the light. According to some embodiments, the edit data The NPR effect, identified by 332, is the light blue bleeding seen in comic books. Or you can use various shadings on an object to manipulate its appearance, such as halftone patterning. In some implementations, the editing data 332 may include textures that are applied to the image. NPR effects, identified by the It may include adjustments to the frame (e.g., chromatic aberration, bloom, etc.) and may also include adjustments to the final image. Artistic manipulation of the image (e.g., texture overlay, style density, pigment darkening / blurring) In some implementations, the edit data 332 may The identified NPR effects are displayed in 3D space via the viewport provided by GUI326. May include localized coloring of 2D or 3D regions, and the use of in-engine tooling Additionally, in some embodiments, the NPs identified by compiling data 332 R Effects include depth for drawover, 2D FX, and smear frame motion blur The result can be produced using oil paint, FX, animation, and Creates 2D lines in 3D space for note-taking. Vector-driven lines are GUI326 Insert one or more 2D animations into a 3D scene by inserting them into an image via the communication can be provided.

[0038] Flowchart 470 further includes a displayed preliminary image at operation 472 and a received preliminary image at operation 473. The edited data 332 is used to render the edited data in an integrated rendering and compositing process. generating NPR images 352 in real time with respect to receiving data 332 (operation 474). The NPR image 352 is processed by the processing hardware 334 of the system 300. The NPR renderer 320 may be executed in operation 474. For example, the NPR software The software plug-in 350 is executed by the processing hardware 334 of the system 300, Controlling the game engine 324 to display the preliminary image displayed in operation 472 and the received image in operation 473 The received editing data 332 is used to create the NPR image 35 through integrated rendering and compositing. 2 can be generated.

[0039] Generate NPR images 352 in real time upon receiving edit data 332 In this regard, as described above, in some embodiments, the NPR renderer 320 may 473 to generate NPR image 352 within 60 seconds of receiving edit data 332 Further, as noted above, in some embodiments, the NPR renderer 320 generates an NPR image 352 within 10 seconds or less after receiving edited data 332. For example, in some embodiments, the NPR renderer 320 may be configured to It is possible to generate NPR images corresponding to NPR image 352 at a frame rate of 24 frames per second. It is possible.

[0040] In some embodiments, the rendering performed as part of act 474 is performed using Kubelka - A diffuse reflectance model based on Munk's theory can be used. Typically, the compositing performed as part of operation 474 may be performed using, for example, a Kuwahara filter or correction A non-linear smoothing filter such as one of the Kuwahara filters may be used.

[0041] As described above with reference to FIG. 3, in some embodiments, the NPR renderer 320 , based on the edit data 332 and the digital assets 331, to generate the NPR image 352. NPR renderer 3 is used when rendering and compositing in action 474. One trained to predict one or more operating parameters for use by 20 The NPR renderer 320 may further include the above machine learning model 328. The inclusion and use of one or more machine learning models 328 may be used to generate a digital asset 331 However, it is difficult to analyze complex digital assets, such as high mesh element count 3D digital models. or several NPR effects, or NPR effects that interact with each other, are compiled by the edit data 332. This can be particularly important in use cases where NPR records are identified. The number of possible permutations for the operating parameters of the driver 320 is too large for humans to effectively evaluate. It can far exceed the capabilities of a human mind, or even a typical computer processor. In those implementations, the NPR renderer 320 performs the real-time NPR described herein. One or more predictive machine learning models to achieve performance improvements, including PR rendering. 28 performance can be relied upon.

[0042] Regarding the method outlined by flowchart 470, in some implementations, Actions 471, 472, 473 and 474 are automated processes from which human participation may be omitted. Note that the method may be implemented in a process.

[0043] Therefore, the present application provides a real-time NPR rendering method that reduces the calculation time for NPR rendering. As mentioned above, several systems and methods for performing filtering are disclosed. In an embodiment, the real-time NPR rendering solution of the present invention is It can act as a plugin to the engine and perform real-time artistic image composition. Using a custom NPR renderer that allows you to override the stock renderer included with the game engine Provides a software plugin to fix NPR, allowing for low-quality output for final rendering. In addition, the custom NPR renderer disclosed in this application requires a lot of overhead. stylized, physically based, or artistically oriented outside the scope of previously developed Advantageously, this provides flexibility in achieving aesthetic appearances. As a result, the concepts disclosed herein are Not just a few render passes compiled in software, but hours per frame existing offline renderers that require long rendering times, such as Advance state-of-the-art solutions significantly beyond existing solutions.

[0044] From the above description, it will be appreciated that various techniques may be employed to implement the present invention without departing from the scope of those concepts. It is clear that the concepts described herein can be used to implement the concepts described herein. Although specifically described with reference to the above-described devices, those skilled in the art will recognize that they may depart from the scope of these concepts. It will be recognized that changes may be made in form and detail without departing from the spirit and scope of the present invention. The described implementations are to be considered in all respects only as illustrative and not restrictive. Furthermore, the present application is not limited to the specific implementations described herein, but rather may be implemented in many different ways. It is understood that configurations, modifications, and substitutions are possible without departing from the scope of the present invention. I want to be.

Claims

1. System memory that stores the processing hardware and the non-photorealistic (NPR) renderer. A system comprising: The processing hardware includes: Receive digital assets; displaying a preliminary image corresponding to the digital asset; receiving editing data identifying an NPR effect; an integrated rendering and compositing process using the preliminary images and the edited data; and generating an NPR image in real time with respect to receiving said editing data. 、 A system configured to execute the NPR renderer.

2. The NPR renderer interacts with the game engine to render and composite the NPR images. NPR software including instructions configured to modify the performance of said game engine so as to and a software plug-in.

3. The game engine performs at least one of rendering and compositing the NPR images. A customized game engine with source code containing some instructions for The system of claim 2 , comprising:

4. The NPR software plug-in is used for rendering and compositing NPR images. The system of claim 2 including all instructions.

5. The integrated rendering and compositing process uses the Kubelka-Munk method for rendering. The system of claim 1 , wherein the system uses a theory-based diffuse reflectance model.

6. The integrated rendering and compositing process uses a nonlinear smoothing filter for compositing. The system of claim 1 .

7. The nonlinear smoothing filter is a Kuwahara filter or a modified Kuwahara filter. The system of claim 6, wherein the number of the plurality of sensors is one.

8. The NPR renderer generates the NPR image within 60 seconds of receiving the edit data. The system of claim 1 , configured to:

9. The NPR renderer renders the NPR image based on the editing data and the digital assets. one or more animations for use by the NPR renderer when rendering and compositing and at least one machine learning model trained to predict the operational parameters. The system of claim 1 .

10. Processing hardware and system memory that houses the non-photorealistic (NPR) renderer and a method of using the system, the method comprising: The NPR renderer executed by the processing hardware receives the digital asset. The step of believing, The NPR renderer executed by the processing hardware renders the digital asset. displaying a preliminary image corresponding to receiving editing data identifying an NPR effect; The NPR renderer executed by the processing hardware processes the preliminary images and edits the and the collected data to receive the edited data in an integrated rendering and compositing process. and generating an NPR image in real time with respect to receiving the NPR image.

11. The NPR renderer interacts with the game engine to render and composite the NPR images. NPR software including instructions configured to modify the performance of said game engine so as to and a software plug-in.

12. The game engine performs at least one of rendering or compositing the NPR images. a customized game engine having source code containing some instructions for The method of claim 11 , comprising:

13. The NPR software plug-in is used for rendering and compositing NPR images. The method of claim 11 including all instructions.

14. The integrated rendering and compositing process uses Kubelka-Mun The method of claim 10, using a diffuse reflectance model based on k-theory.

15. The integrated rendering and compositing process uses a nonlinear smoothing filter for compositing. The method of claim 10, wherein

16. The nonlinear smoothing filter is a Kuwahara filter or a modified Kuwahara filter.

16. The method of claim 15, wherein the

17. The NPR renderer generates the NPR image within 60 seconds of receiving the edit data. The method of claim 10 , configured to:

18. The NPR renderer renders the NPR image based on the editing data and the digital assets. One or more motion patterns for the NPR renderer to use when rendering and compositing. and further comprising at least one machine learning model trained to predict the parameter. The method of claim 10.

19. Non-Photorealistic (NPR) software plug-in containing computer-readable non- a temporary storage medium, wherein the NPR software plug-in is When performed, the method produces steps of: Controlling a game engine to receive the digital assets. Displaying a preliminary image corresponding to the digital asset. receiving editing data identifying an NPR effect; using said preliminary images and edit data in an integrated rendering and compositing process; and generating an NPR image in real time with respect to receiving the editing data. and controlling a game engine.

20. The NPR software plug-in receives the edited data within less than 60 seconds. and controlling the game engine to generate the NPR image in accordance with the 20. The computer-readable non-transitory storage medium of claim 19.

Images