Neural network based image coloring on image / video editing applications

Abstract

Computing systems and methods for neural network-based image colorization are provided. A computing system obtains a reference color image by selective application of a color effect on a region of interest of an input image and controls a display device to display a first node graph on a graphical user interface of an image / video editing application. The first node graph includes a colorization node representing a first workflow for colorizing at least a first object in a grayscale image of a first image feed. The computing system selects the reference color image based on user input and executes the first workflow associated with the colorization node by feeding the reference color image and the first image feed as inputs to a neural network-based colorization model. The computing system receives a second image feed including a colorized image as an output of the neural network-based colorization model for the inputs.

Description

[0001] Cross-reference to related applications / incorporation by reference

[0002] none. Technical Field

[0003] Various embodiments of this disclosure relate to image / video colorization. More specifically, various embodiments of this disclosure relate to neural network-based image colorization methods and systems for image / video editing applications. Background Technology

[0004] Advances in image / video editing applications have led to the development of image colorization techniques, which add color to one or more regions in a grayscale image / video. In conventional rotoscoping-based methods, users must manually place dots around the region of interest within image frames of a video. To colorize an entire sequence of frames in a video, this process must be manually repeated for each frame. This can be time-consuming and cumbersome for users.

[0005] By comparing the described system with some aspects of this disclosure, the further limitations and disadvantages of conventional and traditional methods will become clear to those skilled in the art, as set forth in the remainder of this application and with reference to the accompanying drawings. Summary of the Invention

[0006] A computational system and method for neural network-based image colorization for image / video editing applications are provided, which are basically illustrated in at least one figure and / or described in combination with at least one figure.

[0007] These and other features and advantages of this disclosure can be understood by reading the following detailed description of the disclosure and the accompanying drawings, in which the same reference numerals always denote the same parts. Attached Figure Description

[0008] Figure 1 This is a block diagram illustrating an exemplary network environment for neural network-based image colorization in an image / video editing application according to an embodiment of the present disclosure.

[0009] Figure 2 This is a block diagram illustrating an exemplary computing system for neural network-based image colorization in image / video editing applications according to embodiments of the present disclosure.

[0010] Figure 3 This is a diagram illustrating an exemplary operation of coloring a grayscale image feed using a neural network-based coloring plugin in an image / video editing application according to an embodiment of the present disclosure.

[0011] Figure 4The illustration shows an embodiment of the present disclosure for obtaining a reference color image for use in... Figure 3 A diagram of the exemplary operations used in the exemplary operations.

[0012] Figure 5 This is a flowchart illustrating an exemplary method for neural network-based image colorization in an image / video editing application according to embodiments of the present disclosure. Detailed Implementation

[0013] The embodiments described below can be found in the disclosed computational systems and methods for neural network-based image colorization in image / video editing applications. An exemplary aspect of this disclosure provides a computational system that uses a neural network-based image colorization model to implement a workflow for colorizing at least one object in one or more grayscale images of an image feed using a reference color image. Specifically, the neural network-based colorization model transfers the color effect of a region of interest in the reference color image to an object in each grayscale image of the image feed. This image feed can be a single image or a video containing multiple images.

[0014] In image / video editing applications, users can use the app's rotoscoping tools to obtain a reference color image. For example, a user can use rotoscoping to change the color of an occluded area of ​​an image to obtain a reference color image. The same application provides a node-based interface for constructing node graphs. For example, a user can simply add a grayscale image feed as a source node to a colorizing node, whose output can be linked to a result node. Colorizing nodes can correspond to software plugins (such as the OpenFX (OFX) plugin), which, when executed, invokes a neural network-based colorizing model to apply color effects from regions of interest in the reference color image to objects in one or more grayscale images fed to the image feed. The reference color image or its file path can be passed as input to the colorizing node.

[0015] In conventional rotoscoping-based methods, users must manually mark points around regions of interest (ROIs) in image frames of a video. To colorize an entire sequence of frames in the video, this process must be repeated for each frame. This can be time-consuming and cumbersome. In contrast, this disclosure does not require users to mark any ROIs in each frame of the target grayscale video. Users may simply select a reference color image and a target image or video (with one or more grayscale images) as input to a colorization node on a node-based interface of an image / video editing application. The colorization node can represent a workflow that, when executed, invokes a neural network-based colorization model to transfer color effects from the ROI in the reference color image to one or more objects in the frames of the target image / video.

[0016] Figure 1 This is a block diagram illustrating an exemplary network environment for neural network-based image colorization in image / video editing applications according to embodiments of the present disclosure. Reference Figure 1 The diagram illustrates a network environment 100. The network environment 100 may include a computing system 102 and a display device 104 communicatively coupled to the computing system 102. An image / video editing application 106 is also shown, which may be installed on the computing system 102 or accessed via a web client (such as a web application or web browser) on the computing system 102.

[0017] The network environment 100 may also include a server 110, which can implement a neural network-based coloring model 108. The server 110 can be communicatively coupled to the computing system 102 via a communication network 112. Figure 1 In this disclosure, the computing system 102 and the display device 104 are shown as two separate devices; however, in some embodiments, the full functionality of the display device 104 may be incorporated into the computing system 102 without departing from the scope of this disclosure.

[0018] The computing system 102 may include suitable logic, circuitry, code, and / or interfaces that can be configured to execute a first workflow for colorizing one or more objects in a first image feed comprising one or more grayscale images (such as grayscale image 114). The first workflow may be associated with a software plug-in that may include program instructions for executing the first workflow on an image / video editing application 106. Examples of the computing system 102 may include, but are not limited to, image / video editing machines, servers, computer workstations, mainframes, gaming devices, smartphones, mobile phones, laptops, tablets, extended reality (XR) headsets, and / or any other consumer electronics (CE) devices with image / video editing capabilities.

[0019] Display device 104 may include suitable logic, circuitry, and / or interfaces that can be configured to display a graphical user interface (GUI) 116 of image / video editing application 106. In one embodiment, display device 104 may be a touch-enabled device that allows a user to provide user input via display device 104. Display device 104 may include display units that can be implemented using a number of known technologies, such as, but not limited to, liquid crystal display (LCD), light-emitting diode (LED) display, plasma display, or organic LED (OLED) display technology or at least one of other display technologies.

[0020] Image / video editing application 106 may include suitable logic, code, and / or interfaces that can be configured to edit image feeds comprising one or more grayscale images. For example, editing may include applying color effects to an input image using a transcribing method to obtain a reference color image (e.g., reference color image 118). Another editing may include transferring color effects from the reference color image to objects in a grayscale image of the image feed (e.g., grayscale image 114) using a software plugin that can implement a neural network-based colorization model 108. Image / video editing application 106 may be implemented based on a node graph architecture. In a node graph architecture, a user can construct a node graph to represent a workflow for any editing task, such as colorization of an image feed based on a reference color image (e.g., reference color image 118). Examples of image / video editing application 106 may include, but are not limited to, node-based digital compositing and visual effects applications, image editors, digital effects applications, motion graphics editing applications, compositing applications, non-linear editing (NLE) applications, raster graphics editors, or combinations thereof.

[0021] The neural network-based coloring model 108 can be an image coloring model that can be trained on an image coloring task to color objects in a single image frame or a sequence of image frames fed by an image feed. The neural network-based coloring model 108 can be defined by its hyperparameters, such as activation functions (one or more), multiple weights, cost functions, regularization functions, input size, number of layers, etc.

[0022] The neural network-based coloring model 108 can be referred to as a system of computational networks or artificial neurons (also called nodes). The nodes of the neural network-based coloring model 108 can be arranged in multiple layers, as defined in the neural network topology of the neural network-based coloring model 108. The multiple layers of the neural network-based coloring model 108 may include an input layer, one or more hidden layers, and an output layer. Each of the multiple layers may include one or more nodes (or artificial neurons, e.g., represented by circles). The outputs of all nodes in the input layer can be coupled to at least one node in one or more hidden layers. Similarly, the input of each hidden layer can be coupled to the output of at least one node in the other layers of the neural network-based coloring model 108. The output of each hidden layer can be coupled to the input of at least one node in the other layers of the neural network-based coloring model 108. The nodes in the last layer can receive input from at least one hidden layer to output a result. The number of layers and the number of nodes in each layer can be determined from the hyperparameters of the neural network-based coloring model 108. Such hyperparameters can be set before or simultaneously with training the neural network-based coloring model 108 on a training dataset of images.

[0023] Each node of the neural network-based coloring model 108 can correspond to a mathematical function (e.g., a sigmoid function or a rectified linear unit) having a set of parameters that can be tunable during network training. The parameter set may include, for example, weight parameters, regularization parameters, etc. Each node can use the mathematical function to compute an output based on one or more inputs from nodes in one or more other layers (e.g., one or more previous layers) of the neural network-based coloring model 108. All or some nodes of the neural network-based coloring model 108 can correspond to the same or different mathematical functions.

[0024] In the training of the neural network-based coloring model 108, one or more parameters of each node of the neural network-based coloring model 108 can be updated based on whether the output of the last layer for a given input (from the training dataset) matches the loss function used for the neural network-based coloring model 108 correctly. The above process can be repeated for the same or different inputs until the minimum of the loss function is reached and the training error is minimized. Several training methods are known in the art, such as gradient descent, stochastic gradient descent, batch gradient descent, gradient boosting, metaheuristics, etc.

[0025] In an embodiment, the neural network-based colorization model 108 may include electronic data, which may be implemented as a software component of an application executable, for example, on computing system 102 or server 110. The neural network-based colorization model 108 may rely on libraries, external scripts, or other logic / instructions to be executed by a processing device such as computing system 102 or server 110. The neural network-based colorization model 108 may include computer-executable code or routines to enable a computing device such as computing system 102 or server 110 to perform one or more operations to colorize objects in an input grayscale image. Additionally or alternatively, the neural network-based colorization model 108 may be implemented using hardware including a processor, a microprocessor (e.g., to perform or control the execution of one or more operations), a field-programmable gate array (FPGA), or an application-specific integrated circuit (ASIC). For example, an inference accelerator chip may be included in computing system 102 to accelerate computation of the neural network-based colorization model 108 for image colorization tasks. In some embodiments, the neural network-based colorization model 108 may be implemented using a combination of both hardware and software.

[0026] Examples of neural network-based coloring models 108 may include, but are not limited to, autoencoders, convolutional neural networks (CNNs), regions with CNNs (R-CNN), fast R-CNN, faster R-CNN, You Only See Once (YOLO) networks, residual neural networks (Res-Net), feature pyramid networks (FPN), Retina-Net, and / or combinations thereof.

[0027] Server 110 may include suitable logic, circuitry, and interfaces and / or code that can be configured to implement a neural network-based colorization model 108 for colorizing one or more grayscale images using a reference color image (such as reference color image 118). Server 110 may be a cloud server and can perform operations via web applications, cloud applications, HTTP requests, repository operations, file transfers, etc. Other example implementations of server 110 may include, but are not limited to, web servers, file transfer protocol (FTP) servers, application servers, or mainframe servers.

[0028] In at least one embodiment, server 110 can be implemented as multiple distributed cloud-based resources using several techniques well known to those skilled in the art. Those skilled in the art will understand that the scope of this disclosure is not limited to implementing server 110 and computing system 102 as two separate entities. In some embodiments, the functionality of server 110 may be integrated wholly or at least partially into computing system 102 without departing from the scope of this disclosure.

[0029] Communication network 112 may include a communication medium through which computing system 102 can communicate with server 110 and other devices omitted from disclosure for brevity. Communication network 112 may be either a wired or wireless connection. Examples of communication network 112 may include, but are not limited to, the Internet, cloud networks, Wi-Fi networks, personal area networks (PANs), local area networks (LANs), or metropolitan area networks (MANs). Various devices in network environment 100 may be configured to connect to communication network 112 according to various wired and wireless communication protocols. Examples of such wired and wireless communication protocols may include, but are not limited to, at least one of Transmission Control Protocol and Internet Protocol (TCP / IP), User Datagram Protocol (UDP), Hypertext Transfer Protocol (HTTP), File Transfer Protocol (FTP), ZigBee, EDGE, IEEE 802.11, Li-Fi, 802.16, IEEE 802.11s, IEEE 802.11g, multi-hop communication, wireless access points (APs), device-to-device communication, cellular communication protocols, and Bluetooth (BT) communication protocols.

[0030] In operation, the computing system 102 can control the display device 104 to display the GUI 116 of the image / video editing application 106. The GUI 116 may include a node-based interface to create workflows for image processing tasks, such as image colorization or transcribing tasks. For example, a user might simply place a set of image processing operations as nodes on the node-based interface. Subsequently, to obtain a workflow, these image operations can be linked together by connecting such nodes on the node-based interface. Each of these operations can together generate a node graph.

[0031] First, the computing system 102 can obtain a reference color image (such as reference color image 118) by selectively applying color effects to regions of interest (ROIs) (such as ROI 120) of the input image. The computing system 102 can determine the ROI of the input image based on user input via GUI 116. For example, GUI 116 may include a preview window that can display the input image. User input may include adding multiple points around the ROI in the input image to create a mask that separates the ROI from the rest of the elements in the input image. User input may appear as a first node in the node-based interface of GUI 116. To obtain the reference color image, the computing system 102 can be configured to apply color effects to the ROI of the input image. The color effects may appear as a second node in the node-based interface of GUI 116. In some cases, a set of image filters may be applied to the ROI before or after applying the color effects. Such filters may also appear as one or more nodes connected to the first or second node. All such nodes can form a node graph that represents the entire workflow. When this workflow is executed, Regions of Interest (ROIs) can be selected, and a set of image filters and color effects can be applied to the selected ROIs to obtain a reference color image (such as ROI 120). For example, in Figure 4 Further details regarding the reference color image are provided in the document.

[0032] At any time, a user can create a project on the image / video editing application 106. The computing system 102 can receive user input via a node-based interface of the GUI 116 to construct a first node graph. The computing system 102 can control the display device 104 to display the first node graph on the GUI 116 of the image / video editing application 106. The first node graph may include shaded nodes, which may represent a first workflow for shading at least a first object in one or more grayscale images (such as grayscale image 114) used for a first image feed. The first image feed may include a single image or a sequence of image frames from a video. For example, in... Figure 3 The graph provides detailed information associated with the first node.

[0033] The computing system 102 may receive first user input, which may include selection of a reference color image (such as reference color image 118) via an image / video editing application 106. Thereafter, the computing system 102 may select the reference color image based on the first user input and may execute a first workflow associated with a colorization node. As the first workflow is initiated, the computing system 102 may feed the selected reference color image and a first image feed as input to a neural network-based colorization model 108. The computing system 102 may receive a second image feed as the output of the neural network-based colorization model 108 for this input. The second image feed may include one or more colorized images (such as colorized image 122). Each such colorized image may include at least a first object (such as a T-shirt of a football player 124) colorized based on the color effect on a region of interest (ROI) (such as ROI 120).

[0034] Figure 2 This is a block diagram illustrating an exemplary computing system for neural network-based image colorization in image / video editing applications according to embodiments of the present disclosure. Figure 1 Explanation of elements in Figure 2 . refer to Figure 2 A block diagram 200 of a computing system 102 is shown. The computing system 102 may include a circuit system 202, a memory 204, an input / output (I / O) device 206, and a network interface 208. The circuit system 202 may be communicatively coupled to the memory 204, the I / O device 206, and the network interface 208. In some embodiments, the I / O device 206 may include a display device (such as...) Figure 1 Display device 104).

[0035] Circuit system 202 may include suitable logic, circuitry, and / or interfaces that can be configured to execute program instructions associated with different operations to be performed by computing system 102. Circuit system 202 may include one or more dedicated processing units, which may be implemented as integrated processors or clusters of processors that collectively perform the functions of one or more dedicated processing units. Circuit system 202 may be implemented based on a variety of processor technologies known in the art. Examples of implementations of circuit system 202 may include x86-based processors, graphics processing units (GPUs), reduced instruction set computing (RISC) processors, application-specific integrated circuit (ASIC) processors, complex instruction set computing (CISC) processors, microcontrollers, central processing units (CPUs), and / or other computing circuitry.

[0036] Memory 204 may include suitable logic, circuitry, and / or interfaces configured to store program instructions to be executed by circuitry 202. In at least one embodiment, memory 204 may be configured to store a reference color image (such as reference color image 118) and a first image feed (such as grayscale image 114). Memory 204 may also be configured to store a set of color effects to be used on a region of interest (ROI) of the input image (such as ROI 120). Example implementations of memory 204 may include, but are not limited to, random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), hard disk drive (HDD), solid-state drive (SSD), CPU cache, and / or secure digital card (SD card).

[0037] I / O device 206 may include suitable logic, circuitry, interfaces, and / or code that can be configured to receive input and provide output based on the received input. I / O device 206 may include various input and output devices that can be configured to communicate with circuitry 202. For example, computing system 102 may receive user input via I / O device 206 to select a reference color image, a Region of Interest (ROI) in an input image, and apply color effects to the selected ROI in the input image. Examples of I / O device 206 may include, but are not limited to, a touchscreen, keyboard, mouse, joystick, display device (e.g., display device 104), microphone, or speaker.

[0038] Network interface 208 may include suitable logic, circuitry, interfaces, and / or code that can be configured to facilitate communication between circuitry 202 and server 110, display device 104, and / or other communication devices via communication network 112. Network interface 208 can be implemented using various known technologies to support wireless communication of computing system 102 via communication network 112. Network interface 208 may include, for example, an antenna, radio frequency (RF) transceiver, one or more amplifiers, a tuner, one or more oscillators, a digital signal processor, an encoder-decoder (CODEC) chipset, a subscriber identity module (SIM) card, local buffer circuitry, etc.

[0039] Network interface 208 can be configured to communicate wirelessly with networks such as the Internet, intranets, wireless networks, cellular telephone networks, wireless local area networks (LANs), or metropolitan area networks (MANs). Wireless communication can be configured to use one or more of various communication standards, protocols, and technologies, such as Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), Wideband Code Division Multiple Access (W-CDMA), Long Term Evolution (LTE), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Bluetooth, Wi-Fi (such as IEEE 802.11a, IEEE 802.11b, IEEE 802.11g, or IEEE 802.11n), Voice over Internet Protocol (VoIP), Li-Fi, or Wi-MAX.

[0040] like Figure 1 The functions or operations performed by the computing system 102 described herein can be performed by the circuit system 202. For example, in Figure 3 and 4 The operation performed by the circuit system 202 is described in detail.

[0041] Figure 3 This is a diagram illustrating an exemplary operation for coloring a grayscale image feed using a neural network-based coloring plugin in an image / video editing application, according to an embodiment of the present disclosure. Figure 3 Combination Figure 1 and Figure 2 The elements in the text will be explained. (See reference.) Figure 3 Block diagram 300 is shown, illustrating exemplary operations from 302 to 308 as described herein. The exemplary operations shown in block diagram 300 may begin at 302 and can be performed by any computing system, device, or apparatus, such as by [unclear text - likely a typo]. Figure 1 or Figure 2 The computing system 102 executes.

[0042] At 302, a first image feed can be acquired. Circuit system 202 can acquire the first image feed from a data source. The data source can be, for example, an onboard image sensor of computing system 102, persistent storage on computing system 102, an image capture device, a cloud server, or a combination thereof. The first image feed may include one or more grayscale images (such as grayscale image 302A), each grayscale image may include at least a first object (such as a football player 302B). The first image feed may represent a static scene with a static foreground or background, or it may represent a dynamic scene with one or more moving objects.

[0043] At 304, input can be received to construct a first node graph 304A for coloring a first image feed. For example, the input could be user input, which could include selections of options for creating a project on the image / video editing application 106. When the GUI 116 displays a project window for the created project, the user input could include selections of nodes (such as coloring node 304B, source node 304C, and result node 304D) accessible via a toolbar on the GUI 116 of the image / video editing application 106.

[0044] At 306, the constructed first node graph 304A can be displayed. The circuitry 202 can control the display device 104 to display the constructed first node graph 304A on the GUI 116 of the image / video editing application 106. For example, the GUI 116 may include a node-based interface 306A, which can be updated to include a shaded node 304B between the source node 304C and the result node 304D. While the result node 304D can be linked to the output of the shaded node 304B, the source node 304C can be linked to a first image feed (e.g., including a grayscale image 302A) and a reference color image 306B. The circuitry 202 can select the reference color image 306B based on a first user input.

[0045] The first node diagram 304A may include a shader node 304B, which may represent (or may be linked to) a first workflow for shading a first object (such as a football player 302B) in one or more grayscale images for a first image feed. In an embodiment, the shader node may correspond to a software plugin that may include program instructions for executing the first workflow. A user can select and add this software plugin as a shader node within a node-based interface of the GUI 116.

[0046] In an embodiment, circuitry 202 can control a display device to display settings for shader nodes on the GUI 116 of an image / video editing application 106. For example, the settings can be displayed based on a user's selection of options displayed alongside shader node 304B. These settings may include a set of options corresponding to a set of neural network-based shader models. Circuitry 202 can receive user input, which may include a selection of a first option from the set of options. Based on the received user input, circuitry 202 can select a neural network-based shader model 108 from the set of neural network-based shader models.

[0047] At 308, a first workflow can be executed. At any time, the circuit system 202 can execute the first workflow associated with the shading node 304B. When executed, the circuit system 202 can feed a selected reference color image 306B and a first image feed (such as a grayscale image 302A) as input to the selected neural network-based shading model 108. Subsequently, the circuit system 202 can receive a second image feed as the output of the selected neural network-based shading model 108. For example, the second image feed may include one or more shading images (such as shading image 308A). Each of the one or more shading images may include at least a first object (such as a football player 302B) shading based on color effects on the ROI 306C in the reference color image 306B.

[0048] In one embodiment, a neural network-based shading model 108 can transfer color effects from the ROI 306C of a reference color image 306B to at least a first object (such as a football player 302B) in each of one or more grayscale images in a first image feed to output a second image feed (which includes one or more shading images, such as shading image 308A). As shown, for example, black on the football player's shirt (i.e., ROI 306C) can be transferred to the shirt worn by the football player 302B in the grayscale image 302A of the first image feed. In one embodiment, circuitry 202 can control display device 104 to display the second image feed on the GUI 116 of an image / video editing application 106.

[0049] Although illustrated using discrete blocks, exemplary operations associated with one or more blocks of flowchart 500 may be divided into additional blocks, combined into fewer blocks, or eliminated depending on the implementation of the exemplary operations.

[0050] Figure 4 The illustration shows an embodiment of the present disclosure for obtaining a reference color image for use in... Figure 3 A diagram of the exemplary operations used in the exemplary operations. Figure 4 Combination Figure 1 , 2 Let's interpret the elements in section 3. (See reference.) Figure 4 Block diagram 400 is shown, illustrating exemplary operations from 402 to 410 as described herein. The exemplary operations shown in block diagram 400 may begin at 402 and can be performed by any computing system, device, or apparatus (such as...). Figure 1 or Figure 2 The computing system 102) executes.

[0051] At 402, the input image 402A can be loaded onto the GUI 116 of the image / video editing application 106. The input image 402A can be a color image with at least one object of interest (e.g., a football player 402B).

[0052] At point 404, the ROI 404A of the input image 402A can be selected. This selection can be based on user input via GUI 116. For example, the input image used as the starting point for segmentation can be loaded onto GUI 116, and the segmentation tool of image / video editing application 106 can be used to select ROI 404A. As shown, for example, the ROI surrounding the shirt of the soccer player 402B can be selected by placing points around the edge of the shirt. The selection of ROI 404A can be displayed as a node in the node-based interface of GUI 116.

[0053] At 406, a first set of image filters can be selected for the selected ROI 404A. Such filters may include, for example, tone shift operations, alpha blending, or alpha synthesis operators. These filters can be selected via a menu displayed on the GUI 116 of the image / video editing application 106. In at least one embodiment, such filters can specify the color effect to be applied to the ROI 404A based on modifications to the color values ​​in one or more color channels of the selected ROI 404A.

[0054] At 408, a second workflow can be generated to obtain a reference color image 306B. Operations from 402 to 406 can be performed to generate a second workflow that can be represented by a second node diagram 408A. In an embodiment, the circuit system 202 can control the display device 104 to display the second node diagram 408A on the GUI 116 of the image / video editing application 106. The second node diagram 408A can represent a second workflow for obtaining the reference color image 306B from the input image 402A. The input image 402A and all operations from 402 to 406 can be included as nodes in the second node diagram 408A, and these nodes can be connected together to form the second node diagram 408A.

[0055] At 410, a second workflow can be executed, namely, a node-based workflow associated with the second node diagram 408A. When executed, the circuit system 202 can select a ROI 404A from the input image 402A. For example, the node associated with the selection of ROI 404A can load a mask drawn by the user around the ROI 404A of the input image 402A using a segmentation tool such as a digital transcribing tool. Detailed implementations of transcribing tools are known to those skilled in the art, and therefore, for brevity, a detailed description of transcribing tool 408 is omitted in this disclosure. After making a selection, the circuit system 202 can apply a first set of selected image filters to the selected ROI 404A and can apply color effects to the selected ROI 404A of the input image 402A to obtain a reference color image 306B based on the application of the first set of image filters. As an example and not a limitation, the application of color effects may modify at least one of the following: color saturation, brightness, contrast in the input image 402A, color values ​​in a specific color channel of the selected ROI 404A, and gamma or hue changes in the selected ROI 404A.

[0056] As shown in the figure, for example, the second node 408A can include the input image 402A as a source node, which can be connected to the first merge node (represented by "Merge 1"). The first merge node can also be connected to the ROI node (represented by "ROI"). The first merge node can generate a mask by combining the selection around ROI 404A with the input image 402A. The mask can remove all areas from the input image 402A except for ROI 404A. Image filters (such as filters for modifying color channels (such as green)) can be applied to ROI 404A (which is the output of the first merge node) to obtain the modified ROI. The input image 402A, along with the modified ROI, can be passed as input to the second merge node (represented by "Merge 2"), which can overlay the modified ROI onto the input image 402A to output a reference color image 306B.

[0057] As shown in the figure, for example, the reference color image 306B includes black on a soccer player's shirt (i.e., the selected ROI 404A). Black can be applied as a color effect to ROI 404A of the input image 402A. The image / video editing application 106 can allow the end user to control and modify the reference color image 306B using a transcribing workflow. The reference color image 306B can be stored on the computing system 102 and later used to colorize image feeds of grayscale images, for example, as... Figure 3 As described in [the text].

[0058] Although illustrated using discrete blocks, exemplary operations associated with one or more blocks of block diagram 400 may be divided into additional blocks, combined into fewer blocks, or eliminated, depending on the implementation of the exemplary operations.

[0059] Figure 5 This is a flowchart illustrating an exemplary method for neural network-based image colorization in an image / video editing application according to embodiments of the present disclosure. Figure 5 Combination Figure 1 , 2 Explain the elements in 3 and 4. (See reference.) Figure 5 A flowchart 500 is shown. The method shown in flowchart 500 can be executed by any computing system, such as computing system 102 or circuit system 202. The method can begin at 502 and proceed to 504.

[0060] At 504, a reference color image (such as reference color image 118) can be obtained. In one or more embodiments, circuitry 202 can be configured to obtain the reference color image by selectively applying color effects to the ROI (such as ROI 120) of an input image (such as input image 402A). For example, in Figure 4 The details related to the reference color image are described in the text.

[0061] At 506, a display device (such as display device 104) can be controlled to display a first node graph (such as first node graph 304A). In one or more embodiments, circuitry 202 can be configured to control display device 104 to display the first node graph on the GUI 116 of an image / video editing application 106. The first node graph may include shaded nodes (such as shaded node 304B) representing a first workflow for shading at least a first object (such as a football player 302B) in one or more grayscale images (such as grayscale image 302A) fed by a first image.

[0062] At point 508, a reference color image can be selected. In one or more embodiments, the circuit system 202 can be configured to select the reference color image based on a first user input. For example, in Figure 3 The details of the selection of the reference color image are described in the text.

[0063] At point 510, a first workflow can be executed. In one or more embodiments, circuitry 202 can be configured to execute a first workflow associated with a shaded node (such as shader node 304B). For example, in Figure 3 The details of the execution of the first workflow are described in the document.

[0064] At point 512, the selected reference color image and the first image can be fed as input to the neural network-based colorization model 108. In one or more embodiments, the circuit system 202 can be configured to feed the selected reference color image and the first image as input to the neural network-based colorization model 108, for example in... Figure 3 As described in [the text].

[0065] At 514, a second image feed can be received. In one or more embodiments, the circuit system 202 can be configured to receive a second image feed as the output of a neural network-based coloring model 108, the second image feed comprising one or more colored images (such as colored image 308A), each colored image comprising at least a first object colored based on color effects on a ROI (such as ROI 306C). Control can be passed to the end.

[0066] Although flowchart 500 is shown as discrete operations, such as 502, 504, 506, 508, 510, 512, and 514, this disclosure is not limited thereto. Therefore, in some embodiments, such discrete operations may be further divided into additional operations, combined into fewer operations, or eliminated, depending on the particular implementation without departing from the essence of the disclosed embodiments.

[0067] Various embodiments of this disclosure may provide a non-transitory computer-readable medium and / or storage medium having instructions stored thereon that are executable by a machine and / or computer to operate a computing system (such as computing system 102). These instructions may cause the machine and / or computer to perform operations including obtaining a reference color image (such as reference color image 118) based on the selective application of color effects on a region of interest (ROI) (such as ROI 120) of an input image (such as input image 402A). Operations may also include controlling a display device (such as display device 104) to display a first node graph (such as first node graph 304A) on a graphical user interface (GUI) (such as GUI 116) of an image / video editing application (such as image / video editing application 106). The first node graph may include shading nodes (such as shading node 304B) representing a first workflow for shading at least a first object (such as a T-shirt of a football player 302B) in one or more grayscale images (such as grayscale image 302A) for a first image feed. The operation may also include selecting a reference color image based on a first user input and executing a first workflow associated with a coloring node. This execution includes feeding the selected reference color image and a first image feed as input to a neural network-based coloring model (such as neural network-based coloring model 108), and receiving a second image feed as the output of neural network-based coloring model 108 to the input, comprising one or more colorized images (such as colorized image 308A), each colorized image comprising at least a first object colorized based on color effects on a ROI.

[0068] Exemplary aspects of this disclosure may provide a computing system (such as circuit system 202) including a circuit system (such as circuit system 202). Figure 1The computing system 102. The circuit system 202 can be configured to obtain a reference color image (such as reference color image 118) based on the selective application of color effects on a region of interest (ROI) (such as ROI 120) of an input image (such as input image 402A). The circuit system 202 can be configured to control a display device (such as display device 104) to display a first node graph (such as first node graph 304A) on a graphical user interface (GUI) (such as GUI 116) of an image / video editing application (such as image / video editing application 106). The first node graph may include coloring nodes (such as coloring node 304B) representing a first workflow for coloring at least a first object (such as a T-shirt of a football player 302B) in one or more grayscale images fed with the first image. The circuit system 202 can be configured to select an acquired reference color image based on a first user input, and to perform a first workflow associated with a shading node by feeding the selected reference color image and a first image feed as input to a neural network-based shading model (such as neural network-based shading model 108), and to receive a second image feed, which is the output of neural network-based shading model 108 for that input, comprising one or more shading images (such as shading image 308A). Each of the one or more shading images may include at least a first object shading based on color effects on a ROI of the reference color image.

[0069] According to an embodiment, the circuit system 202 is also configured to control the display device to display a second node diagram (such as second node diagram 408A) on the GUI of an image / video editing application. The second node diagram may represent a second workflow for obtaining a reference color image from an input image.

[0070] According to an embodiment, the circuit system 202 is further configured to perform a second workflow by selecting an ROI from an input image, applying a first set of image filters to the selected ROI, and applying color effects to the selected ROI of the input image based on the application of the first set of image filters, in order to obtain a reference color image.

[0071] According to an embodiment, the shader node corresponds to a software plugin, which includes program instructions for executing a first workflow. According to an embodiment, the circuit system 202 is also configured to control the display device 104 to display settings for the shader node on the GUI of an image / video editing application. These settings may include a set of options corresponding to a set of neural network-based shader models. The circuit system 202 may also be configured to receive second user input, which includes a selection of a first option from the set of options. The circuit system 202 may also be configured to select a neural network-based shader model from the set of neural network-based shader models based on the second user input.

[0072] According to an embodiment, a neural network-based colorization model transfers color effects on the ROI of a reference color image to at least a first object (such as a T-shirt of a football player 302B) in each of one or more grayscale images in a first image feed, to output a second image feed that includes one or more colored images (such as colored image 308A).

[0073] According to an embodiment, the circuit system 202 can also be configured to control the display device 104 to display a second image feed on the GUI of an image / video editing application.

[0074] This disclosure can be implemented in hardware or a combination of hardware and software. It can be implemented in a centralized manner, on at least one computer system, or in a distributed manner, wherein different components can be distributed across multiple interconnected computer systems. A computer system or other apparatus suitable for performing the methods described herein may be appropriate. The combination of hardware and software can be a general-purpose computer system having a computer program that, when loaded and executed, can control the computer system to perform the methods described herein. This disclosure can be implemented in hardware that includes a portion of an integrated circuit that also performs other functions.

[0075] This disclosure can also be embedded in a computer program product that includes all features enabling the implementation of the methods described herein and, when loaded into a computer system, is capable of executing those methods. In this context, a computer program means any expression of a set of instructions represented in any language, code, or notation, which is intended to cause a system with information processing capabilities to directly perform a particular function, or to perform a particular function after one or both of the following: a) being translated into another language, code, or notation; or b) being copied in a different material form.

[0076] While this disclosure has been described with reference to certain embodiments, those skilled in the art will understand that various changes and substitutions can be made without departing from the scope of this disclosure. Furthermore, many modifications can be made to suit particular situations or materials to the teachings of this disclosure without departing from the scope of this disclosure. Therefore, it is intended that this disclosure be limited to the specific embodiments disclosed, but rather that this disclosure will include all embodiments falling within the scope of the appended claims.

Claims

1. A method comprising: In computing systems: A reference color image is obtained by selectively applying color effects to the region of interest (ROI) of the input image. The control display device displays the first node graph on the graphical user interface (GUI) of an image / video editing application, where... The first node graph includes shaded nodes representing a first workflow for shading at least a first object in one or more grayscale images fed by a first image. A reference color image obtained based on the first user's input selection; as well as Execute the first workflow associated with the shaded node, which includes: The selected reference color image and the first image are fed as input to the neural network-based colorization model; and As the output of the neural network-based coloring model in response to the input, it receives a second image feed comprising one or more colorized images, each colorized image including at least a first object colored based on color effects on a region of interest (ROI). The method of obtaining a reference color image by selectively applying color effects to the region of interest (ROI) of the input image includes: The control display device displays a second node graph on the GUI of an image / video editing application, where the second node graph represents a second workflow for obtaining a reference color image from an input image; and Execute the second workflow, which includes: Select ROIs from the input image based on third-user input; Apply a first set of selected image filters to the selected ROI, wherein the first set of selected image filters specifies the color effect to be applied to the ROI by modifying at least one of the following: color saturation, brightness, contrast, color values ​​in one or more color channels, gamma, or hue change; and Based on the application of a first set of image filters, a specified color effect is applied to the selected ROI of the input image to obtain a reference color image.

2. The method of claim 1, wherein the shaded node corresponds to a software plugin, the software plugin comprising program instructions for executing the first workflow.

3. The method according to claim 1, further comprising: Controls the display settings of shader nodes on the GUI of image / video editing applications. This setting includes a set of options corresponding to the set of neural network-based coloring models; Receive second user input, which includes a selection of a first option from a set of options; as well as Based on the second user input, a neural network-based coloring model is selected from a set of neural network-based coloring models.

4. The method of claim 1, wherein the neural network-based coloring model passes color effects on the ROI of a reference color image to at least a first object in each of the one or more grayscale images of the first image feed, to output a second image feed comprising the one or more colored images.

5. The method of claim 1, further comprising controlling a display device to display a second image feed on the GUI of an image / video editing application.

6. A computing system, comprising: The circuit system is configured as follows: A reference color image is obtained by selectively applying color effects to the region of interest (ROI) of the input image. The control display device displays the first node graph on the graphical user interface (GUI) of an image / video editing application, where... The first node graph includes shaded nodes representing a first workflow for shading at least a first object in one or more grayscale images fed by a first image. A reference color image obtained based on the first user's input selection; as well as The first workflow associated with the colored node is executed through the following operations: The selected reference color image and the first image are fed as input to the neural network-based colorization model; and As the output of the neural network-based coloring model in response to the input, it receives a second image feed comprising one or more colorized images, each colorized image including at least a first object colored based on color effects on a region of interest (ROI). The selective application of color effects on the region of interest (ROI) of the input image to obtain a reference color image includes: The control display device displays a second node graph on the GUI of an image / video editing application, where the second node graph represents a second workflow for obtaining a reference color image from an input image; and Execute the second workflow, which includes: Select ROIs from the input image based on third-user input; Apply a first set of selected image filters to the selected ROI, wherein the first set of selected image filters specifies the color effect to be applied to the ROI by modifying at least one of the following: color saturation, brightness, contrast, color values ​​in one or more color channels, gamma, or hue change; and Based on the application of a first set of image filters, a specified color effect is applied to the selected ROI of the input image to obtain a reference color image.

7. The computing system of claim 6, wherein the shaded nodes correspond to software plugins, the software plugins including program instructions for executing the first workflow.

8. The computing system of claim 6, wherein the circuit system is further configured as follows: Controls the display settings of shader nodes on the GUI of image / video editing applications. This setting includes a set of options corresponding to the set of neural network-based coloring models; Receive second user input, which includes a selection of a first option from a set of options; and Based on the second user input, a neural network-based coloring model is selected from a set of neural network-based coloring models.

9. The computing system of claim 6, wherein a neural network-based coloring model passes color effects on the ROI of a reference color image to at least a first object in each of the one or more grayscale images of the first image feed to output a second image feed comprising the one or more colored images.

10. The computing system of claim 6, wherein the circuitry is further configured to control the display device to display a second image feed on the GUI of an image / video editing application.

11. A non-transitory computer-readable medium having computer-executable instructions stored thereon, the computer-executable instructions causing the computing system to perform operations when executed by a computing system, said operations including: A reference color image is obtained by selectively applying color effects to the region of interest (ROI) of the input image. The control display device displays the first node graph on the graphical user interface (GUI) of an image / video editing application, where... The first node graph includes shaded nodes representing a first workflow for shading at least a first object in one or more grayscale images fed by a first image. A reference color image obtained based on the first user's input selection; as well as Execute the first workflow associated with the shaded node, which includes: The selected reference color image and the first image are fed as input to the neural network-based colorization model; and As the output of the neural network-based coloring model in response to the input, it receives a second image feed comprising one or more colorized images, each colorized image including at least a first object colored based on color effects on a region of interest (ROI). The method of obtaining a reference color image by selectively applying color effects to the region of interest (ROI) of the input image includes: The control display device displays a second node graph on the GUI of an image / video editing application, where the second node graph represents a second workflow for obtaining a reference color image from an input image; and Execute the second workflow, which includes: Select ROIs from the input image based on third-user input; Apply a first set of selected image filters to the selected ROI, wherein the first set of selected image filters specifies the color effect to be applied to the ROI by modifying at least one of the following: color saturation, brightness, contrast, color values ​​in one or more color channels, gamma, or hue change; and Based on the application of a first set of image filters, a specified color effect is applied to the selected ROI of the input image to obtain a reference color image.

12. The non-transitory computer-readable medium of claim 11, wherein the shaded nodes correspond to software plug-ins, the software plug-ins including program instructions for executing a first workflow.

13. The non-transitory computer-readable medium of claim 11, wherein the operation further comprises: Controls the display settings of shader nodes on the GUI of image / video editing applications. This setting includes a set of options corresponding to the set of neural network-based coloring models; Receive second user input, which includes a selection of a first option from a set of options; as well as Based on the second user input, a neural network-based coloring model is selected from a set of neural network-based coloring models.

14. The non-transitory computer-readable medium of claim 11, wherein a neural network-based colorization model passes color effects on ROIs of a reference color image to at least a first object in each of the one or more grayscale images of the first image feed to output a second image feed comprising the one or more colored images.

Images