Methods, apparatus, devices, and media for generating images with lighting and shadow effects.

Abstract

A method, apparatus, device, and medium for generating images with lighting and shadow effects are provided. In one method, an original image including a colored object is received. A lighting and shadow image associated with the original image is obtained using a machine learning model. The machine learning model describes the association between the colored reference object in the reference image and a reference lighting and shadow image of the reference object, the reference lighting and shadow image representing the lighting and shadow effects of the reference object. Based on the original image and the lighting and shadow image, a target image with lighting and shadow effects associated with the original image is generated. In this way, images with lighting and shadow effects can be generated in a simpler and more efficient manner, thereby improving the efficiency of image processing.

Description

Technical Field

[0001] Exemplary implementations of this disclosure generally relate to image processing, and more particularly to methods, apparatus, devices, and computer-readable storage media for generating images with lighting effects. Background Technology

[0002] With the development of digital image processing technology, various comic book creation tools have been developed. For example, comic artists can use professional drawing tools to create line drawings on digital canvases and use image processing tools to assist in coloring comics, and so on. However, readers' aesthetic demands for comic book characters are constantly increasing. For instance, readers may not be satisfied with comic book characters with a single coloring effect and expect to see comics with lighting and shadow effects. At this point, comic artists have to process each comic book image individually and add lighting and shadow effects manually, which leads to a huge expenditure of manpower and time. Therefore, there is a need for a simpler and more efficient way to generate images with lighting and shadow effects. Summary of the Invention

[0003] In a first aspect of this disclosure, a method for generating an image with lighting and shadow effects is provided. In this method, an original image including a colored object is received. A lighting and shadow image associated with the original image is obtained using a machine learning model, the machine learning model describing the association between the colored reference object in the reference image and a reference lighting and shadow image of the reference object, the reference lighting and shadow image representing the lighting and shadow effects of the reference object. Based on the original image and the lighting and shadow image, a target image with lighting and shadow effects associated with the original image is generated.

[0004] In a second aspect of this disclosure, an apparatus for generating an image with lighting and shadow effects is provided. The apparatus includes: a receiving module configured to receive an original image including a colored object; an acquiring module configured to acquire a lighting and shadow image associated with the original image using a machine learning model, the machine learning model describing the association between a colored reference object in the reference image and a reference lighting and shadow image of the reference object, the reference lighting and shadow image representing the lighting and shadow effects of the reference object; and a generating module configured to generate a target image with lighting and shadow effects associated with the original image based on the original image and the lighting and shadow image.

[0005] In a third aspect of this disclosure, an electronic device is provided. The electronic device includes: at least one processing unit; and at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions causing the device to perform a method according to a first aspect of this disclosure when executed by the at least one processing unit.

[0006] In a fourth aspect of this disclosure, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, causes the processor to implement the method according to a first aspect of this disclosure.

[0007] It should be understood that the content described in this content section is not intended to limit the key or essential features of the implementation of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0008] The above and other features, advantages, and aspects of various implementations of this disclosure will become more apparent in the following detailed description, taken in conjunction with the accompanying drawings. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0009] Figure 1 A block diagram of an example environment in which the implementation of this disclosure can be implemented is shown;

[0010] Figure 2 A block diagram illustrating a process for generating an image with lighting and shadow effects according to a technical solution is shown.

[0011] Figure 3 A block diagram is shown illustrating a process for obtaining a machine learning model according to some implementations of this disclosure;

[0012] Figure 4 A block diagram is shown illustrating a process for generating a predicted image according to some implementations of this disclosure;

[0013] Figure 5 A block diagram of another machine learning model according to some implementations of this disclosure is shown;

[0014] Figure 6 A block diagram of another machine learning model according to some implementations of this disclosure is shown;

[0015] Figure 7 A block diagram of a target image under different lighting scenarios according to some implementations of this disclosure is shown;

[0016] Figure 8 A block diagram showing the layer structure of the output image according to some implementations of this disclosure is provided;

[0017] Figure 9 A flowchart is shown of a method for generating images with lighting and shadow effects according to some implementations of this disclosure;

[0018] Figure 10 A block diagram of an apparatus for generating images with lighting and shadow effects according to some implementations of this disclosure is shown; and

[0019] Figure 11 A block diagram of a device capable of implementing various implementations of the present disclosure is shown. Detailed Implementation

[0020] Implementations of this disclosure will now be described in more detail with reference to the accompanying drawings. While some implementations of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the implementations set forth herein. Rather, these implementations are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and implementations of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0021] In the description of the implementation methods disclosed herein, the term "comprising" and similar terms should be understood as open inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one implementation" or "the implementation" should be understood as "at least one implementation". The term "some implementations" should be understood as "at least some implementations". Other explicit and implicit definitions may also be included below. As used herein, the term "model" can represent the relationships between various data. For example, the aforementioned relationships can be obtained based on various currently known and / or future-developed technical solutions.

[0022] It is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and related provisions.

[0023] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure through appropriate means in accordance with relevant laws and regulations, and user authorization should be obtained.

[0024] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the software or hardware, such as the electronic device, application, server, or storage medium performing the operations of this disclosed technical solution, based on the prompt message.

[0025] As an optional but non-restrictive implementation, in response to a user's active request, a prompt message can be sent to the user, for example, via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose whether to "agree" or "disagree" to provide personal information to the electronic device.

[0026] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0027] Example Environment

[0028] For ease of description, the process of generating images with lighting and shadow effects will be described within the context of comic book production. In comic book production, comic artists need to perform a manual coloring process. Specifically, the coloring process can be further divided into three stages: flat coloring, retouching, and special effects. The flat coloring stage fills various parts of the character with fixed colors and is the foundation of the coloring process. The retouching stage adds lighting and shadow effects to the flat-colored image, and the special effects stage further enriches the character's visual information.

[0029] Figure 1 A block diagram of an example environment 100 in which the implementation of this disclosure can be implemented is shown. For example... Figure 1 As shown, the retouching process can perform lighting and shadow effects processing on the original image 110 (i.e., the image after flat coloring) to generate the target image 120. At this time, the original image 110 can be a colored image after flat coloring, such as a line drawing image of a comic character and a colored flat color image. The original image 110 can, for example, include multiple layers to store the line drawing image and the flat color image separately. Alternatively and / or additionally, the original image 110 can include only a single layer, in which case the line drawing image and the flat color image are stored in a single layer.

[0030] Comic artists can add lighting and shadow effects to the original image 110 based on their experience. For example, they can add highlights 122 to the hair of a comic character and shadows 124 to their clothing, and so on. Typically, a comic book can include a large number of characters and involve many different lighting and shadow scenes. In this case, the comic artist has to process each comic image individually, resulting in significant time and manpower costs. Although retouching tools are currently available, the lighting and shadow effects generated by these tools are not satisfactory and still require substantial manual post-processing by the comic artist. Therefore, there is a desire for a simpler and more efficient way to generate images with lighting and shadow effects.

[0031] Overview of lighting and shadow processing

[0032] To address the shortcomings of the aforementioned technical solutions, a method for generating images with lighting and shadow effects is proposed based on an exemplary implementation of this disclosure. See also... Figure 2 To describe the outline of the method, Figure 2 A block diagram 200 illustrates a process for generating an image with lighting and shadow effects according to a technical solution. For example... Figure 2 As shown, an original image 110 can be received, which may include one or more colored objects (e.g., comic book characters). It will be understood that the original image 110 may include at least one line depicting the outline of various parts of the object (e.g., hair, clothing, etc.), and the portions of the original image 110 corresponding to each part have a single color. Although the original image 110 has been colored, its visual effect is relatively simple and lacks the sense of depth brought about by lighting effects.

[0033] A machine learning model 210 can be used to obtain a lighting image 220 associated with the original image 110. Here, the machine learning model 210 can be a pre-trained model. This model can describe the association between a colored reference object in the reference image and a reference lighting image of the reference object, and the reference lighting image can represent the lighting effect of the reference object. The reference image and the corresponding reference lighting image of the reference object can be used as training data, and the machine learning model 210 can be iteratively updated based on various training methods known to the target and / or to be developed in the future, in order to obtain the association between the image and the lighting effect.

[0034] In the inference phase following the training phase, the original image 110 is input into the machine learning model 210 to obtain a light and shadow image 220 corresponding to the original image 110. This light and shadow image 220 can include the light and shadow effects of various parts of the object, such as highlight areas with different brightness, shadow areas, and other areas with light and shadow effects, etc.

[0035] Furthermore, based on the original image 110 and the lighting image 220, a target image 120 with lighting effects associated with the original image 110 can be generated. For example, when a comic book character is in a given lighting scene, a highlight effect can be presented on the side facing the light source, and a shadow effect is expected on the side facing away from the light, and so on. At this time, the parts of the target image corresponding to each part have multiple colors. For example, the hair part of the target image 120 may include the original hair color and a light color representing the highlight effect, and the upper body part of the target image 120 may include the original upper body color and a dark color representing the shadow effect, and so on.

[0036] In this way, comic artists do not need to manually process each image one by one. Instead, they can use machine learning model 210 to generate a corresponding lighting and shadow image 220 for each image in a uniform manner. Furthermore, the original image 110 and the lighting and shadow image 220 can be superimposed to obtain a target image 120 with lighting and shadow effects. In this way, by providing a "one-click retouching" function, the manual workload in the comic production process can be greatly reduced, thereby improving the efficiency of comic production.

[0037] Detailed process of light and shadow processing

[0038] Having described an outline of an exemplary implementation according to this disclosure, further details regarding the lighting and shadow processing will be described below. Figure 3 A block diagram 300 illustrates a process for obtaining a machine learning model according to some implementations of this disclosure. For example... Figure 3 As shown, training data 360 can be used to perform the training process in order to optimize the machine learning model 210.

[0039] Specifically, the training data 360 may include a reference image 330 and a reference lighting image 332 (i.e., ground truth data). Here, the reference image 330 may include a line drawing image and a flat-painted image of a comic character; in other words, the reference image 330 is a colored image. The reference lighting image 332 may be a ground truth image that accurately describes the lighting effects of the comic character in the reference image 330, obtained through various methods. For example, the reference lighting image 332 may be hand-drawn by the comic artist, or it may be an accurate lighting image generated using existing retouching tools and confirmed by the comic artist. In this case, the reference lighting image 332 will participate in the training process as labeled ground truth data.

[0040] like Figure 2 As shown, a machine learning model 210 with untrained parameters (and / or pre-trained parameters) can be obtained, and the machine learning model 210 can be used to generate a predicted image 340 of the reference lighting image 332. Specifically, the machine learning model 210 can be implemented using a UNet network structure. A loss function 350 of the machine learning model 210 can be generated based on the difference between the predicted image 340 and the reference lighting image 332, which serves as the ground truth. Furthermore, the parameters in the machine learning model 210 can be updated based on the loss function 350.

[0041] According to an exemplary implementation of this disclosure, a training dataset comprising a large amount of training data can be obtained, and the training process can be iteratively executed based on various training methods currently known and / or to be developed in the future, so that the machine learning model 210 can grasp the correlation between the reference image 330 and the reference lighting image 332. In this way, the machine learning model 210 will be progressively optimized to generate corresponding lighting images for the input image to be processed.

[0042] At this point, the machine learning model 210 may include: a color channel model 310 (e.g., a first sub-model) for processing data in the three color channels (RGB, red, green, and blue) of the image; and a transparency channel model 320 (e.g., a second sub-model) for processing data in the transparency channel (i.e., alpha) of the image. Specifically, the color channel model 310 can describe the relationship between the colors of each pixel in the reference image 330 and the reference lighting image 332, and the transparency channel model 320 can describe the relationship between the illumination intensity of each pixel in the reference image and the reference lighting image. According to an exemplary implementation of this disclosure, the color channel model 310 and the transparency channel model 320 can be trained separately; alternatively and / or additionally, the two models can be trained in combination.

[0043] It will be understood that the reference image 330 here may only include the three color channels RGB, while the image in the lighting and shadow processing process may involve four channels RGBA (red, green, blue, and alpha). Specifically, the machine learning model 210 can process the input reference image 330 (i.e., the flat-painted image, for example, denoted by A) and output a predicted image 340 (i.e., the refined image, for example, denoted by B) of the reference lighting and shadow image 332. At this time, the flat-painted image A and the refined image B in the training data are both RGB images, while the machine learning model needs to use a refined image with four RGBA channels. In order for the model to learn to generate an RGBA refined image based on the RGB training data, the refined image B can be divided into two parts: a color channel image 312 (for example, denoted by RGB) and a alpha channel image 322 (for example, denoted by alpha).

[0044] During training, the color channel model 310 in the machine learning model 210 can be used to generate a color channel image 312 associated with the reference image 330; further, the transparency channel model 320 in the machine learning model 210 can be used to generate a transparency channel image 322 associated with the reference image 330. Using the exemplary implementation of this disclosure, the original color information of the image to be processed and the lighting information in the reference lighting image can be fully considered during the lighting and shadow processing. In this way, the machine learning model 210 can fully grasp various aspects of knowledge for generating corresponding lighting and shadow effects from a specific image. This improves the accuracy of the machine learning model 210, thereby enhancing the realism of the lighting and shadow images output by the model.

[0045] According to an exemplary implementation of this disclosure, a predicted image 340 can be generated based on the color channel image 312 and the transparency channel image 322. Specifically, the predicted image 340 can be determined based on the following formula 1:

[0046] Bp = (1-alpha)*A + alpha*RGB

[0047] Formula 1

[0048] In the above formula, Bp represents the predicted image 340, alpha represents the generated alpha channel image 322, A represents the input reference image 330, and RGB represents the generated color channel image 312. Furthermore, a loss function 350 can be determined based on the difference between the generated predicted image Bp and the reference lighting image 332 in the training data 360, and then the parameters in the machine learning model 210 can be updated in the direction that minimizes the loss function 350.

[0049] According to an exemplary implementation of this disclosure, the images may have the same size; for example, each image may include M*N (where M and N are positive integers) pixels. To generate the prediction image 340 based on the color channel image 312 and the alpha channel image 322, for a pixel position (e.g., position (i,j)) in the reference image 330, the prediction image 340 may be generated based on the pixel data (e.g., denoted as A) corresponding to that pixel position in the reference image 330. i,j ), the pixel data corresponding to that pixel position in the color channel image 312 (e.g., represented as RGB) i,j ), the pixel data corresponding to the pixel position in the alpha channel image 322 (e.g., denoted as alpha). i,j ), determine the pixel data (e.g., denoted as Bp) in the predicted image 340 corresponding to the pixel location. i,jHere, the value of each pixel position in the alpha channel image 322 can be selected from the range [0,1], and the value can represent the transparency of the pixel at the corresponding position.

[0050] Figure 4 A block diagram 400 illustrating a process for generating a predicted image according to some implementations of this disclosure is shown. As shown, pixel data 410, 420, 430, and 440 represent data at position (i,j) in the reference image 330, the color channel image 312, the transparency channel image 322, and the predicted image 340, respectively. Here, pixel data 410, 420, and 440 each include data from the RGB channels, and pixel data 430 represents the transparency at the corresponding position. Formula 1 described above can be applied to each pixel in the image to determine each pixel in the predicted image 340. Formula 1 can then be refined to Formula 2 as follows:

[0051] Bp i,j =(1-alpha) i,j )*A i,j +alpha i,j *RGB i,j

[0052] Formula 2

[0053] In Formula 2, i and j represent positive integers, 1 ≤ i ≤ M and 1 ≤ j ≤ N. At this point, each position in the image can be traversed to generate the corresponding predicted image 340. Using the exemplary implementation of this disclosure, pixel data from the reference image 330 and the color channel image 312 can be superimposed according to the transparency of pixels at each position in the transparency channel 322 to determine the predicted image 340. In this way, the machine learning model 210 can fully grasp the manually labeled ground truth lighting effects in the training data 360, thereby generating corresponding lighting images for the cartoon characters in the input image in a more accurate and realistic manner.

[0054] See above Figure 3 and Figure 4 The core processing of the machine learning model 210 is described, and alternatively and / or additionally, the machine learning model 210 may involve more auxiliary processing. According to an exemplary implementation of this disclosure, the light and shadow processing described above can be implemented based on a Generative Adversarial Network (GAN) architecture.

[0055] A GAN architecture can include a generative model (e.g., the machine learning model described above 210) and a discriminator model. The generative model generates instances that appear natural and realistic, while the discriminator model determines whether a given instance appears natural and realistic or artificially generated. During training, the goal of the generative model is to generate realistic images as much as possible and deceive the discriminator model, while the goal of the discriminator model is to distinguish between generated and real images as much as possible. At this point, the generative model and the discriminator model can engage in a dynamic "game" to achieve a relative equilibrium.

[0056] Figure 5 A block diagram 500 of another machine learning model according to some implementations of this disclosure is shown. Figure 5 As shown, a discriminator model 520 associated with the machine learning model 210 can be constructed to determine the degree of matching between the predicted image 340 and the ground truth reference lighting image 332 (i.e., whether the predicted image 340 is a ground truth image or a generated image). Furthermore, the loss function 350 can be updated based on the matching degree output by the discriminator model 520. In this way, the machine learning model 210 can be updated in a direction that makes the image output by the machine learning model 210 "difficult to distinguish from the real one," thereby obtaining a more realistic output image. According to an exemplary implementation of this disclosure, the loss function of the machine learning model 210 can be further updated based on perceptual loss and / or other losses, thereby further improving the accuracy of the machine learning model 210.

[0057] According to an exemplary implementation of this disclosure, richer auxiliary information can be input into the machine learning model 210, thereby making it easier for the machine learning model 210 to consider more influencing factors when generating light and shadow images. Figure 6 A block diagram 600 of another machine learning model according to some implementations of this disclosure is shown. For example... Figure 6 As shown, the machine learning model 210 can be a multi-head processing model. In addition to receiving the original image 110 to be processed, the machine learning model 210 can further acquire other information associated with the original image 110, such as material information 610 and lighting scene information 620.

[0058] According to an exemplary implementation of this disclosure, multiple semantic regions can be predefined for a comic character, including but not limited to: the whites of the eyes, the irises of the eyes, eyebrows, mouth, face, hair, hair accessories, earrings, glasses, upper garment, hands, torso skin, pants, skirt, socks, shoes, legs, etc. Each semantic region can have a specific definition; for example, a hair accessory can represent decorations near the hair, including but not limited to hats, hoodies, headbands, headphones, decorative rabbit / cat ears, flowers, crowns, hair clips, etc.; the upper garment can include underwear, vests, T-shirts, coats, the upper part of dresses / suspenders, necklaces, etc. Material information 610 at various locations in the reference images from the training data can be further obtained to acquire lighting and shadow effects matching different materials.

[0059] Specifically, hair accessories, earrings, and glasses can be made using reflective or transparent materials, which will have a more pronounced reflective effect. Similarly, clothing for comic book characters can be made from different materials, and different materials can create different light and shadow effects. For example, silk has a strong reflective effect, while cotton typically has a weaker reflective effect.

[0060] According to an exemplary implementation of this disclosure, material information can be stored in a flat-painted layer in the reference image. For example, corresponding material information can be assigned to each color block of the flat-painted layer. A glass material can be set for the hair ornaments of a cartoon character, a silk material for the upper garment of a cartoon character, a cotton material for the pants of a cartoon character, and so on. In this way, material information of different semantic regions can be acquired during training, allowing the machine learning model 210 to fully learn the lighting effects of different materials in the training data, thereby generating corresponding lighting images for different materials in a more realistic manner.

[0061] During the inference phase, the machine learning model 210 described above can be used to generate the corresponding lighting and shadow images. Specifically, the original image 110 to be processed can be input into the machine learning model 210, and material information associated with the clothing of the comic character can be obtained. Furthermore, the machine learning model can generate the corresponding lighting and shadow images based on the material information and the original image 110. According to an exemplary implementation of this disclosure, material information of each region can be extracted from the flat-painted layer of the original image 110. Assuming that the comic character's upper garment is made of satin and the pants are made of cotton, the area where the upper garment is located in the generated lighting and shadow image can have a strong reflective effect, and the area where the pants are located in the generated lighting and shadow image can have a weak reflective effect.

[0062] According to an exemplary implementation of this disclosure, lighting scene information 620 involving the reference image in the training data can be further obtained. In this way, the machine learning model 210 can obtain the correlation between the image, the lighting scene in which the image is located, and the lighting data of the image, thereby generating light and shadow data matching the lighting scene with a finer granularity.

[0063] The lighting scene information 620 can include various aspects, such as location information, time information, light source type, light source direction, etc., associated with the original image. During the training phase, a large amount of training data involving different lighting scene information 620 can be collected to perform the training process. Specifically, training can be performed using training data depicting outdoor and indoor locations, training data depicting different times of day (e.g., morning, noon, evening, night, etc.), training data depicting different seasons of the year, training data involving different light source types (e.g., sunlight, artificial light, etc.), and training data involving different light source directions (e.g., frontal illumination, side illumination, etc.).

[0064] According to an exemplary implementation of this disclosure, lighting scene information 620 can be obtained in various ways. For example, user settings for specifying lighting scene information can be directly read to obtain the lighting scene of the reference image in the training data. In this way, lighting scene information can be obtained more accurately without significantly increasing the complexity of the machine learning model 210, thereby improving the processing accuracy of the machine learning model 210.

[0065] Alternatively and / or additionally, image analysis can be performed on reference images in the training data to determine the corresponding lighting scene. Specifically, location information, time information, light source type and direction, etc., related to the reference image can be determined based on techniques such as image recognition. In this way, the amount of input data in the training phase can be reduced, thereby obtaining the machine learning model 210 in a more automated manner.

[0066] According to an exemplary implementation of this disclosure, a machine learning model 210 generated based on the above method can be used to process the input raw image, thereby generating a corresponding target image. According to an exemplary implementation of this disclosure, lighting scene information associated with the raw image can be obtained, and then a light and shadow image can be generated based on the lighting scene information and the machine learning model. Figure 7A block diagram 700 is shown of light and shadow images under different lighting scenarios according to some implementations of this disclosure. Specifically, the original image 110 can be input into the machine learning model 210, and lighting scene data associated with the original image 110 can be further specified.

[0067] For example, you can specify generating a lighting image for a nighttime lighting scene, in which case lighting image 710 shows the lighting effect in a nighttime lighting scene. As another example, you can specify generating a lighting image for a daytime lighting scene, in which case lighting image 720 shows the lighting effect in a daytime lighting scene. Figure 7 As shown, although the same original image 110 is input into the machine learning model 210, the machine learning model 210 can generate corresponding light and shadow images 710 and 720 for nighttime lighting scenes and daytime lighting scenes, respectively. In this way, more flexibility can be provided for the light and shadow effect processing, thereby improving the realism of comic production.

[0068] According to one exemplary implementation of this disclosure, the generated target image can be stored in one or more layers. For example, in the case of using multiple layers, the original image can be stored in the first layer of the target image, and the lighting image can be stored in the second layer of the target image. Figure 8 A block diagram 800 shows the layer structure of the output image according to some implementations of this disclosure. For example... Figure 8 As shown, an output image can be obtained, and this output image can include multiple layers. The line art image of the original image 110 can be stored in the line art layer 810 of the output image, the flat color image of the original image 110 can be stored in the flat color layer 820, and the generated light and shadow image can be stored in the light and shadow layer 830. At this time, when the above three layers are presented in an overlay manner, a target image with light and shadow effects (e.g., as shown) will be presented. Figure 2 (The target image 120 is shown in the image).

[0069] Alternatively and / or additionally, the target image 120 can be stored in a new layer of the output image. In this way, the output image can record the results of different stages of comic production, thus facilitating adjustments by the comic artist as needed. Alternatively and / or additionally, the target image 120 can be stored in the output image, which includes a single layer. This reduces the data size of the output image, thereby reducing the overall data storage requirements of the comic production process.

[0070] Although the above describes the case where all images in the lighting and shading process are the same size, alternatively and / or additionally, the size of the generated lighting and shading image can differ from (e.g., be smaller than) the original image. In this case, the computational load involved in lighting and shading can be reduced without significantly affecting the visual effect, thus facilitating the processing of a large number of original images with higher performance. In this way, the efficiency of the retouching stage in comic book production can be improved, thereby reducing the overall time consumption of comic book production.

[0071] Although the method of this disclosure has been described above using a single image as an example, alternatively and / or additionally, one or more images to be colored can be extracted from multimedia data, and the coloring process described above can be performed on each extracted image. Here, multimedia data can include at least one of the following: graphic data, image sets, video data, game data, etc. Specifically, the image to be processed can be extracted from graphic data, for example, data including images and text, and lighting and shading can be performed on objects in the image, thus obtaining graphic data including lighting and shading effects. For example, the image to be processed can be extracted from a specified directory in an image set, the image to be processed can be extracted from each image frame in video data, or the image to be processed can be extracted from game data, etc. In this way, lighting and shading processes can be performed on multimedia data of different formats in a more flexible manner, thereby making the lighting and shading processes of this disclosure compatible with more existing multimedia production applications.

[0072] It will be understood that although objects are shown as human figures in the accompanying drawings, alternatively and / or additionally, the image to be processed may include other entities. For example, in a landscape image, objects may include trees, houses, mountains, lakes, etc. Parts in this case may include, for example, tree trunks, branches, leaves, walls, rooftops, etc. As another example, in an interior image, objects may include furniture, appliances, etc. Parts in this case may include tabletops, table legs, cushions, etc. In this way, the diverse content included in a comic book image can be processed more efficiently, thereby improving the efficiency of comic book production.

[0073] Example process

[0074] Figure 9A flowchart of a method 900 for generating an image with lighting effects according to some implementations of this disclosure is shown. At block 910, an original image including a colored object is received. At block 920, a lighting image associated with the original image is obtained using a machine learning model. The machine learning model describes the association between a colored reference object in the reference image and a reference lighting image of the reference object, the reference lighting image representing the lighting effects of the reference object. At block 930, a target image with lighting effects associated with the original image is generated based on the original image and the lighting image.

[0075] According to an exemplary implementation of this disclosure, the method 900 further includes obtaining a machine learning model based on: using the machine learning model to generate a predicted image of a reference lighting image; generating a loss function for the machine learning model based on the difference between the predicted image and the reference lighting image; and updating the machine learning model based on the loss function.

[0076] According to an exemplary implementation of this disclosure, generating a predicted image includes: using a first sub-model in a machine learning model to generate a color channel image associated with a reference image, the first sub-model describing the relationship between the colors of individual pixels in the reference image and the reference lighting image; using a second sub-model in a machine learning model to generate a transparency channel image associated with the reference image, the second sub-model describing the relationship between the illumination intensity of individual pixels in the reference image and the reference lighting image; and generating a predicted image based on the color channel image and the transparency channel image.

[0077] According to an exemplary implementation of this disclosure, generating a predicted image based on a color channel image and a transparency channel image includes: for a pixel position in a reference image, determining pixel data corresponding to the pixel position in the predicted image based on pixel data corresponding to the pixel position in the reference image, pixel data corresponding to the pixel position in the color channel image, and pixel data corresponding to the pixel position in the transparency channel image.

[0078] According to an exemplary implementation of this disclosure, the method 900 further includes: using a discriminator model associated with a machine learning model to determine the degree of matching between the predicted image and a reference lighting image; and updating a loss function based on the degree of matching.

[0079] According to an exemplary implementation of this disclosure, generating a target image includes: for a pixel position in the original image, determining pixel data corresponding to the pixel position in the target image based on pixel data corresponding to the pixel position in the original image and pixel data corresponding to the pixel position in the light and shadow image.

[0080] According to an exemplary implementation of this disclosure, obtaining a light and shadow image further includes: obtaining lighting scene information associated with the original image; and generating a light and shadow image based on the lighting scene information and a machine learning model.

[0081] According to an exemplary implementation of this disclosure, obtaining lighting scene information includes extracting lighting scene information based on at least one of the following: user settings for specifying lighting scene information; and image analysis of the original image.

[0082] According to one exemplary implementation of this disclosure, the lighting scene information includes at least one of the following: location information associated with the original image, time information, light source type, and light source direction.

[0083] According to an exemplary implementation of this disclosure, the object includes a comic book character, and obtaining the lighting image further includes: obtaining material information associated with the semantic regions of the comic book character, the semantic regions including at least one of the following: sclera of the eyes, iris of the eyes, eyebrows, mouth, face, hair, hair ornaments, earrings, glasses, upper garment, hands, torso skin, pants, skirt, socks, shoes, and legs; and generating the lighting image based on the material information and a machine learning model.

[0084] According to an exemplary implementation of this disclosure, the original image includes at least one line depicting the outline of a part of an object, the portion of the original image corresponding to the part has a single color, and the portion of the target image corresponding to the part has multiple colors.

[0085] According to an exemplary implementation of this disclosure, generating a target image includes: storing an original image in a first layer of the target image; and storing a light and shadow image in a second layer of the target image.

[0086] According to an exemplary implementation of this disclosure, the method 900 further includes: extracting the image to be colored from multimedia data, wherein the multimedia data includes at least one of the following: text data, image set, video data, and game data.

[0087] Example devices and equipment

[0088] Figure 10A block diagram of an apparatus 1000 for generating an image with lighting effects according to some implementations of the present disclosure is shown. The apparatus includes: a receiving module 1010 configured to receive an original image including a colored object; an acquiring module 1020 configured to acquire a lighting image associated with the original image using a machine learning model, the machine learning model describing the association between a colored reference object in the reference image and a reference lighting image of the reference object, the reference lighting image representing the lighting effects of the reference object; and a generating module 1030 configured to generate a target image with lighting effects associated with the original image based on the original image and the lighting image.

[0089] According to an exemplary implementation of this disclosure, it further includes: a prediction module configured to generate a predicted image of a reference lighting image using a machine learning model; a comparison module configured to generate a loss function of the machine learning model based on the difference between the predicted image and the reference lighting image; and an update module configured to update the machine learning model based on the loss function.

[0090] According to an exemplary implementation of this disclosure, the prediction module comprises: a color channel generation module configured to generate a color channel image associated with a reference image using a first sub-model in a machine learning model, the first sub-model describing the relationship between the colors of individual pixels in the reference image and the reference lighting image; a transparency channel generation module configured to generate a transparency channel image associated with the reference image using a second sub-model in a machine learning model, the second sub-model describing the relationship between the illumination intensity of individual pixels in the reference image and the reference lighting image; and a prediction image generation module configured to generate a prediction image based on the color channel image and the transparency channel image.

[0091] According to an exemplary implementation of this disclosure, the predicted image generation module includes: a pixel determination module, configured to determine, for a pixel position in a reference image, pixel data corresponding to the pixel position in the reference image, pixel data corresponding to the pixel position in the color channel image, and pixel data corresponding to the pixel position in the alpha channel image.

[0092] According to an exemplary implementation of this disclosure, the apparatus further includes: a discrimination module configured to determine the degree of matching between the predicted image and the reference lighting image using a discriminator model associated with a machine learning model; and a loss update module configured to update a loss function based on the degree of matching.

[0093] According to an exemplary implementation of this disclosure, the generation module includes: a pixel-based generation module configured to determine, based on pixel data corresponding to the pixel position in the original image and pixel data corresponding to the pixel position in the light and shadow image, pixel data corresponding to the pixel position in the original image.

[0094] According to an exemplary implementation of this disclosure, the apparatus further includes: a scene acquisition module configured to acquire lighting scene information associated with the original image; and a scene-based generation module configured to generate a light and shadow image based on the lighting scene information and a machine learning model.

[0095] According to one exemplary implementation of this disclosure, the lighting scene information is extracted based on at least one of the following: user settings for specifying the lighting scene information; image analysis of the original image.

[0096] According to one exemplary implementation of this disclosure, the lighting scene information includes at least one of the following: location information associated with the original image, time information, light source type, and light source direction.

[0097] According to an exemplary implementation of this disclosure, the object includes a comic book character, and the acquisition module further includes: a material acquisition module configured to acquire material information associated with the semantic regions of the comic book character, the semantic regions including at least one of the following: sclera of the eyes, iris of the eyes, eyebrows, mouth, face, hair, hair ornaments, earrings, glasses, upper garment, hands, torso skin, pants, skirt, socks, shoes, and legs; and a material-based generation module configured to generate a light and shadow image based on the material information and a machine learning model.

[0098] According to an exemplary implementation of this disclosure, the original image includes at least one line depicting the outline of a part of an object, the portion of the original image corresponding to the part has a single color, and the portion of the target image corresponding to the part has multiple colors.

[0099] According to an exemplary implementation of this disclosure, the generation module includes: a first storage module configured to store the original image in a first layer of the target image; and a second storage module configured to store the light and shadow image in a second layer of the target image.

[0100] According to an exemplary implementation of this disclosure, the apparatus further includes an extraction module configured to: extract an image to be colored from multimedia data, the multimedia data including at least one of the following: text data, image sets, video data, and game data.

[0101] Figure 11A block diagram of a device 1100 capable of implementing various implementations of the present disclosure is shown. It should be understood that... Figure 11 The computing device 1100 shown is merely exemplary and should not be construed as limiting the functionality and scope of the implementation described herein. Figure 11 The computing device 1100 shown can be used to implement the method described above.

[0102] like Figure 11 As shown, computing device 1100 is in the form of a general-purpose computing device. Components of computing device 1100 may include, but are not limited to, one or more processors or processing units 1110, memory 1120, storage devices 1130, one or more communication units 1140, one or more input devices 1150, and one or more output devices 1160. Processing unit 1110 may be a physical or virtual processor and is capable of performing various processes according to programs stored in memory 1120. In a multiprocessor system, multiple processing units execute computer-executable instructions in parallel to improve the parallel processing capability of computing device 1100.

[0103] Computing device 1100 typically includes multiple computer storage media. Such media can be any available media accessible to computing device 1100, including but not limited to volatile and non-volatile media, removable and non-removable media. Memory 1120 can be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. Storage device 1130 can be removable or non-removable media and can include machine-readable media, such as flash drives, disks, or any other media that can be used to store information and / or data (e.g., training data for training) and can be accessed within computing device 1100.

[0104] The computing device 1100 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not explicitly stated... Figure 11 As shown, disk drives for reading from or writing to removable, non-volatile disks (e.g., "floppy disks") and optical disk drives for reading from or writing to removable, non-volatile optical disks can be provided. In these cases, each drive can be connected to a bus (not shown) via one or more data media interfaces. Memory 1120 may include computer program product 1125 having one or more program modules configured to perform various methods or actions of various implementations of this disclosure.

[0105] The communication unit 1140 enables communication with other computing devices via a communication medium. Additionally, the components of the computing device 1100 can function as a single computing cluster or multiple computing machines capable of communicating via communication connections. Therefore, the computing device 1100 can operate in a networked environment using logical connections to one or more other servers, networked personal computers (PCs), or another network node.

[0106] Input device 1150 can be one or more input devices, such as a mouse, keyboard, trackball, etc. Output device 1160 can be one or more output devices, such as a monitor, speaker, printer, etc. Computing device 1100 can also communicate as needed with one or more external devices (not shown) via communication unit 1140. These external devices include storage devices, display devices, etc., and can communicate with one or more devices that enable user interaction with computing device 1100, or with any device that enables computing device 1100 to communicate with one or more other computing devices (e.g., network card, modem, etc.). Such communication can be performed via input / output (I / O) interfaces (not shown).

[0107] According to exemplary implementations of this disclosure, a computer-readable storage medium is provided that stores computer-executable instructions thereon, wherein the computer-executable instructions are executed by a processor to implement the methods described above. According to exemplary implementations of this disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, which are executed by a processor to implement the methods described above. According to exemplary implementations of this disclosure, a computer program product is provided that stores a computer program thereon, which, when executed by a processor, implements the methods described above.

[0108] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, devices, and computer program products implemented according to this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0109] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0110] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions that execute on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0111] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0112] Various implementations of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is chosen to best explain the principles, practical applications, or improvements to technology in the market, or to enable others skilled in the art to understand the various implementations disclosed herein.

Claims

1. A method for generating an image with lighting effects, comprising: receiving an original image including a colored object; acquiring a lighting image associated with the original image using a machine learning model that describes an association between a colored reference object in a reference image and a reference lighting image of the reference object, the reference lighting image representing lighting effects of the reference object, wherein the machine learning model is acquired based on: generating a color channel image associated with the reference image using a first sub-model in the machine learning model that describes an association between colors of respective pixels in the reference image and the reference lighting image; generating a transparency channel image associated with the reference image using a second sub-model in the machine learning model that describes an association between lighting intensities of respective pixels in the reference image and the reference lighting image; and generating a predicted image of the reference lighting image based on the color channel image and the transparency channel image; and updating the machine learning model based on the predicted image and the reference lighting image; and generating a target image with lighting effects associated with the original image based on the original image and the lighting image.

2. The method of claim 1, updating the machine learning model based on the predicted image and the reference lighting image comprises: generating a loss function of the machine learning model based on a difference between the predicted image and the reference lighting image; and updating the machine learning model based on the loss function. determining pixel data in the predicted image corresponding to a pixel position in the reference image based on pixel data in the reference image corresponding to the pixel position, pixel data in the color channel image corresponding to the pixel position, and pixel data in the transparency channel image corresponding to the pixel position.

4. The method of claim 2, further comprising: determining a matching degree of the predicted image and the reference lighting image using a discriminator model associated with the machine learning model; and updating the loss function based on the matching degree.

3. The method of claim 1, wherein generating the predicted image based on the color channel image and the transparency channel image comprises: determining pixel data in the target image corresponding to a pixel position in the original image based on pixel data in the original image corresponding to the pixel position and pixel data in the lighting image corresponding to the pixel position.

6. The method of claim 1, wherein acquiring the lighting image further comprises: acquiring lighting scene information associated with the original image; and generating the lighting image according to the lighting scene information and the machine learning model.

7. The method of claim 6, wherein acquiring the lighting scene information comprises extracting the lighting scene information based on at least any of:

5. The method of claim 1, wherein generating the target image comprises: a user setting for specifying the lighting scene information; image analysis for the original image. ​ ​ ​ ​ ​ ​ 8.The method of claim 6, wherein the lighting scene information comprises at least one of the following: location information, time information, light source type, light source direction associated with the original image. 9.The method of claim 1, wherein the object comprises a cartoon character, and the obtaining the light-and-shadow image further comprises: obtaining material information associated with semantic regions of the cartoon character, the semantic regions comprising at least one of the following: eye white, eye black, eyebrow, mouth, face, hair, hair accessory, ear accessory, glasses, upper garment, hand, torso skin, trousers, skirt, sock, shoe, leg; and generating the light-and-shadow image according to the material information and the machine learning model. The original image comprises at least one line depicting an outline of a part of the object, a portion of the original image corresponding to the part has a single color, and a portion of the target image corresponding to the part has multiple colors. 11.The method of claim 1, wherein the generating the target image comprises: storing the original image in a first layer of the target image; and storing the light-and-shadow image in a second layer of the target image.

10. The method of claim 1, wherein: extracting an image to be colored from multimedia data, the multimedia data comprising at least one of the following: text-image data, image set, video data, game data. 13.An apparatus for generating an image with light-and-shadow effect, comprising: a receiving module configured to receive an original image comprising a colored object; an obtaining module configured to obtain a light-and-shadow image associated with the original image by utilizing a machine learning model, the machine learning model describing an association relationship between a colored reference object in a reference image and a reference light-and-shadow image of the reference object, the reference light-and-shadow image representing a light-and-shadow effect of the reference object, wherein the machine learning model is obtained based on: generating a color channel image associated with the reference image by utilizing a first sub-model in the machine learning model, the first sub-model describing an association relationship between colors of respective pixel points in the reference image and the reference light-and-shadow image; generating a transparency channel image associated with the reference image by utilizing a second sub-model in the machine learning model, the second sub-model describing an association relationship between lighting intensities of respective pixel points in the reference image and the reference light-and-shadow image; and generating a predicted image of the reference light-and-shadow image based on the color channel image and the transparency channel image; and updating the machine learning model based on the predicted image and the reference light-and-shadow image; and generating a target image with light-and-shadow effect associated with the original image based on the original image and the light-and-shadow image. 14.An electronic device, comprising: at least one processing unit; and ​ ​ 12. The method of claim 1, further comprising: ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions when executed by the at least one processing unit cause the electronic device to perform the method according to any one of claims 1 to 12.

15. A computer readable storage medium having stored thereon a computer program, which, when executed by a processor, causes the processor to carry out the method according to any one of claims 1 to 12.

Images