Methods, apparatus, devices and media for image coloring

Abstract

Methods, apparatus, devices, and media for image coloring are provided. In one method, a set of lines is extracted from an image to be colorized. The set of lines is converted into a set of thinned lines, wherein a target thinned line in the set of thinned lines includes a portion of pixels in a target line corresponding to the target thinned line in the set of lines. A target color is used to color a filled area in the image represented by the set of thinned lines. Based on the image and the colorized filled area, a colorized image of the image is generated. In this way, the areas in the image to be colorized can be fully filled, reducing the workload of manual labor in the coloring process and thus improving coloring efficiency.

Description

Technical Field

[0001] Exemplary implementations of this disclosure generally relate to image processing, and more particularly to methods, apparatus, devices, and computer-readable storage media for coloring comic images. Background Technology

[0002] With the development of digital image processing technology, various comic creation tools have been developed. For example, comic artists can use professional drawing tools to create line art on digital canvases, and can use image processing tools to assist in coloring comics, and so on. However, existing tools may have some errors, such as failing to completely fill the area to be colored, or the filled color potentially exceeding the area to be colored, and so on. Comic artists have to check and fix errors in each image one by one, resulting in a huge expenditure of manpower and time. Therefore, there is a desire for a simpler and more efficient way to complete the coloring process. Summary of the Invention

[0003] In a first aspect of this disclosure, a method for colorizing an image is provided. In this method, a set of lines is extracted from an image to be colorized. The set of lines is converted into a set of thinned lines, wherein a target thinned line in the set of thinned lines includes a portion of pixels in a target line corresponding to the target thinned line in the set of lines. A target color is used to color a filled area in the image represented by the set of thinned lines. Based on the image and the colorized filled area, a colorized image of the image is generated.

[0004] In a second aspect of this disclosure, an apparatus for coloring an image is provided. The apparatus includes: an extraction module configured to extract a set of lines from an image to be colored; a conversion module configured to convert the set of lines into a set of thinned lines, wherein a target thinned line in the set of thinned lines includes a portion of pixels in a target line corresponding to the target thinned line in the set of lines; a coloring module configured to color a filled area in the image represented by the set of thinned lines using a target color; and a generation module configured to generate a colorized image of the image based on the image and the colorized filled area.

[0005] In a third aspect of this disclosure, an electronic device is provided. The electronic device includes: at least one processing unit; and at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions causing the device to perform a method according to a first aspect of this disclosure when executed by the at least one processing unit.

[0006] In a fourth aspect of this disclosure, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, causes the processor to implement the method according to a first aspect of this disclosure.

[0007] It should be understood that the content described in this content section is not intended to limit the key or essential features of the implementation of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0008] The above and other features, advantages, and aspects of various implementations of this disclosure will become more apparent in the following detailed description, taken in conjunction with the accompanying drawings. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0009] Figure 1 A block diagram of an example environment in which the implementation of this disclosure can be implemented is shown;

[0010] Figure 2 A block diagram of an image coloring process according to a technical solution is shown;

[0011] Figure 3 A block diagram illustrating the determination of a fill region according to some implementations of this disclosure is shown;

[0012] Figure 4 A block diagram is shown for generating a colored image according to some implementations of this disclosure;

[0013] Figure 5 A block diagram of an image to be colored, including non-closed regions, is shown according to some implementations of this disclosure;

[0014] Figure 6 Some implementations of this disclosure are shown for use with Figure 5 The image shown is a coloring diagram;

[0015] Figure 7 A block diagram of a machine learning model according to some implementations of this disclosure is shown;

[0016] Figure 8 A block diagram of training data for a machine learning model according to some implementations of this disclosure is shown;

[0017] Figure 9 A block diagram is shown illustrating the processing of an image to be colored, including non-closed regions, according to some implementations of this disclosure;

[0018] Figure 10 A flowchart is shown of a method for image colorization according to some implementations of this disclosure;

[0019] Figure 11 A block diagram of an apparatus for image colorization according to some implementations of the present disclosure is shown; and

[0020] Figure 12 A block diagram of a device capable of implementing various implementations of the present disclosure is shown. Detailed Implementation

[0021] Implementations of this disclosure will now be described in more detail with reference to the accompanying drawings. While some implementations of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the implementations set forth herein. Rather, these implementations are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and implementations of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0022] In the description of the implementation methods disclosed herein, the term "comprising" and similar terms should be understood as open inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one implementation" or "the implementation" should be understood as "at least one implementation". The term "some implementations" should be understood as "at least some implementations". Other explicit and implicit definitions may also be included below. As used herein, the term "model" can represent the relationships between various data. For example, the aforementioned relationships can be obtained based on various currently known and / or future-developed technical solutions.

[0023] It is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and related provisions.

[0024] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure through appropriate means in accordance with relevant laws and regulations, and user authorization should be obtained.

[0025] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the software or hardware, such as the electronic device, application, server, or storage medium performing the operations of this disclosed technical solution, based on the prompt message.

[0026] As an optional but non-restrictive implementation, in response to a user's active request, a prompt message can be sent to the user, for example, via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose whether to "agree" or "disagree" to provide personal information to the electronic device.

[0027] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0028] Example Environment

[0029] Figure 1 A block diagram of an example environment 100 in which the implementation of this disclosure can be implemented is shown. For example... Figure 1 As shown, a coloring process can be performed on the image 110 to be colored in order to obtain a colored image 120. In existing technical solutions, the cartoonist needs to manually select the areas to be colored in the image 110 and then fill them with color. Specifically, the coloring process can be further subdivided into three stages: flat coloring, retouching, and special effects. Here, the flat coloring stage fills each part of the character with a fixed color, which is the basic stage of coloring. The retouching stage adds light and shadow effects to the flat-colored image, and the special effects stage further enriches the visual information of the character.

[0030] Although region selection tools and color fill tools have been developed, their accuracy is often unsatisfactory. For example, in... Figure 1 In the image 120 shown, errors may occur in the colored mouth of a cartoon character 130. For example, blank areas may appear near the mouth's outline. In this case, the cartoonist has to use a brush to fill in the blank pixels one by one. Another example is that when the line art of a cartoon character involves non-closed areas, the color fill tool may overflow the correct colored area, and so on.

[0031] Comic artists need to color a large number of comics every day. At this time, they hope to complete the coloring process in a simpler and more efficient way, and they also hope that there will be no blank spaces and / or overflow errors in the colored images.

[0032] A summary of the image coloring process

[0033] To address the shortcomings of the aforementioned technical solutions, a method for image colorization is proposed based on an exemplary implementation of this disclosure. See [link to relevant documentation]. Figure 2 Describe the outline of the image coloring, Figure 2 A block diagram 200 for an image coloring process according to one technical solution is shown. (See diagram 200 for example.) Figure 2 As shown, a set of lines can be extracted from the image 110 to be colored (e.g., a line drawing of a comic book image). For ease of description, the detailed process of coloring is described using only the mouth area in image 110 as an example. Figure 2 As shown, the extracted set of lines 210 can represent the outline of the mouth of a cartoon character, and the set of lines 210 can include all the original pixels of the mouth outline. At this time, the width of each line in the set of lines 210 is the same as the width of the original outline in image 110.

[0034] Furthermore, a set of lines can be converted into a set of thinned lines 220. Then, for each target line in the set of lines 220, a thinning operation can be performed to generate a target thinned line corresponding to that target line. In this case, the target thinned line does not include all pixels of the target line, but only a portion of them. A target color can be used to color the filled area 230 in the image represented by the set of thinned lines 220; for example, pink can be filled into the filled area 230 where the mouth is located. Subsequently, a colored image 240 can be generated based on the image 110 and the colored filled area 230.

[0035] It will be understood that when using existing color filling techniques, if the lines are wide and / or involve gradient colors, threshold-based filling algorithms cannot accurately determine the mouth area surrounded by a set of lines 210, resulting in unfilled blank pixels at the mouth contour location. Unlike existing filling algorithms, because the target thinning lines contain fewer pixels than the target lines, the filling area will expand outward during the filling process and include the area at the mouth contour location (i.e., including blank pixels that cannot be filled by existing filling techniques).

[0036] Using the exemplary implementation of this disclosure, the fill area 230 will "expand" outward to a certain extent, thereby filling the original blank pixels with color. At this time, the colored image 240 will no longer include blank pixels; that is, there will be no incorrect "white edges" at the mouth outline position, thus improving the accuracy of image coloring. In this way, comic artists do not need to manually process the "white edges" in each image one by one, but can use a one-click fill function to complete the coloring process for each image. This method can greatly reduce the manual workload in the comic production process, thereby improving comic production efficiency.

[0037] Detailed process of image coloring

[0038] Having described an outline of an exemplary implementation according to this disclosure, further details regarding the image coloring process will be described below. Figure 3 A block diagram 300 for determining a fill region is shown according to some implementations of this disclosure. It will be understood that... Figure 3 The process of determining the fill area is described using only a simplified image containing triangles as an example. The image to be colored here may include other complex lines, and may include combinations of multiple lines, etc.

[0039] like Figure 3 As shown, Figure 312 represents the lines in image 310, Figure 322 represents the thinned lines, and Figure 332 represents the filled area. Here, the three lines (horizontal line 316, vertical line 318, and diagonal line 214) represent triangular regions. Each line can be processed individually to determine the corresponding thinned lines. Specifically, for a target line within a set of lines, the line center of the target line can be determined. The line center of the pixel region containing horizontal line 316 can be determined based on various methods currently known and / or those to be developed in the future.

[0040] For example, consider the horizontal line 316 in image 310, which has a width of 3 pixels. Multiple pixels located at the center of the horizontal line 316 can be identified. A target thinning line can be generated based on these pixels, resulting in a thinner line width smaller than the target line width. The pixel at the center of the line width can be used as the line center, thereby generating a thinned horizontal line 326 with a width of 1 pixel, as shown in image 320. Using this exemplary implementation, the extracted thinning lines ensure both line continuity and prevent the use of pixels that should be filled with color, thus accurately determining the range of the filling area.

[0041] Other lines in the image can be processed in a similar manner. For example, a thinned vertical line 328 with a width of one pixel can be generated based on the pixels at the center of the vertical line 318; similarly, a thinned diagonal line 324 with a width of one pixel can be generated based on the pixels at the center of the diagonal line 314. The area surrounded by the thinned lines is then the fill area 332. This fill area 332 can be colored using a target color to obtain a colored fill area.

[0042] It will become clear that the continuity of the thinned lines should be the same as that of the original lines. In other words, the thinning process does not change the closure of the area surrounded by a set of lines. If a set of lines defines a closed area, then the fill area defined by the set of thinned lines remains closed. In this way, when filling the fill area with color, there will be no problem of color overflowing from non-closed areas.

[0043] According to an exemplary implementation of this disclosure, a target color can be used to color a filled area. The target color can be obtained in various ways, such as receiving a target color specified by a user. Specifically, during the coloring process, the cartoonist can pre-select a target color, for example, by defining the target color based on specific RGB values, and so on. In one example, the image to be colored can be a blank line drawing of a cartoon character. In this case, a target color can be pre-selected and used to color various parts of the blank line drawing. For example, the mouth area in the image to be colored can be filled with pink, the hair area in the image to be colored can be filled with golden yellow, and so on.

[0044] Alternatively and / or additionally, the image to be colored here can be a partially colored line drawing, for example, a line drawing that has already been coarsely colored using other methods. In this case, a portion of the filled area has already been colored, so a target color can be extracted from the filled area and used to color the uncolored portions of the filled area. For example, assuming the center of the mouth area in the image to be colored is pink and there is a white border around it, the entire mouth area can be filled with pink.

[0045] According to an exemplary implementation of this disclosure, if the filled region 332 has already been colored, a colored image can be generated based on the image and the colored filled region. Figure 4 A block diagram 400 for generating a colored image is shown, according to some implementations of this disclosure. For example... Figure 4 As shown, the lines 312 and the fill area in image 310 can be combined to generate a colored image 420. Specifically, a new layer can be created in image 310 to store the colored fill area 332. Here, the new layer can be located below the original layer in image 310 (i.e., the layer containing the lines 312). In other words, the new layer and the original layer can be overlaid to obtain the colored image 420.

[0046] By using the exemplary implementation of this disclosure, data generated at each stage of the coloring process can be stored in different layers, which facilitates later adjustments and modifications. For example, when it is desired to use a different color to perform the coloring process, the color in the fill area 332 in the new layer can be directly replaced, without having to re-execute the processes of extracting lines, determining thinning lines, and determining the fill area.

[0047] It will be understood that, although the above description addresses the case where the line width is odd, when the line width is even, thinner lines can be generated based on pixels closer to the outer edge of the line. In other words, a set of thinner lines can be based on the outermost ring of pixels in a set of lines. In this case, the fill area surrounded by a set of thinner lines will expand outward by one or more pixels relative to the area surrounded by the set of lines, thereby obtaining a more reliable fill area. Using the exemplary implementation of this disclosure, the fill area 332 will be larger than the area surrounded by the original set of lines. In this way, white edges will not appear in the colored image 420, thus improving the accuracy of image coloring.

[0048] Will understand, although Figure 3 The illustration shows a case where all pixels in a line have the same color, but the colors of individual pixels can also differ. Typically, in comic book line art, pixels at the center of the line are darker (e.g., black) and pixels at the edges are typically lighter (e.g., gray). In this case, if a fill algorithm based on a predetermined threshold is applied directly to the area surrounded by a set of lines, different predetermined thresholds will result in white borders of varying widths in that area. However, using the exemplary implementation of this disclosure, the fill area 332 is generated based on thinned lines 322 (i.e., lines at the center of the line), allowing for a wider range of color filling and thus avoiding the problem of white borders.

[0049] According to an exemplary implementation of this disclosure, two scenarios may arise when determining a region represented by a set of thinned lines: the region is a closed region, or the region is an open region. First, details regarding the handling of closed regions are described. If the region represented by the set of thinned lines is determined to be a closed region, it can be directly identified as a filled region. Then, a currently known and / or future-developed filling algorithm can be used to color the filled region. If the set of thinned lines is determined to represent an open region, directly calling a conventional filling algorithm for coloring may result in color overflow.

[0050] Figure 5 A block diagram 500 of an image to be colored, including non-closed regions, is shown according to some implementations of this disclosure. For example... Figure 5 As shown, image 110 may include discontinuous lines; for example, there may be discontinuous lines 510 at the position of the upper garment of a cartoon character. In this case, if a conventional fill algorithm is directly applied to color the upper garment of the cartoon character, an overflow problem 530, as shown in image 520, will occur. That is, the gaps in the lines at the upper garment cause the color of the upper garment to overflow into the background area, and the entire background area of ​​the image is colored.

[0051] According to one exemplary implementation of this disclosure, in order to perform the coloring process in a more accurate manner, non-closed areas can be corrected based on a machine learning model in order to determine the filling area. Figure 6 Some implementations of this disclosure are shown for use with Figure 5 The image shown is colored using a bounding box 600. (See attached image 600.) Figure 6 As shown, machine learning model 610 can be used to correct non-closed regions caused by discontinuous lines 510. Color can be filled into the corrected regions, and in the resulting colored image 620, there is only slight overflow 630 near the discontinuous lines.

[0052] Here, the machine learning model 610 can describe the relationship between a set of lines in an image and a filled area of ​​a single color represented by the set of lines. In other words, the machine learning model 610 can correct a large, unclosed region represented by a set of lines into a closed region. Although the corrected closed region may have overflow problems, the severity of the overflow problem will be significantly reduced, and it will not overflow into a large surrounding area.

[0053] See below. Figure 7 More details about machine learning model 610 are described below. Figure 7 A block diagram 700 of a machine learning model according to some implementations of this disclosure is shown. For example... Figure 7 As shown, the machine learning model 610 may include a semantic segmentation model 720 (e.g., referred to as the first model), which can describe the association between a set of lines and the semantics of the regions represented by those lines (e.g., parts of a cartoon character). According to an exemplary implementation of this disclosure, multiple semantics can be predefined, including but not limited to: the background of an image, and the whites of the eyes, irises, eyebrows, mouth, face, hair, hair accessories, earrings, glasses, clothing, hands, torso skin, pants, skirt, socks, shoes, legs, etc., of a cartoon character in the image. Here, each semantic has a specific definition; for example, background can represent the background of a cartoon character; the whites of the eyes can generally be the white area between the eyelashes and the pupil; hair accessories can represent decorations near the hair, including but not limited to hats, hoodies, headbands, headphones, decorative rabbit / cat ears, flowers, crowns, hair clips, etc.

[0054] According to an exemplary implementation of this disclosure, a semantic segmentation model 720 in a machine learning model can be used to determine a semantic segmentation map associated with a set of lines, where semantic blocks in the semantic segmentation map correspond to semantic regions in the image to be colored. Specifically, the semantic segmentation model 720 can be implemented based on various model architectures currently known and / or to be developed in the future (e.g., the HRNet architecture), and the loss function can be determined based on cross-entropy loss to complete the training. The semantic segmentation model 720 can be trained using training data, see [link to relevant documentation]. Figure 8 Describe more details about the training data.

[0055] Should Figure 8 A block diagram 800 showing training data for a machine learning model according to some implementations of this disclosure is illustrated. For example... Figure 8 As shown, the training data for the semantic segmentation model 720 may include a line drawing 810 and corresponding semantic annotations 820. Different colors can be used to represent different semantics; for example, background 822 can represent the background of an image, hair 824 can represent the hair of a cartoon character, and so on. A large number of line drawings and corresponding semantic annotations can be obtained, and further, this training data can be used to iteratively train the semantic segmentation model 720. After the training process is completed, when the line drawing 710 containing the image to be processed is input into the semantic segmentation model 720, the semantic segmentation model 720 can output a semantic segmentation map 722.

[0056] return Figure 7 According to an exemplary implementation of this disclosure, the machine learning model 610 may further include a color partitioning model 730 (e.g., referred to as a second model). The color partitioning model 730 can describe the association between a set of lines and the colors of the regions represented by those lines. Here, the color partitioning model 730 can receive a line drawing and output a corresponding color partitioning map 732. At this point, the color partitioning model 730 in the machine learning model 610 can be used to determine the color partitioning map associated with a set of lines, where the color blocks in the color partitioning map correspond to filled areas with a single color in the image to be colored. The color partitioning map 732 can represent the color areas to be colored in the line drawing, and these color areas are independent of whether the lines in the line drawing are actually strictly closed. In this way, it is possible to support the determination of specific color areas and thus correct the boundaries of non-closed areas.

[0057] The color partitioning model 730 can be implemented based on various network architectures currently known and / or those to be developed in the future (e.g., the UNet architecture). During training, a large number of images of cartoon characters can be collected, and line art and color block maps for each character can be derived from them. Furthermore, corresponding partition maps can be obtained based on the color block maps, so that different colored regions in the color block maps have different color labels. Furthermore, a loss function can be determined based on the difference between the predicted color partition map and the ground truth data. This loss function can include loss determined based on labeled data and perceptual loss, etc. Since the color block map is based on color image processing, the color partition map 732 output by the trained color partitioning model 730 does not depend on whether the region defined by the lines in the line art is strictly closed, but can infer whether the region should be closed based on the specific distribution of the lines.

[0058] The color partitioning model 730 can be trained using training data, which may include line art of a cartoon character and already colored cartoon characters. The color partitioning model 730 can then acquire knowledge about which areas to colorize, thus completing the training process. When the line art 710 of the image to be processed is input into the color model 730, the color partitioning model 730 can output a color partitioning map 732.

[0059] Furthermore, non-closed regions can be updated based on semantic segmentation map 722 and color partition map 732 to determine the filling region. It will be understood that regions with a certain semantic meaning can be quickly determined based on semantic segmentation map 720; however, the edges of segmentation map 720 may not be accurate enough and affect the accuracy of coloring. The edges of color partition map 732 depend on the line direction in the line drawing, thus allowing for pixel-level accuracy. According to an exemplary implementation of this disclosure, machine learning model 610 may include an edge optimizer 750, which can be used to process semantic segmentation map 722 and color partition map 732 to determine edge data 750 in line drawing 710. Here, the primary goal of the edge optimizer is to ensure that the edges of semantic segmentation map 722 and color partition map 732 accurately coincide, thereby outputting the final edge data 750. In this way, a more accurate filling region can be obtained.

[0060] According to an exemplary implementation of this disclosure, for a certain color block in the color partitioning map 732, the semantic meaning corresponding to the color block can be determined based on a comparison between the color block and semantic blocks in the semantic segmentation map. Specifically, a blank segmentation map can be established, and each color block in the color partitioning map 732 can be traversed. For example, for the orange block where the top is located, based on a comparison between the orange block and each block in the semantic segmentation map 722, it can be determined that the orange block corresponds to the semantic "top". In other words, the semantic label with the most pixels in the current region can be found, a new region can be established in the blank segmentation map that is consistent with the current region, and the found segmentation map label can be set.

[0061] Furthermore, edge optimization can be performed based on the comparison of semantic blocks and non-closed regions in the semantic segmentation map to determine the filling region. After traversing all regions of the segmentation map, a new segmentation map is created. The edges of the new segmentation map completely coincide with the edges of the segmentation map, thereby achieving edge optimization of the segmentation map and obtaining edge data 750. Furthermore, the edge data 750 can be used to update the non-closed regions determined based on the process described above to obtain the corresponding filling regions. Using the exemplary implementation of this disclosure, the edges of the determined filling regions depend on multiple factors, and not only on whether the regions defined by individual lines are closed. In this way, the range of the filling region can be determined based on pixel granularity, thereby avoiding errors caused by large-scale overflow due to gaps in discontinuous lines in the line drawing.

[0062] Figure 9 A block diagram 900 illustrates the processing of an image to be colored, including non-closed regions, according to some implementations of this disclosure. For example... Figure 9 As shown, the machine learning model 610 described above can be used to update the filled region, thereby obtaining image 620. Figure 9 As shown, image 620 currently only includes a small area of ​​overflow 630. Furthermore, the user can use the eraser tool 910 to process this overflow 630, thus obtaining image 920. At this point, in the area 930 near the discontinuous lines, although there are gaps in the lines of the cartoon character's upper garment, simple modifications can be made to control the range of the upper garment color within the correct upper garment area.

[0063] Although the method of this disclosure has been described above using a single image as an example, alternatively and / or additionally, one or more images to be colored can be extracted from multimedia data, and the coloring process described above can be performed on each extracted image. Here, multimedia data can include at least one of the following: graphic data, image sets, video data, game data, etc. Specifically, images to be colored can be extracted from graphic data, for example, data including images and text, and coloring can be performed on objects in the images, thus obtaining graphic data including colored images. For example, images to be colored can be extracted from a specified directory in an image set, images to be colored can be extracted from each image frame in video data, or images to be colored can be extracted from game data, etc. In this way, the coloring process can be performed on multimedia data with different formats in a more flexible manner, thereby making the coloring process of this disclosure compatible with more existing multimedia production applications.

[0064] It will be understood that although objects are shown as human figures in the accompanying drawings, alternatively and / or additionally, the image to be colored may include other entities. For example, in a landscape image, objects may include trees, houses, mountains, lakes, etc. Parts in this case may include, for example, tree trunks, branches, leaves, walls, rooftops, etc. As another example, in an interior image, objects may include furniture, appliances, etc. Parts in this case may include tabletops, table legs, cushions, etc. Using the exemplary implementation of this disclosure, the appropriate color can be selected and the coloring process performed based on the specific semantics of the areas defined by the lines in the image. In this way, the various contents included in a comic image can be processed more efficiently, thereby improving the efficiency of comic production.

[0065] According to an exemplary implementation of this disclosure, when an image to be colored involves multiple layers, a layer can be selected and the characters in that layer can be colored using the techniques described above. In this case, coloring is performed only on the selected layer, and characters in unselected layers are not processed.

[0066] According to an exemplary implementation of this disclosure, a color template can be used to first perform coarse-grained one-click coloring. The obtained image can then be further processed using the techniques described above, for example, by selecting a target color and then clicking on a specific area in the line drawing to fill in blank pixels within that area. Alternatively and / or additionally, assuming the local area is not closed, the image processed using the techniques described above will not produce large-scale color overflow, but rather the fill color can be limited to the correct coloring range.

[0067] Example process

[0068] Figure 10A flowchart of a method 1000 for coloring an image according to some implementations of this disclosure is shown. At box 1010, a set of lines is extracted from the image to be colored; at box 1020, the set of lines is converted into a set of thinned lines, wherein a target thinned line in the set of thinned lines includes a portion of pixels in a target line corresponding to the target thinned line in the set of lines; at box 1030, a target color is used to color a filled area in the image represented by the set of thinned lines; and at box 1040, a colored image of the image is generated based on the image and the colored filled area.

[0069] According to an exemplary implementation of this disclosure, converting a set of lines into a set of thinned lines includes: determining the line center of the target lines; and generating the target thinned lines based on the pixels at the line center.

[0070] According to one exemplary implementation of this disclosure, the width of the target thinning line is smaller than the width of the target line.

[0071] According to one exemplary implementation of this disclosure, the width of the target thinning line is 1 pixel.

[0072] According to an exemplary implementation of this disclosure, the method 1000 further includes at least one of the following: in response to determining that a set of thinned lines represents a closed region, identifying the closed region as a filled region; in response to determining that a set of thinned lines represents a non-closed region, correcting the non-closed region based on a machine learning model to determine the filled region, the machine learning model describing the association between a set of lines in an image and a filled region of a single color represented by the set of lines.

[0073] According to an exemplary implementation of this disclosure, determining the filling region based on a machine learning model includes: using a first model in the machine learning model to determine a semantic segmentation map associated with a set of lines, wherein semantic blocks in the semantic segmentation map correspond to semantics in the image to be colored; using a second model in the machine learning model to determine a color partition map associated with a set of lines, wherein color blocks in the color partition map correspond to a single color in the image to be colored; and updating non-closed regions based on the semantic segmentation map and the color partition map to determine the filling region.

[0074] According to an exemplary implementation of this disclosure, determining the filling region based on the semantic segmentation map and the color partitioning map includes: for a color block in the color partitioning map, determining the semantic corresponding to the color block based on a comparison between the color block and the semantic block in the semantic segmentation map; and determining the filling region based on a comparison between semantic blocks with semantic meaning and non-closed regions in the semantic segmentation map.

[0075] According to an exemplary implementation of this disclosure, generating a colored image of an image includes: creating a new layer in the image, the new layer being located below the layers of the image; and storing the colored fill area in the new layer.

[0076] According to an exemplary implementation of this disclosure, the method 1000 further includes determining a target color based on at least one of the following: extracting a target color from a filled area; receiving a target color specified by a user.

[0077] According to an exemplary implementation of this disclosure, the image to be colored includes a comic image, and the comic image includes at least one of the following: a line drawing of the comic image, or a partially colored line drawing of the comic image.

[0078] According to an exemplary implementation of this disclosure, the method 1000 further includes: extracting the image to be colored from multimedia data, wherein the multimedia data includes at least one of the following: text data, image set, video data, and game data.

[0079] Example devices and equipment

[0080] Figure 11 A block diagram 1100 of an apparatus for coloring an image according to some implementations of the present disclosure is shown. The apparatus 1100 includes: an extraction module 1110 configured to extract a set of lines from an image to be colored; a conversion module 1120 configured to convert the set of lines into a set of thinned lines, wherein a target thinned line in the set of thinned lines includes a portion of pixels in a target line corresponding to the target thinned line in the set of lines; a coloring module 1130 configured to color a filled area in the image represented by the set of thinned lines using a target color; and a generation module 1140 configured to generate a colored image of the image based on the image and the colored filled area.

[0081] According to an exemplary implementation of this disclosure, the conversion module 1120 includes: a center determination module configured to determine the line center of the target line for the target line; and a line generation module configured to generate a target thinning line based on the pixels at the line center.

[0082] According to one exemplary implementation of this disclosure, the width of the target thinning line is smaller than the width of the target line.

[0083] According to one exemplary implementation of this disclosure, the width of the target thinning line is 1 pixel.

[0084] According to an exemplary implementation of this disclosure, the apparatus 1100 further includes at least one of the following: a first filling region determination module configured to identify a closed region as a filling region in response to determining that a set of thinned lines represents a closed region; and a second filling region determination module configured to correct the non-closed region based on a machine learning model in response to determining that a set of thinned lines represents a non-closed region in order to determine the filling region, wherein the machine learning model describes the association between a set of lines in an image and a filling region of a single color represented by the set of lines.

[0085] According to an exemplary implementation of this disclosure, the second filling region determination module includes: a segmentation map determination module configured to determine a semantic segmentation map associated with a set of lines using a first model in a machine learning model, wherein semantic blocks in the semantic segmentation map correspond to semantics in the image to be colored; a partition map determination module configured to determine a color partition map associated with a set of lines using a second model in a machine learning model, wherein color blocks in the color partition map correspond to a single color in the image to be colored; and an update module configured to update non-closed regions based on the semantic segmentation map and the color partition map in order to determine the filling region.

[0086] According to an exemplary implementation of this disclosure, the update module includes: a first comparison module configured to determine the semantics corresponding to a color block in a color partitioning map based on a comparison between the color block and a semantic block in a semantic segmentation map; and a second comparison module configured to determine the filling region based on a comparison between semantic blocks with semantics and non-closed regions in the semantic segmentation map.

[0087] According to an exemplary implementation of this disclosure, the generation module 1140 includes: a creation module configured to create a new layer in an image, the new layer being located below the layers of the image; and a storage module configured to store the colored fill area in the new layer.

[0088] According to an exemplary implementation of this disclosure, the apparatus 1100 further includes a target color determination module configured to determine a target color based on at least one of the following: extracting a target color from a filled area; or receiving a target color specified by a user.

[0089] According to an exemplary implementation of this disclosure, the image to be colored includes a comic image, and the comic image includes at least one of the following: a line drawing of the comic image, or a partially colored line drawing of the comic image.

[0090] According to an exemplary implementation of this disclosure, the apparatus 1100 further includes: extracting an image to be colored from multimedia data, wherein the multimedia data includes at least one of the following: text data, image set, video data, and game data.

[0091] Figure 12 A block diagram of a device 1200 capable of implementing various implementations of the present disclosure is shown. It should be understood that... Figure 12 The computing device 1200 shown is merely exemplary and should not be construed as limiting the functionality and scope of the implementation described herein. Figure 12 The computing device 1200 shown can be used to implement the method described above.

[0092] like Figure 12 As shown, computing device 1200 is in the form of a general-purpose computing device. Components of computing device 1200 may include, but are not limited to, one or more processors or processing units 1210, memory 1220, storage devices 1230, one or more communication units 1240, one or more input devices 1250, and one or more output devices 1260. Processing unit 1210 may be a physical or virtual processor and is capable of performing various processes according to programs stored in memory 1220. In a multiprocessor system, multiple processing units execute computer-executable instructions in parallel to improve the parallel processing capability of computing device 1200.

[0093] Computing device 1200 typically includes multiple computer storage media. Such media can be any available media accessible to computing device 1200, including but not limited to volatile and non-volatile media, removable and non-removable media. Memory 1220 can be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. Storage device 1230 can be removable or non-removable media and can include machine-readable media, such as flash drives, disks, or any other media that can be used to store information and / or data (e.g., training data for training) and can be accessed within computing device 1200.

[0094] The computing device 1200 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not explicitly stated... Figure 12As shown, disk drives for reading from or writing to removable, non-volatile disks (e.g., "floppy disks") and optical disk drives for reading from or writing to removable, non-volatile optical disks can be provided. In these cases, each drive can be connected to a bus (not shown) via one or more data media interfaces. Memory 1220 may include computer program product 1225 having one or more program modules configured to perform various methods or actions of various implementations of this disclosure.

[0095] The communication unit 1240 enables communication with other computing devices via a communication medium. Additionally, the functionality of the components of the computing device 1200 can be implemented as a single computing cluster or multiple computing machines capable of communicating via communication connections. Therefore, the computing device 1200 can operate in a networked environment using logical connections to one or more other servers, networked personal computers (PCs), or another network node.

[0096] Input device 1250 can be one or more input devices, such as a mouse, keyboard, trackball, etc. Output device 1260 can be one or more output devices, such as a monitor, speaker, printer, etc. Computing device 1200 can also communicate with one or more external devices (not shown) via communication unit 1240 as needed. These external devices include storage devices, display devices, etc., and can communicate with one or more devices that enable user interaction with computing device 1200, or with any device (e.g., network card, modem, etc.) that enables computing device 1200 to communicate with one or more other computing devices. Such communication can be performed via input / output (I / O) interface (not shown).

[0097] According to exemplary implementations of this disclosure, a computer-readable storage medium is provided that stores computer-executable instructions thereon, wherein the computer-executable instructions are executed by a processor to implement the methods described above. According to exemplary implementations of this disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, which are executed by a processor to implement the methods described above. According to exemplary implementations of this disclosure, a computer program product is provided that stores a computer program thereon, which, when executed by a processor, implements the methods described above.

[0098] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, devices, and computer program products implemented according to this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0099] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0100] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions that execute on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0101] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0102] Various implementations of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is chosen to best explain the principles, practical applications, or improvements to technology in the market, or to enable others skilled in the art to understand the various implementations disclosed herein.

Claims

1. A method for image coloring, comprising: extracting a set of lines from an image to be colored; converting the set of lines to a set of thinned lines, a target thinned line in the set of thinned lines including a portion of pixels in a target line in the set of lines corresponding to the target thinned line; coloring a filled region in the image represented by the set of thinned lines with a target color, including, in response to determining that the set of thinned lines represents a non-closed region, determining a semantic segmentation map associated with the set of lines using a first model in a machine learning model, the machine learning model describing an association between a set of lines in an image and a filled region having a single color represented by the set of lines, a semantic patch in the semantic segmentation map corresponding to a semantic in the image to be colored; determining a color partition map associated with the set of lines using a second model in the machine learning model, a color patch in the color partition map corresponding to a single color in the image to be colored; and updating the non-closed region based on the semantic segmentation map and the color partition map so as to determine the filled region; and generating a colored image of the image based on the image and the filled region colored.

2. The method of claim 1, wherein converting the set of lines to a set of refined lines comprises: for the target line, determining a line center of the target line; and generating the target thinned line based on a pixel at the line center.

3. The method of claim 2, wherein a width of the target thinned line is less than a width of the target line.

4. The method of claim 3, wherein the width of the target thinned line is 1 pixel.

5. The method of claim 1, further comprising: in response to determining that the set of thinned lines represents a closed region, identifying the closed region as the filled region.

6. The method of claim 1, wherein determining the fill region based on the semantic segmentation map and the color partition map comprises: for a color patch in the color partition map, determining a semantic corresponding to the color patch based on a comparison of the color patch with a semantic patch in the semantic segmentation map; and and determining the filled region based on a comparison of a semantic patch having the semantic in the semantic segmentation map and the non-closed region.

7. The method of claim 1, wherein generating a colored image of the image comprises: creating a new layer in the image, the new layer being below a layer of the image; and storing the filled region colored in the new layer.

8. The method of claim 1, further comprising determining the target color based on at least one of: extracting the target color from the filled region; receiving the target color specified by a user.

9. The method of claim 1, wherein the image to be colored comprises a comic image, and the comic image comprises at least one of: a line art of the comic image, a partially colored line art of the comic image.

10. The method of claim 1, further comprising: extracting the image to be colored from multimedia data, the multimedia data comprising at least one of: text data, a set of images, video data, game data.

11. An apparatus for image coloring, comprising: an extraction module configured to extract a set of lines from an image to be colored; a conversion module configured to convert the set of lines into a set of thinned lines, a target thinned line in the set of thinned lines including a portion of pixels in a target line in the set of lines corresponding to the target thinned line; a coloring module configured to color a filled region in the image represented by the set of thinned lines with a target color, the coloring module further configured to: determine, in response to determining that the set of thinned lines represents a non-closed region, a semantic segmentation map associated with the set of lines using a first model in a machine learning model, the machine learning model describing an association between a set of lines in an image and a filled region having a single color represented by the set of lines, a semantic patch in the semantic segmentation map corresponding to a semantic in the image to be colored; a color partition map associated with the set of lines using a second model in the machine learning model, a color patch in the color partition map corresponding to a single color in the image to be colored; and update the non-closed region based on the semantic segmentation map and the color partition map so as to determine the filled region; and a generation module configured to generate a colored image of the image based on the image and the filled region colored.

12. An electronic device, comprising: at least one processing unit; and at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions when executed by the at least one processing unit cause the electronic device to perform the method according to any one of claims 1-10.

13. A computer-readable storage medium having stored thereon a computer program which, when executed by a processor, causes the processor to implement the method according to any one of claims 1-10. ​

Images