Line draft graph coloring processing method and device, computer equipment and storage medium

Abstract

The invention relates to a color draft pattern coloring processing method and device, computer equipment, a computer readable storage medium and a computer program product. The method comprises the steps of obtaining a target line draft graph to be colored and a reference line draft graph corresponding to a reference color draft graph from a line draft graph sequence corresponding to a target object; with the reference color draft graph as a reference, coloring the target line draft graph based on the time information of the target line draft graph and the reference line draft graph in the line draft graph sequence to obtain an initial color draft graph corresponding to the target line draft graph; determining a first area contained in the corresponding initial color draft image based on the closed area in the target line draft image, and determining a second area contained in the corresponding reference color draft image based on the closed area in the reference line draft image; and performing color correction on a target first area matched with the target second area in the first area based on the area image of the target second area in the second area to obtain a target color draft image corresponding to the target line draft image. By adopting the method, the coloring accuracy can be improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, apparatus, computer equipment, computer-readable storage medium, and computer program product for coloring a line drawing. Background Art

[0002] With the rapid development of computer technology, more and more users use computers to draw line drawings and color the drawn line drawings. On this basis, a technology that uses artificial intelligence methods to automatically color line drawings has also emerged.

[0003] Traditionally, coloring is done by using a specific color draft as a reference to colorize the line drawing. This automated coloring technology is far more efficient than manual coloring, but the coloring results can easily deviate from actual requirements, resulting in poor coloring accuracy. Summary of the Invention

[0004] Based on this, it is necessary to provide a line drawing coloring processing method, device, computer equipment, computer-readable storage medium and computer program product that can improve coloring accuracy in response to the above technical problems.

[0005] In a first aspect, the present application provides a method for coloring a line drawing. The method comprises:

[0006] Obtaining the target line drawing to be colored and the reference line drawing corresponding to the reference color drawing from the line drawing sequence corresponding to the target object;

[0007] Taking the reference color draft image as a reference, coloring the target line draft image based on time information of the target line draft image and the reference line draft image in the line draft image sequence to obtain an initial color draft image corresponding to the target line draft image;

[0008] Determine a first area included in the corresponding initial color draft image based on the closed area in the target line draft image, and determine a second area included in the corresponding reference color draft image based on the closed area in the reference line draft image;

[0009] A target first area and a target second area having a matching relationship are determined from the first area and the second area, and color correction is performed on the target first area based on the area image of the target second area to obtain a target color draft image corresponding to the target line draft image.

[0010] In one embodiment, the method further comprises:

[0011] Determining a first time corresponding to the target line drawing in the line drawing sequence, and a second time corresponding to the candidate line drawing in the line drawing sequence;

[0012] determine a matching degree between the candidate sketch and the target sketch based on a time interval between the second time and the first time; the matching degree is inversely related to the time interval.

[0013] In one of the embodiments, the method further comprises:

[0014] determine a first object pose of the target object in the target sketch and a second object pose of the target object in the candidate sketch;

[0015] determine that the candidate sketch matches the target sketch when a pose similarity between the second object pose and the first object pose satisfies a similarity condition.

[0016] In one of the embodiments, the determining, from the second regions, a candidate region with the highest similarity of region image features to the first region comprises:

[0017] determine a number of pixel points corresponding to each of the second regions and a number of pixel points corresponding to the first region;

[0018] determine, from the second regions, a candidate region with the highest similarity of region image features to the first region;

[0019] determine, from the candidate regions, a candidate region with the highest similarity of region image features to the first region.

[0020] In one of the embodiments, the determining, from the first region and the second region, a target first region and a target second region with a matching relationship comprises:

[0021] determine, according to image information contained in the first region and the second region, an image correspondence relationship between the first region and the second region;

[0022] determine, according to the image correspondence relationship, a target second region from the second region that matches at least part of image content of the first region;

[0023] determine, from the first region, a target first region corresponding to the at least part of image content.

[0024] In a second aspect, the application further provides a sketch coloring processing device. The device comprises:

[0025] an obtaining module configured to obtain, from a sequence of sketches corresponding to a target object, a target sketch to be colored and a reference sketch corresponding to a reference color sketch;

[0026] a coloring module, configured to color the target line drawing based on the reference line drawing and time information of the target line drawing and the reference line drawing in the sequence of line drawings, to obtain an initial color draft corresponding to the target line drawing;

[0027] a region division module, configured to determine a first region contained in the initial color draft based on a closed region in the target line drawing, and determine a second region contained in the reference color draft based on a closed region in the reference line drawing;

[0028] a color correction module, configured to determine a target first region and a target second region with a matching relationship from the first region and the second region, and perform color correction on the target first region based on a region image of the target second region, to obtain a target color draft corresponding to the target line drawing.

[0029] In a third aspect, a computer device is provided. The computer device includes a memory and a processor. The memory stores a computer program. The processor implements the line drawing coloring processing method when executing the computer program.

[0030] In a fourth aspect, a computer readable storage medium is provided. The computer readable storage medium stores a computer program. The computer program is executed by a processor to implement the line drawing coloring processing method.

[0031] In a fifth aspect, a computer program product is provided. The computer program product includes a computer program. The computer program is executed by a processor to implement the line drawing coloring processing method.

[0032] The above-mentioned line drawing coloring processing method, apparatus, computer device, computer-readable storage medium, and computer program product obtain a target line drawing to be colored and a reference line drawing corresponding to a reference color drawing from a line drawing sequence corresponding to a target object. Then, using the reference color drawing as a reference, the target line drawing in the line drawing sequence is colored based on the time information of the target line drawing and the reference line drawing in the line drawing sequence, thereby obtaining an initial color drawing corresponding to the target line drawing. Because the coloring process refers to the reference color drawing corresponding to the reference line drawing in the line drawing sequence of the same target object, the semantic consistency and detail correspondence between the reference color drawing and the target line drawing can be ensured, so that the initial color drawing and the reference color drawing can maintain a high degree of consistency in color and style, which is conducive to improving coloring accuracy. In addition, the coloring process also takes into account the time information of the target line drawing and the reference line drawing in the line drawing sequence, which can better understand the semantic relationship between the line drawings, thereby more accurately transferring the color information of the reference color drawing to the target line drawing, which is conducive to further improving coloring accuracy. On this basis, the first area included in the corresponding initial color draft image is determined based on the closed area in the target line draft image, and the second area included in the corresponding reference color draft image is determined based on the closed area in the reference line draft image. Then, the target first area and the target second area with a matching relationship are determined from the first area and the second area. Based on the area image of the target second area, the target first area is color corrected to obtain the target color draft image corresponding to the target line draft image. On the basis of the initial color draft image, based on the area matching relationship between the reference color draft image and the initial color draft image, the target first area in the initial color draft image that matches the target second area can be color corrected, which can further ensure the consistency in color and style, and is conducive to further improving the coloring accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 A diagram illustrating an application environment of a method for coloring a line drawing in one embodiment;

[0034] Figure 2 A schematic flow chart of a method for coloring a line drawing in one embodiment;

[0035] Figure 3 Schematic diagram of a coloring process based on a reference color draft image in one embodiment;

[0036] Figure 4 A schematic diagram of a process of dividing a color draft image sequence into multiple target line draft groups for coloring in one embodiment;

[0037] Figure 5 A schematic diagram of a process for batch coloring multiple frames of target line drawings with reference to the same frame of reference color drawings in one embodiment;

[0038] Figure 6 A schematic diagram of a pixel correction process in one embodiment;

[0039] Figure 7 A schematic diagram of a process for performing region similarity matching in one embodiment;

[0040] Figure 8 is a schematic diagram of a color pair extraction process in one embodiment;

[0041] Figure 9 Schematic diagram of a color complementation process in one embodiment;

[0042] Figure 10 A schematic flow chart of a method for coloring a color draft image in another embodiment;

[0043] Figure 11 A schematic diagram of an application process of coloring a color draft image in one embodiment;

[0044] Figure 12 A schematic diagram of the finishing process of coloring a color draft image in one embodiment;

[0045] Figure 13 FIG. 1 is a schematic diagram of a coloring process for a target line drawing 1302 in one embodiment;

[0046] Figure 14 It is a structural block diagram of a color draft image coloring processing device in one embodiment;

[0047] Figure 15 is a diagram of the internal structure of a computer device in one embodiment;

[0048] Figure 16 FIG. 4 is a diagram showing the internal structure of a computer device in another embodiment. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0050] The line drawing coloring processing method provided in the embodiment of the present application can be applied to Figure 1In the application environment shown. Among them, the terminal 100 communicates with the server 200 through a network. The communication network can be a wired network or a wireless network. Therefore, the terminal 100 and the server 200 can be directly or indirectly connected through wired or wireless communication. For example, the terminal 100 can be indirectly connected to the server 200 through a wireless access point, or the terminal 100 can be directly connected to the server 200 through the Internet, and this application does not limit this. The data storage system can be set up separately, integrated on the server 200, or placed on the cloud or other devices.

[0051] The terminal 100 may be, but is not limited to, various desktop computers, laptops, smartphones, tablet computers, IoT devices, and portable wearable devices. IoT devices may include smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, and the like. Portable wearable devices may include smart watches, smart bracelets, head-mounted devices, and the like. The server 200 may be an independent physical server, or a server cluster or distributed system consisting of multiple physical servers. It may also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. The data storage system may store data that the server 200 needs to process.

[0052] It should be noted that the line drawing processing method in the embodiment of the present application can be executed by the terminal 100 or the server 200 alone, or by the terminal 100 and the server 200 together. Figure 1 As shown, the terminal 100 can obtain a line drawing sequence from the server 200, and the line drawing sequence may include a target line drawing to be colored, and a reference line drawing corresponding to a reference color drawing. Among them, the line drawing sequence can be uploaded to the server 200 by the line drawing provider through the terminal held by the line drawing provider, so that the server 200 forwards it to the terminal 100. The line drawing provider can be a draftsman or other person with the right to use the line drawing. Similarly, the reference color drawing can be uploaded to the server 200 by the color drawing provider through the terminal held by the color drawing provider, so that the server 200 forwards it to the terminal 100. The color drawing provider can be a coloring person or other person with the right to use the color drawing. Optionally, the user can directly draw the line drawing based on the terminal 100 to form a line drawing sequence 110, and manually color the reference line drawing 111 in the line drawing sequence 110 to obtain a reference color drawing 112.

[0053] When the line drawing sequence 110 and the reference color drawing 112 are obtained, the terminal 100 can use the reference color drawing 112 as a reference, and based on the target line drawing 113 and the time information of the reference line drawing corresponding to the reference color drawing 112 in the line drawing sequence, color the target line drawing 113 to obtain the initial color drawing 114 corresponding to the target line drawing 113. On this basis, the terminal 100 can determine the first area included in the corresponding initial color drawing 114 based on the closed area in the target line drawing 113, and determine the second area included in the corresponding reference color drawing 112 based on the closed area in the reference line drawing 111. Then, from the first area and the second area, determine the target first area and the target second area that have a matching relationship, and perform color correction on the target first area based on the area image of the target second area to obtain the target color drawing 115 corresponding to the target line drawing 113. Figure 1 , the matching target first area may be, for example, the eyeball area 1141 .

[0054] In some embodiments, a method for coloring a line drawing is provided, which can be applied to a computer device, which can be a terminal or a server. Figure 2 As shown, the method includes the following steps:

[0055] Step S202 : obtaining a target line drawing to be colored and a reference line drawing corresponding to the reference color drawing from a line drawing sequence corresponding to the target object.

[0056] Among them, the line drawing is also called line drawing, sketch or hand-drawn draft. It is a sketch drawn by using lines to express the shape, structure and texture of an object. The line drawing can use simple and clear lines to outline the outline and internal structure of the object, showing the basic form and composition structure of the drawn object. The various lines in the line drawing can be combined to form multiple different areas. Each area is a blank area and needs to be filled with color, so as to achieve coloring processing for the line drawing and obtain a color drawing with added color. Figure 3 As shown, by coloring the line drawing 301 , a corresponding color drawing 302 can be obtained.

[0057] The target object refers to the object depicted in the line drawing. This target object can include people, animals, vehicles, and more. People include anime characters, comic characters, ancient Chinese characters, and two-dimensional characters. Animals include cartoon animals, real animals, and pets. Vehicles include cars, trains, and airplanes. The object's structural components can include the facial features and clothing of people and animals, doors, windows, lighting, and beltlines of cars, and the fuselage, wings, tail, and landing gear of airplanes.

[0058] A line drawing sequence is an image sequence formed by arranging multiple frames of line drawings in time. It can be understood that through the line drawing sequence, the changes in posture, expression, position, etc. of the target object can be depicted. Both the target line drawing and the reference line drawing belong to the line drawing sequence. The difference is that the target line drawing is the line drawing to be colored, while the reference line drawing is the line drawing corresponding to the reference color drawing. Among them, the reference color drawing is the color drawing obtained by coloring the reference line drawing. Optionally, the number of target line drawings can be one frame or multiple frames. Similarly, the number of reference line drawings can be one frame or multiple frames, which is not limited here.

[0059] Specifically, the computer device can obtain a line drawing sequence corresponding to the target object, and determine the target line drawing to be colored and the reference line drawing corresponding to the reference color drawing from the line drawing sequence. The way for the computer device to obtain the line drawing sequence is not unique. Optionally, the user can directly draw lines based on the computer device to obtain multiple frames of line drawings to form a line drawing sequence; optionally, the line drawing provider can directly upload the line drawing sequence to the computer device; optionally, the computer device can also obtain the line drawing sequence from the server, wherein the line drawing sequence in the server can be uploaded to the server by the line drawing provider through the terminal held by the line drawing provider. The line drawing provider can be a draftsman or other person who has the right to use the line drawing sequence.

[0060] In an optional embodiment, the computer device may obtain a sequence of line drawings corresponding to the target object, select a reference line drawing with complex lines from the sequence of line drawings, and prompt the user to manually color the reference line drawing, thereby obtaining a reference color drawing corresponding to the reference line drawing. Optionally, the computer device may determine the line complexity of the line drawing based on the number and line density of the line drawings, where the greater the number of lines and the greater the line density, the more complex the lines of the line drawing.

[0061] In an optional embodiment, the computer device can obtain a line drawing sequence and a reference color drawing corresponding to the target object, and then, through image matching, determine the reference line drawing corresponding to the reference color drawing from the line drawing sequence, and determine the line drawings in the line drawing sequence other than the reference line drawing as the target line drawing to be colored. Optionally, the computer device can also determine the line drawing corresponding to the color drawing as a candidate line drawing, and when the candidate line drawing matches the target line drawing, determine the candidate line drawing as the reference line drawing, and determine the color drawing corresponding to the candidate line drawing as the reference color drawing as a reference for coloring the target line drawing. Wherein, the candidate line drawing matches the target line drawing, which can mean that the time interval between the candidate line drawing and the target line drawing in the line drawing sequence is less than or equal to the interval threshold, or it can mean that the posture similarity between the first object posture of the target object in the target line drawing and the second object posture of the target object in the candidate line drawing meets the similarity condition.

[0062] Step S204 , using the reference color draft image as a reference, based on the time information of the target line draft image and the reference line draft image in the line draft image sequence, coloring the target line draft image to obtain an initial color draft image corresponding to the target line draft image.

[0063] Among them, the time information of the line drawing in the line drawing sequence corresponds to the order of the line drawing in the line drawing sequence. Optionally, the computer device can use the arrangement order of the line drawing in the line drawing sequence as the time information corresponding to the line drawing in the line drawing sequence. For example, the time information corresponding to the first frame of the line drawing in the line drawing sequence is 1, and the time information corresponding to the second frame of the line drawing is 2. Optionally, the computer device can also determine the playback time corresponding to each line drawing in the line drawing sequence according to the playback time axis set for the line drawing sequence, and determine the playback time as the time information corresponding to the line drawing in the line drawing sequence. The playback time can be the playback start time or the playback end time, which is not limited here. Specifically, the computer device can use the reference color draft image as a reference, and based on the time information of the target line drawing and the reference line drawing in the line drawing sequence, color the target line drawing to obtain the initial color draft image corresponding to the target line drawing. For example, as Figure 3 As shown, the reference color draft image 303 and the target line draft image 301 depict the same target object. The computer device can color the target line draft image 301 with reference to the reference color draft image 303 to obtain the initial color draft image 302 corresponding to the target line draft image 301.

[0064] In an optional embodiment, the computer device may construct guidance information including a reference color draft image, and temporal information of the target line draft image and the reference line draft image in a line draft image sequence, and under the guidance of the guidance information, use a trained neural network model to color the target line draft image to obtain an initial color draft image corresponding to the target line draft image. Optionally, the computer device may extract multi-scale line draft texture features based on the target line draft image, and obtain multi-scale noise features extracted from a preset noise image. Then, for each scale in the multi-scale, under the guidance of the guidance information, the noise features at that scale are encoded with reference to the line draft texture features at that scale to obtain multi-scale image features. Finally, the multi-scale image features are multi-scale decoded to obtain an initial color draft image corresponding to the target line draft image.

[0065] In an optional embodiment, the computer device can analyze the changes of the target object in the target line drawing relative to the reference line drawing based on the time information corresponding to the target line drawing and the reference line drawing in the line drawing sequence, and based on the changes, determine the image correspondence between the reference line drawing and the target line drawing, and then based on the image correspondence, color the target line drawing to obtain an initial color draft image corresponding to the target line drawing.

[0066] In an optional embodiment, when a reference color draft image matches multiple frames of target line draft images, the computer device can use the reference color draft image as a reference, and based on the time information corresponding to the multiple frames of target line draft images and the reference line draft images in the line draft image sequence, batch colorize the multiple frames of target line draft images to obtain multiple frames of initial color draft images corresponding to the multiple frames of target line draft images. Among them, the multi-time information can be incorporated into the batch coloring process through the temporal attention mechanism. Optionally, the computer device can generate corresponding guidance information based on the reference color draft image as a reference for the coloring process; it can also use the image correspondence between the reference line draft image and the target line draft image as a reference for the coloring process.

[0067] Step S206 : determining a first area included in the corresponding initial color draft image based on the closed area in the target line draft image, and determining a second area included in the corresponding reference color draft image based on the closed area in the reference line draft image.

[0068] The closed area in the line drawing refers to a closed shape enclosed by lines or paths, which serves as the basis for coloring and filling. The boundary of the closed area is continuous, without gaps or disconnected parts. Optionally, the closed area can be a simple geometric shape, such as a circle or square, or a complex polygon or curve combination; the closed area can correspond to the component structure of the target object, such as the target object's face or hair, or it can be a part of the component structure of the target object, such as the highlight area or shadow area in the target object's hair.

[0069] Specifically, the computer device determines the closed areas contained in the target line drawing and the reference line drawing respectively, and then, based on the closed areas in the target line drawing, determines the first area contained in the initial color draft drawing corresponding to the target line drawing, and based on the closed areas in the reference line drawing, determines the second area contained in the reference color draft drawing corresponding to the reference line drawing.

[0070] In an optional embodiment, the computer device may use a pre-trained semantic segmentation network to segment the target line drawing and the reference line drawing into closed regions corresponding to different semantics. Based on the positional mapping relationship between the color drawing and the line drawing, the computer device may determine, from the initial color drawing, a first region corresponding to the closed region of the target line drawing, and from the reference color drawing, a second region corresponding to the closed region of the reference line drawing. For example, in the case where the target object is an anime character, the segmented closed regions may include hair, facial, and eye regions, among others.

[0071] In an optional embodiment, the process of determining the first area and the second area includes: based on the color lines contained in the line drawing, determining a closed area enclosed by the color lines from the line drawing; based on the mapping position of the closed area in the color draft image corresponding to the line drawing, determining the colored area contained in the color draft image.

[0072] The line drawing includes a target line drawing and a reference line drawing; the coloring area includes a first area in the initial color drawing corresponding to the target line drawing, and a second area in the reference color drawing corresponding to the reference line drawing. Color lines are lines with color properties used in line drawings. In addition to defining the outline of the objects drawn in the line drawing, these lines can also be used to express the light and shadow effects, layering, and details of the picture. The color lines in the line drawing can be used as a reference for coloring the line drawing. During the painting process, lines of various colors such as red, blue, and yellow can be used to mark different areas to be colored. This way, when coloring the line drawing, you can more clearly know what color to use for each part. For example, the area enclosed by the red line needs to be filled with a highlight color, the area enclosed by the blue line needs to be filled with a primary shadow color, and the area enclosed by the green line needs to be filled with a secondary shadow color. If the lines do not have a special color, the enclosed area is filled with a normal color, that is, the inherent color.

[0073] Specifically, the computer device may determine, from the target line drawing image, a closed region enclosed by the color lines based on the color lines included in the target line drawing image, and further determine, based on the mapping position of the closed region in the initial color drawing image, a first region included in the initial color drawing image. Similarly, the computer device may determine, from the reference line drawing image, a closed region enclosed by the color lines based on the color lines included in the reference line drawing image, and further determine, based on the mapping position of the closed region in the reference color drawing image, a second region included in the reference color drawing image.

[0074] In one possible implementation, a computer device can use a flood fill algorithm to segment a line drawing, breaking it into multiple closed regions to be colored. The flood fill algorithm is an algorithm that segments an image by filling a region. Its core concept is to start with a starting point (seed point) and fill that point and its adjacent pixels according to a specific rule until the entire region is processed. In image segmentation, the flood fill algorithm can be used to identify independent regions with similar features, thereby achieving image segmentation. In a line drawing, different closed regions depict different content. The flood fill algorithm can segment these regions, resulting in multiple closed regions.

[0075] In the above embodiment, a closed area enclosed by color lines is determined from the line drawing, and the colored area in the color draft drawing is determined based on the mapping position of the closed area in the corresponding color draft drawing. This enables fine-grained area matching in the color draft drawing, so that fine-grained color correction can be performed based on the area matching results subsequently, which is conducive to further improving the coloring accuracy.

[0076] Step S208, determining a target first area and a target second area having a matching relationship from the first area and the second area, performing color correction on the target first area based on the area image of the target second area, and obtaining a target color draft image corresponding to the target line draft image.

[0077] The target second area may refer to a portion of a second area that matches the first area, that is, a sub-area in the second area; or it may refer to a second area that matches the first area in multiple second areas, that is, a complete second area. Similarly, the target first area may refer to a first area that matches the target second area in multiple first areas, that is, a complete first area; or it may refer to a portion of a first area that matches the target second area, that is, a sub-area in the first area. That is to say, when the first and second areas are divided, the local images in the area may be matched, and color correction may be performed on the matched local image areas, or the entire area may be matched, and color correction may be performed on the successfully matched areas.

[0078] Specifically, the computer device can determine a matching relationship between the second region and the first region, determine a target first region and a target second region that have a matching relationship from the first and second regions, and perform color correction on the target first region based on the region image of the target second region. When the computer device traverses all first regions in the initial color draft image, a target color draft image corresponding to the target line draft image can be obtained.

[0079] Optionally, the target second area corresponds to a complete second area, and the target first area corresponds to a complete first area. For each first area in the initial line draft image, if there is a target second area that matches the first area in the second area included in the reference color draft image, the computer device can perform color correction on the first area based on the area image of the target second area. Optionally, if the target second area that matches the first area does not exist in the second area included in the reference color draft image, it is not necessary to perform color correction on the first area. When the computer device traverses all the first areas in the initial color draft image, the target color draft image corresponding to the target line draft image can be obtained.

[0080] Optionally, the first area includes at least one sub-area, and the second area includes at least one sub-area. For each sub-area in the first area, in the case of a target second area matching the sub-area in each sub-area of ​​the second area, it is determined that the sub-area in the first area and the target second area have a matching relationship, and the target first area and the target second area with a matching relationship are obtained, and then the target first area can be subsequently color corrected based on the regional image of the target second area. The sub-areas can be obtained through semantic analysis or shape division. Taking anime characters as an example, through semantic analysis, the mouth area can be further divided into sub-areas such as the upper lip area, the lower lip area, and the teeth area; the eye area can be further divided into sub-areas such as the eyelid area and the eyeball area.

[0081] In an optional embodiment, determining a target first area and a target second area having a matching relationship from the first area and the second area includes: determining an image correspondence relationship between the first area and the second area based on image information respectively contained in the first area and the second area; determining a target second area that matches at least a portion of the image content of the first area from the second area based on the image correspondence relationship; and determining a target first area corresponding to the at least a portion of the image content from the first area.

[0082] Specifically, the computer device can determine an image correspondence between the first and second regions based on the image information contained in each of the first and second regions, and based on the image correspondence, determine a target second region from the second region that matches at least a portion of the image content of the first region, and determine a target first region from the first region to which the image content matching the target second region belongs. Furthermore, the computer device can perform color correction on the image content of the target first region based on the regional image of the target second region to obtain a target color draft image corresponding to the target line draft image.

[0083] The image correspondence can be determined using at least one of semantic matching and shape matching. For example, in the case of a cartoon character, if both the first and second regions are mouth regions, the content depicted in the first and second regions may differ due to the different poses of the target character's mouth in different line drawings. For example, the first region includes the upper lip region, the lower lip region, and the teeth region; the second region only includes the upper lip region and the lower lip region. Based on this, the computer device can determine, based on the image correspondence, the upper lip regions contained in the first and second regions as a first set of matching target first and second regions, and the lower lip regions contained in the first and second regions as a second set of matching target first and second regions. Then, based on the regional image of the upper lip region in the second region, color correction is performed on the upper lip region in the first region, and based on the regional image of the lower lip region in the second region, color correction is performed on the lower lip region in the first region. However, since there is no matching target second region in the second region, color correction is not performed on the teeth region in the first region. After the above processing, a color draft image of the corrected lip region in the target line drawing is obtained. The target first area and the target second area whose image content matches are determined through the image correspondence relationship, so that the target first area can be subsequently color-corrected based on the target second area whose image content matches, which can ensure the accuracy of the color correction results and is conducive to further improving the coloring accuracy of the line drawing.

[0084] The above-mentioned line drawing coloring processing method obtains a target line drawing to be colored and a reference line drawing corresponding to a reference color drawing from a line drawing sequence corresponding to a target object. Then, using the reference color drawing as a reference, the target line drawing in the line drawing sequence is colored based on the temporal information of the target line drawing and the reference line drawing in the line drawing sequence, thereby obtaining an initial color drawing corresponding to the target line drawing. Because the coloring process refers to the reference color drawing corresponding to the reference line drawing in the line drawing sequence of the same target object, the semantic consistency and detail correspondence between the reference color drawing and the target line drawing can be ensured, so that the initial color drawing and the reference color drawing can maintain a high degree of consistency in color and style, which is conducive to improving coloring accuracy. In addition, the coloring process also takes into account the temporal information of the target line drawing and the reference line drawing in the line drawing sequence, which can better understand the semantic relationship between the line drawings, thereby more accurately transferring the color information in the reference color drawing to the target line drawing, which is conducive to further improving coloring accuracy. On this basis, the first area included in the corresponding initial color draft image is determined based on the closed area in the target line draft image, and the second area included in the corresponding reference color draft image is determined based on the closed area in the reference line draft image. Then, the target first area and the target second area with a matching relationship are determined from the first area and the second area. Based on the area image of the target second area, the target first area is color corrected to obtain the target color draft image corresponding to the target line draft image. On the basis of the initial color draft image, based on the area matching relationship between the reference color draft image and the initial color draft image, the target first area in the initial color draft image that matches the target second area can be color corrected, which can further ensure the consistency in color and style, and is conducive to further improving the coloring accuracy.

[0085] In actual applications, in order to ensure the accuracy of coloring, the computer equipment can determine the matching between the target line drawing to be colored and the reference line drawing that has been completed before coloring, and when the two match, determine the coloring reference relationship between the target line drawing and the reference color drawing corresponding to the reference line drawing.

[0086] In one embodiment, a target line drawing to be colored and a reference line drawing corresponding to a reference color drawing are obtained from a line drawing sequence corresponding to a target object, including: obtaining a target line drawing to be colored and a candidate line drawing corresponding to a colored drawing from a line drawing sequence corresponding to a target object; when the candidate line drawing matches the target line drawing, determining the candidate line drawing as a reference line drawing, and determining the color drawing corresponding to the candidate line drawing as a reference color drawing.

[0087] The candidate line drawing is the line drawing in the line drawing sequence that corresponds to the colored drawing, and the target line drawing is the line drawing in the line drawing sequence that needs to be colored to obtain the corresponding colored drawing. The colored drawing can be obtained by manual coloring and can reflect the actual coloring requirements.

[0088] Specifically, the computer device can obtain a line drawing sequence corresponding to the target object, and a color drawing obtained by coloring at least one line drawing frame contained in the line drawing sequence. Then, the computer device can determine, from the line drawing sequence, a candidate line drawing corresponding to the color drawing, and a target line drawing to be colored. Next, the computer device can determine a matching relationship between the candidate line drawing and the target line drawing, and if the two match, determine the candidate line drawing as a reference line drawing, and determine the color drawing corresponding to the candidate line drawing as a reference color drawing, as a reference for subsequent coloring of the target line drawing.

[0089] In an optional embodiment, the number of target line drawings is multiple frames, and the number of candidate color drawings is at least two frames. In this embodiment, the computer device can determine, for each target line drawing, a degree of match between the target line drawing and each candidate line drawing, and based on the degree of match, determine a reference color drawing from each candidate line drawing that matches the target line drawing.

[0090] In the above embodiment, when the candidate line draft image corresponding to the colored draft image matches the target line draft image, the candidate line draft image is determined as the reference line draft image, and the color draft image corresponding to the candidate line draft image is determined as the reference color draft image, which serves as a reference for subsequent coloring of the target line draft image. This can ensure the compatibility between the reference color draft image and the target line draft image, which is conducive to further improving the coloring accuracy.

[0091] There is no unique way to determine the matching relationship between the candidate line drawing and the target line drawing.

[0092] In an optional embodiment, when the line drawing sequence includes multiple candidate line drawings, the computer device can use the candidate line drawings as a dividing point to divide the target line drawings to be colored into multiple target line drawing groups, and use any candidate line drawing adjacent to the target line drawing as a reference line drawing for each target line drawing included in the target line drawing group. Figure 4 As shown, the line drawing sequence 401 includes 8 frames of line drawings, and the postures of the target object in different line drawings are different. In addition, when the 1st frame line drawing and the 6th frame line drawing are candidate line drawings corresponding to the colored drawings, the computer device can use the 1st frame candidate line drawing as the reference line drawing for the 2nd to 5th frame target line drawings, and use the 6th frame candidate line drawing as the reference line drawing for the 7th to 8th frame target line drawings. In other words, it can be divided into two target line drawing groups for coloring respectively, as shown in FIG. Figure 4 As shown, the initial color draft and the reference color draft Figure 1 Together, the color draft image sequence 402 corresponding to the line draft image sequence 401 is formed.

[0093] In an optional embodiment, the line drawing coloring processing method also includes: determining a first time corresponding to the target line drawing in the line drawing sequence, and a second time corresponding to the candidate line drawing in the line drawing sequence; based on the time interval between the second time and the first time, determining the matching degree between the candidate line drawing and the target line drawing.

[0094] The degree of match between the candidate line drawing and the target line drawing is inversely correlated with the time interval. Anticorrelation between two variables means that, with all other conditions remaining unchanged, the two variables change in opposite directions: when one variable changes from large to small, the other variable changes from small to large. It should be understood that anticorrelation here refers to the opposite directions of change, but does not require that a slight change in one variable necessarily also changes the other. The time corresponding to a line drawing in the line drawing sequence corresponds to the order of the line drawing in the line drawing sequence. Optionally, the computer device can use the order of the line drawing in the line drawing sequence as the time corresponding to the line drawing in the line drawing sequence. For example, the first line drawing in the line drawing sequence corresponds to time 1, and the second line drawing corresponds to time 2. Optionally, the computer device can also determine the playback time corresponding to each line drawing in the line drawing sequence based on the playback timeline set for the line drawing sequence, and determine the playback time as the time corresponding to the line drawing in the line drawing sequence. This playback time can be the playback start time or the playback end time, without limitation.

[0095] In actual applications, a line drawing sequence usually contains dozens of lines. These line drawings describe the changes in the movement of the target object in the time period from the first frame to the last frame. The target object has a relatively small movement amplitude between different line drawings that are closer, but a relatively large movement amplitude between different line drawings that are farther away. If a line drawing that is farther away is selected as a coloring reference, the accuracy of the coloring result may be affected due to the large movement amplitude. Based on this, the computer device can determine the first time corresponding to the target line drawing in the line drawing sequence and the second time corresponding to the candidate line drawing in the line drawing sequence according to the arrangement order of each line drawing in the line drawing sequence. Then, the time interval between the second time and the first time is calculated, and the matching degree that is inversely correlated with the time interval is further determined. Optionally, the computer device can determine that the candidate line drawing matches the target line drawing when the matching degree between the candidate line drawing and the target line drawing is greater than or equal to the matching degree threshold.

[0096] In one possible implementation, the computer device may determine that the candidate line drawing matches the target line drawing if the time interval between the second time and the first time is less than or equal to the interval threshold. Optionally, for target line drawings whose time interval with any candidate line drawing is greater than the interval threshold, the computer device may not color them in the current round. Instead, after completing the coloring process for a portion of the target line drawings, the computer device may use these target line drawings as new candidate line drawings for the next round of time interval determination, until a reference line drawing with a time interval less than or equal to the interval threshold is selected.

[0097] For example, when the line drawing sequence includes 20 frames of line drawings, and the 1st and 10th frame line drawings are candidate line drawings for corresponding colored drawings, the computer device can use the 1st frame candidate line drawing as a reference line drawing for the 2nd to 5th frame target line drawings, and use the 10th frame candidate line drawing as a reference line drawing for the 6th to 9th and 11th to 14th frame target line drawings. As for the 15th to 20th frame target line drawings, since they are farther away from the candidate line drawings, they can be not colored in the current round. Instead, when the 14th frame target line drawing has been colored, the target colored drawings corresponding to the target line drawing are used as a reference to color the 15th to 19th frame target line drawings. Similarly, through multiple rounds of coloring, the coloring process for the entire line drawing sequence can be completed, and a colored drawing sequence corresponding to the line drawing sequence can be obtained.

[0098] like Figure 4 In the embodiment, when the line drawing sequence 401 includes 8 frames of line drawing, and the 1st frame of line drawing and the 6th frame of line drawing are candidate line drawing images for the corresponding colored drawings, the computer device can use the 1st frame of candidate line drawing image as a reference line drawing image for the 2nd-3rd frame of target line drawing images, and use the 6th frame of candidate line drawing image as a reference line drawing image for the 4th-5th and 7th-8th frame of target line drawing images.

[0099] In the above embodiment, based on the times corresponding to the target line draft image and the candidate line draft image in the line draft image sequence, the time interval between the target line draft image and the candidate line draft image in the line draft image sequence is determined, and the matching degree inversely correlated with the time interval is further determined. This can ensure that the time interval between the reference color draft image used as a coloring reference and the target line draft image is small, and further ensure that the movement amplitude between the two is relatively small, so that the color information in the reference color draft image can be accurately transferred to the target line draft image, which is conducive to further improving the coloring accuracy.

[0100] In an optional embodiment, the line drawing coloring processing method also includes: determining a first object posture of the target object in the target line drawing, and a second object posture of the target object in the candidate line drawing; when the posture similarity between the second object posture and the first object posture meets the similarity condition, determining that the candidate line drawing matches the target line drawing.

[0101] The object pose refers to the physical state of the target object in a specific scene or action. The object pose can be represented by a series of pose data. This pose data can include, for example, pose data corresponding to multiple pose parameters, such as Euler angles, homogeneous transformation matrices, and direction cosine matrices. The pose data can also include position data corresponding to multiple key points.

[0102] In one possible implementation, the object's pose can also be represented using a graph structure. The nodes in this graph structure represent different pose elements, and the lines between the nodes represent the relationships between these pose elements. For example, in the case of an animated character, pose elements may include joints, hands, and head. By connecting these pose elements, a skeleton graph representing the object's pose can be generated.

[0103] Specifically, the computer device can determine the object pose of the target object depicted in the line drawing by performing object pose recognition on the target object depicted in the line drawing. Optionally, the computer device can extract key points of the target object from the line drawing to obtain the object pose represented by the coordinates of these key points. Optionally, the computer device can input the line drawing depicting the target object into a trained pose recognition network based on computer vision and deep learning technologies to obtain the object pose of the target object in the line drawing. The pose recognition network can be a depthwise separable convolution network (DSC), a convolutional neural network (CNN), or the like.

[0104] When the posture of the first object and the posture of the second object are determined, the computer device can further calculate the posture similarity between the posture of the second object and the posture of the first object, and determine that the candidate line drawing matches the target line drawing when the posture similarity meets the similarity condition. Here, the posture similarity meets the similarity condition, which means that the posture similarity is greater than or equal to the similarity threshold. The posture similarity can be characterized by cosine similarity, Euclidean distance or Mahalanobis distance. Take the case where the object posture is represented by the coordinates of key points as an example. Optionally, the computer device can use the coordinates of each key point as a vector to obtain a posture matrix containing each vector, and by calculating the similarity between different posture matrices, obtain the posture similarity between the object postures represented by each posture matrix.

[0105] Exemplarily, when the line drawing sequence includes 20 frames of line drawings, and the 1st and 10th frames of line drawings are candidate line drawings for corresponding colored drawings, the computer device can calculate the posture similarity between each target line drawing to be colored and each candidate line drawing, and filter out reference line drawings whose posture similarity with the target line drawing meets the similarity condition from each candidate line drawing. Optionally, for target line drawings whose posture similarity with any candidate line drawing does not meet the similarity condition, the computer device may not color them in the current round, but after completing the coloring process for some target line drawings, use these target line drawings as new candidate line drawings and perform the next round of posture similarity calculation until a reference line drawing whose posture similarity meets the similarity condition can be filtered out.

[0106] In the above embodiment, the first object posture of the target object in the target line draft image and the second object posture of the target object in the candidate line draft image are determined. When the posture similarity between the second object posture and the first object posture meets the similarity condition, the candidate line draft image is determined to match the target line draft image. This can ensure that the object postures between the reference color draft image used as a coloring reference and the target line draft image are relatively similar, and further ensure that the color information in the reference color draft image can be accurately transferred to the target line draft image, which is conducive to further improving the coloring accuracy.

[0107] In an optional embodiment, the line drawing coloring processing method also includes: determining a first time corresponding to the target line drawing in the line drawing sequence, and a second time corresponding to the candidate line drawing in the line drawing sequence; when the time interval between the second time and the first time is less than or equal to the interval threshold, determining a first object posture of the target object in the target line drawing, and a second object posture of the target object in the candidate line drawing; if the posture similarity between the second object posture and the first object posture meets the similarity condition, determining that the candidate line drawing matches the target line drawing.

[0108] As mentioned above, close time intervals usually mean smaller movement amplitudes. Therefore, matching the object postures when the time intervals are close can reduce the amount of calculation in the posture matching process, while ensuring that the object postures between the reference color draft image and the target line draft image used as a coloring reference are relatively similar, thereby ensuring that the color information in the reference color draft image can be accurately transferred to the target line draft image, which can improve coloring accuracy while ensuring work efficiency.

[0109] In one embodiment, with a reference color draft image as a reference, the target line draft image is colored based on the time information of the target line draft image and the reference line draft image in the line draft image sequence to obtain an initial color draft image corresponding to the target line draft image, including: extracting features of the reference color draft image to obtain reference color draft features of the reference color draft image; extracting features of the target line draft image to obtain target line draft features of the target line draft image; determining target fusion features corresponding to the target line draft image based on the target line draft features and the reference color draft features; integrating the time information of the target line draft image and the reference line draft image in the line draft image sequence into the target fusion features to obtain time enhancement features corresponding to the target line draft image; decoding the time enhancement features to obtain the initial color draft image corresponding to the target line draft image.

[0110] Reference color draft features refer to feature information that characterizes the characteristics of the reference color draft image. These reference color draft features may include color features, shape features, texture features, etc. Target line draft features refer to feature information that characterizes the characteristics of the target line draft image. These target line draft features may include geometric features, texture features, etc. The data format of reference color draft features and target line draft features can be either a matrix or a vector. Temporal enhancement features are enhanced features that incorporate temporal information.

[0111] Specifically, the computer device can perform feature extraction on the reference color draft image to obtain the reference color draft features of the reference color draft image; and perform feature extraction on the target line draft image to obtain the target line draft features of the target line draft image.

[0112] There are many ways to extract features from a target line drawing. Alternatively, the computer device can perform edge detection and geometric feature analysis on the target line drawing to obtain target line drawing features. Alternatively, the computer device can extract features from the target line drawing using texture feature extraction and a generative adversarial network model to obtain target line drawing features. Alternatively, the computer device can extract features from the target line drawing using multi-scale feature extraction and an attention mechanism to obtain target line drawing features. In one possible implementation, the computer device can input the target line drawing into a line drawing extraction network (Lineart Guider) to obtain target line drawing features.

[0113] There is no unique way to extract features from the reference color draft image. Optionally, the computer device can extract the color features of the reference color draft image through a color histogram, a color matrix, or a color aggregation vector, extract the texture features of the reference color draft image through a grayscale co-occurrence matrix, a local binary pattern, a wavelet transform, etc., extract the shape features of the reference color draft image through edge detection, contour extraction, etc., and superimpose and fuse the above features to obtain the reference color draft features of the reference color draft image. The computer device can also extract the features of the reference color draft image at different scales through convolution operations at different scales, and superimpose and fuse these features to obtain the reference color draft features of the reference color draft image.

[0114] In one possible implementation, the computer device may extract features from the reference color draft image using an encoder and a feature extraction network (Reference U-Net) to obtain reference color draft features of the reference color draft image. The encoder may be a variational autoencoder (VAC).

[0115] After obtaining the reference color draft features of the reference color draft image and the target line draft features of the target line draft image, the computer device may determine a target fusion feature corresponding to the target line draft image based on the target line draft features and the reference color draft features. The computer device may then integrate temporal information of the target line draft image and the reference line draft image in the line draft image sequence into the target fusion feature to obtain a temporal enhancement feature corresponding to the target line draft image. The computer device may then decode the temporal enhancement feature to obtain an initial color draft image corresponding to the target line draft image.

[0116] Optionally, the computer device may fuse the reference color draft features with the target line draft features to obtain target fusion features corresponding to the target line draft image.

[0117] Optionally, the computer device can use a temporal attention mechanism to integrate temporal information of the target line drawing and the reference line drawing in the line drawing sequence into the target fusion feature to obtain a temporal enhancement feature corresponding to the target line drawing. The temporal attention mechanism calculates the correlation between different temporal information and assigns a weight to the feature corresponding to each temporal information, thereby highlighting the temporal information that is more important to the current task.

[0118] Optionally, the computer device can determine the time features corresponding to the target line drawing and the reference line drawing respectively based on the time information of the target line drawing and the reference line drawing in the line drawing sequence, and obtain the time enhancement features corresponding to the target line drawing by fusing the time features and the target fusion features.

[0119] Optionally, methods for fusing two features include concatenation, feature operation, etc.

[0120] In the above embodiment, based on the target line draft features of the target line draft image and the reference color draft features of the reference color draft image, the target fusion features corresponding to the target line draft image are determined, and then the time information of the target line draft image and the reference line draft image in the line draft image sequence are integrated into the target fusion features to obtain the time enhancement features corresponding to the target line draft image, and the time enhancement features are decoded to obtain the initial color draft image corresponding to the target line draft image. Since the time information is integrated into the processing process, the semantic relationship between the line draft images can be better understood, thereby more accurately transferring the color information in the reference color draft image to the target line draft image, which is conducive to further improving the coloring accuracy.

[0121] In one embodiment, the method for coloring a line drawing further includes: performing semantic analysis on a reference color drawing to obtain reference semantic features of the reference color drawing. In this embodiment, a target fusion feature corresponding to the target line drawing is determined based on the target line drawing features and the reference color drawing features, including: obtaining an initial fusion feature corresponding to the target line drawing based on the splicing result of the target line drawing features and the reference color drawing features; and integrating the semantic information represented by the reference semantic features into the initial fusion feature to obtain a target fusion feature corresponding to the target line drawing.

[0122] The reference semantic features are feature information that can characterize the semantic characteristics of the reference color draft image. These semantic characteristics can be represented by the content, structure, and visual information depicted in the reference color draft image. Specifically, the computer device can perform semantic analysis on the reference color draft image to obtain the reference semantic features of the reference color draft image.

[0123] Optionally, the computer device can divide the line drawing into different semantic areas through a pre-trained semantic segmentation network, thereby extracting the semantic features of each area, and fusing the features into the reference semantic features of the reference color draft image. Optionally, the computer device can extract reference semantic features that represent semantic characteristics from the reference color draft image through a deep learning model. The deep learning model may include, for example, a convolutional neural network, a Transformer, etc. In one possible implementation, the computer device can perform semantic analysis on the reference color draft image by using a multimodal pre-training module (Contrastive Language-Image Pre-training, CLIP) to obtain the reference semantic features of the reference color draft image.

[0124] Optionally, the computer device can obtain the initial fusion feature corresponding to the target line draft image based on the splicing result of the target line draft feature and the reference color draft feature, and then integrate the semantic information represented by the reference semantic feature into the initial fusion feature to obtain the target fusion feature corresponding to the target line draft image.

[0125] Optionally, the semantic information represented by the reference semantic features can be integrated into the initial fused features based on the cross-attention mechanism. Exemplarily, the weight matrix Q (Query) of the cross-attention mechanism can come from the initial fused features, and the weight matrix K (Key) and weight matrix V (Value) of the cross-attention mechanism can come from the reference semantic features. Specifically, the cross-attention mechanism allows the model to extract information from two different feature spaces. Specifically, the Q matrix usually comes from the target feature space, while the K matrix and the V matrix come from the source feature space. This mechanism enables the model to extract information from one feature space and interact it with the information in another feature space, thereby enhancing the expressive power of the features. Optionally, the initial fused features and the reference semantic features can be respectively passed through a learnable parameter matrix W Q 、W K 、W V Perform linear transformation to obtain query matrix, key matrix and value matrix.

[0126] In the above embodiment, the initial fusion features corresponding to the target line draft image are first obtained based on the splicing results of the target line draft features and the reference color draft features, and then the semantic information of the reference few features is integrated into the initial fusion features to obtain the target fusion features. This can ensure that the target fusion features have strong feature expression capabilities, which is conducive to further improving the coloring accuracy.

[0127] In an optional embodiment, based on the splicing results of the target line draft features and the reference color draft features, the initial fusion features corresponding to the target line draft image are obtained, including: splicing the target line draft features and the reference color draft features to obtain the splicing features corresponding to the target line draft image; and performing feature enhancement on the splicing features in the spatial dimension to obtain the initial fusion features corresponding to the target line draft image.

[0128] Specifically, the computer device can obtain the splicing features corresponding to the target line draft image by splicing the target line draft features and the reference color draft features, and then perform feature enhancement on the splicing features in the spatial dimension to obtain the initial fusion features corresponding to the target line draft image. Optionally, the computer device can utilize the spatial attention mechanism to perform feature enhancement on the splicing features in the spatial dimension to obtain the initial fusion features corresponding to the target line draft image. The spatial attention mechanism can capture the relationship between different positions in the feature map, thereby improving the expressive power of the features. Optionally, the channel attention module and the spatial attention module can be connected in parallel to perform attention calculations on the splicing features in the channel and spatial dimensions respectively, and then the color output results of the two modules can be fused to form the initial fusion features corresponding to the target line draft image. The multi-dimensional feature attention mechanism can capture important information in different dimensions, thereby further enhancing the representation power of the initial fusion features.

[0129] In the above embodiment, the target line draft features and the reference color draft features are first spliced ​​together to obtain the splicing features corresponding to the target line draft image, and then the splicing features are enhanced in the spatial dimension to obtain the initial fusion features corresponding to the target line draft image. This can further enhance the representation ability of the initial fusion features, thereby further improving the coloring accuracy.

[0130] In one embodiment, extracting features from a target line drawing to obtain target line drawing features of the target line drawing includes: extracting features from the target line drawing to obtain initial line drawing features of the target line drawing; and superimposing noise on the initial line drawing features to obtain target line drawing features corresponding to the target line drawing. In this embodiment, decoding the temporal enhancement features to obtain an initial color draft image corresponding to the target line drawing includes: performing denoising and decoding on the temporal enhancement features to obtain the initial color draft image corresponding to the target line drawing.

[0131] In practical applications, the computer device can apply noise addition and denoising to the coloring process to improve the clarity and robustness of the initial color draft image. Specifically, the computer device can extract features from the target line drawing to obtain initial line drawing features of the target line drawing. Then, noise is superimposed on the initial line drawing features to obtain target line drawing features corresponding to the target line drawing. Subsequently, the computer device can perform denoising and decoding on the temporal enhancement features to obtain the initial color draft image corresponding to the target line drawing.

[0132] Optionally, the computer device may superimpose randomly generated Gaussian noise on the initial line drawing features, and generate target line drawing features corresponding to the target line drawing image. The target line drawing features are essentially line drawing features with noise.

[0133] Optionally, the computer device may gradually add noise to the initial line drawing features over multiple time steps to obtain target line drawing features with noise. After a series of processing on the target line drawing features, a time-enhanced feature with noise may be obtained. Finally, the computer device may gradually remove the noise from the time-enhanced feature over multiple time steps, encode the feature, and restore the feature to an image, thereby obtaining an initial color draft image corresponding to the target line drawing image.

[0134] Optionally, during training, the computer device can optimize the loss function by maximizing the difference between two noises. Specifically, the computer device can randomly sample images from the dataset and add noise at each time step to generate noisy images. The noise in these noisy images is then predicted and the prediction error is calculated to minimize the loss function. During inference, the computer device can sample noise from a standard normal distribution and then gradually remove the noise through a denoising process to ultimately generate an image. This process typically starts with pure noise, denoises it over multiple time steps, and then uses the noise predicted by the model to restore the image.

[0135] In the above embodiment, applying noise addition and denoising to the process of generating the initial color draft image can improve the clarity of the quality of the initial color draft image while enhancing the robustness.

[0136] In one embodiment, the time information of the target line drawing and the reference line drawing in the line drawing sequence is integrated into the target fusion feature to obtain the time enhancement feature corresponding to the target line drawing, including: determining the weight matrix corresponding to the target line drawing based on the target line drawing feature and the reference line drawing feature corresponding to the reference line drawing; using the weight matrix as a parameter, the target fusion feature is enhanced based on the temporal attention mechanism to obtain the time enhancement feature corresponding to the target line drawing.

[0137] Among them, the weight matrix is ​​the core parameter of the attention mechanism, including the query matrix, key matrix and value matrix. Specifically, the computer device can determine the weight matrix corresponding to the target line drawing based on the target line drawing features, and then, using the weight matrix as a parameter, perform feature enhancement processing on the target fusion features based on the temporal attention mechanism to obtain the time enhancement features corresponding to the target line drawing. Optionally, the target line drawing features and the reference line drawing features corresponding to the reference line drawing can be fused to obtain the fused line drawing features, and then the fused line drawing features can be obtained through the learnable parameter matrix W. Q 、W K 、W V Perform linear transformation to obtain query matrix, key matrix and value matrix.

[0138] In the above embodiment, feature enhancement processing is performed on the target fusion features based on the temporal attention mechanism, and the weight matrix of the temporal attention mechanism is derived from the target line draft features. The temporal information of the target line draft image and the reference line draft image can be integrated into the target fusion features through the temporal attention mechanism, thereby enhancing the feature representation capability, which is conducive to further improving the coloring accuracy.

[0139] In one embodiment, the number of target line drawings is multiple frames, and the number of reference color drawings is at least two frames. In this embodiment, the target line drawings are colored based on the time information of the target line drawings and the reference line drawings in the line drawing sequence, thereby obtaining an initial color drawing corresponding to the target line drawings, including: for each frame of the target line drawings, determining a selected color drawing that matches the target line drawings from each reference line drawing; if the selected color drawing matches multiple frames of the target line drawings, using the selected color drawing as a reference, based on the time information corresponding to the multiple frames of the target line drawings and the selected line drawings in the line drawing sequence, coloring the multiple frames of the target line drawings, thereby obtaining multiple initial color drawings corresponding to the multiple frames of the target line drawings.

[0140] Specifically, when the number of target line drawings and reference color drawings are both multiple frames, the computer device can determine, for each frame of the target line drawing, the selected color drawings that match the target line drawing from each reference line drawing. When the selected color drawing matches one frame of the target line drawing, the computer device can use the selected color drawing as a reference, and based on the time information corresponding to the target line drawing and the selected line drawing in the line drawing sequence, color the target line drawing to obtain the initial color drawing corresponding to the target line drawing. When the selected color drawing matches multiple frames of target line drawings, the computer device can perform batch coloring processing on these target line drawings. Specifically, it can use the selected color drawing as a reference, and based on the time information corresponding to the multiple frames of target line drawings and the selected line drawing in the line drawing sequence, color these multiple frames of target line drawings to obtain multiple frames of initial color drawings corresponding to the multiple frames of target line drawings.

[0141] Optionally, the computer device can determine the matching degree between each frame of the target line drawing and each reference line drawing; and determine the selected color drawing with the highest matching degree with the target line drawing from each reference line drawing.

[0142] Optionally, the computer device may determine a first time of the target line drawing in the line drawing sequence and a reference time corresponding to each reference line drawing in the line drawing sequence, and then, for each reference time, determine a degree of match between the reference line drawing at the reference time and the target line drawing based on a time interval between the reference time and the first time. The degree of match is inversely correlated with the time interval.

[0143] Optionally, the computer device can determine the first object posture of the target object in the target line drawing, and the reference object postures corresponding to the target object in each reference line drawing; for each reference object posture, based on the similarity between the reference object posture and the first object posture, determine the matching degree between the reference line drawing to which the reference object posture belongs and the target line drawing. Among them, the matching degree is positively correlated with the similarity. When two variables are positively correlated, it means that when other conditions remain unchanged, the two variables change in the same direction. When one variable changes from large to small, the other variable also changes from large to small. It can be understood that the positive correlation here means that the direction of change is the same, but it does not require that when one variable changes a little, the other variable must also change.

[0144] Optionally, the computer device may determine a first time of the target line drawing in the line drawing sequence, as well as reference times corresponding to each reference line drawing in the line drawing sequence. The computer may then determine a first object pose of the target object in the target line drawing, and perform pose extraction on each frame of the reference line drawing whose time interval with the first time is less than or equal to an interval threshold, thereby determining the reference object poses corresponding to the target object in each of these reference line drawings. Finally, the computer device may determine, for each reference object pose, a degree of match between the reference line drawing to which the reference object pose belongs and the target line drawing based on the similarity between the reference object pose and the first object pose.

[0145] Optionally, the computer device can perform feature extraction on the selected color draft image to obtain the reference color draft features of the selected color draft image; perform feature extraction on each target line draft image to obtain the target line draft features corresponding to each target line draft image; for each target line draft feature, based on the target line draft feature and the reference color draft feature, determine the target fusion feature corresponding to the target line draft image; integrate the time information corresponding to each target line draft image and the reference line draft image in the line draft image sequence into each target fusion feature to obtain the time enhancement feature corresponding to each target line draft image; decode the time enhancement feature to obtain the initial color draft image corresponding to each target line draft image.

[0146] Optionally, the computer device can perform semantic analysis on the reference color draft image to obtain reference semantic features of the reference color draft image, and then obtain initial fusion features corresponding to the target line draft image based on the splicing results of the target line draft features and the reference color draft features, and then integrate the semantic information represented by the reference semantic features into the initial fusion features to obtain target fusion features corresponding to the target line draft image.

[0147] Optionally, the computer device can splice the target line draft features and the reference color draft features to obtain the spliced ​​features corresponding to the target line draft image, and then enhance the spliced ​​features in the spatial dimension to obtain the initial fusion features corresponding to the target line draft image.

[0148] Optionally, the computer device may perform feature extraction on the target line drawing to obtain initial line drawing features of the target line drawing; then, superimpose noise on the initial line drawing features to obtain target line drawing features corresponding to the target line drawing. In this embodiment, decoding the temporal enhancement features to obtain an initial color draft image corresponding to the target line drawing includes: performing denoising decoding on the temporal enhancement features to obtain the initial color draft image corresponding to the target line drawing.

[0149] Optionally, the computer device may combine the line drawing features of each target image and the reference line drawing features of the reference line drawing to obtain a comprehensive line drawing feature corresponding to the multiple target line drawing images, and combine the target fusion features to obtain a comprehensive fusion feature corresponding to the multiple target line drawing images. The computer device may then determine a weight matrix corresponding to the multiple target line drawing images based on the comprehensive line drawing features, and perform feature enhancement processing on the comprehensive fusion features based on a temporal attention mechanism using the weight matrix as a parameter to obtain a temporally enhanced feature corresponding to each target line drawing image.

[0150] The following describes the coloring process of the initial color draft, taking the case of batch coloring of multiple frames of target line drafts with reference to the same frame of reference color draft as an example.

[0151] In a specific embodiment, Figure 5 As shown, the computer device uses the reference color draft image 501 as a reference to batch colorize multiple frames of the target line drawing 502. In this case, the computer device can input the reference color draft image 501 into the multimodal pre-training module CLIP to obtain the reference semantic feature Feature_ref_clip; and pass the reference color draft through the encoding module VAE Enc and the feature extraction network Reference U-Net to obtain the potential feature representation of the reference color draft image, namely the reference color draft feature Feature_ref_reference_unet. The reference color draft feature can characterize the appearance details of the target object (such as facial features, clothing texture, etc.) as well as information in spatial and temporal dimensions. During the coloring process of the target line drawing, it is used to ensure the consistency of the coloring result of the target line drawing with the reference color draft image.

[0152] Optionally, the computer device can input multiple frames of target line drawings 502 into the line drawing feature extraction module Lineart Guider respectively to obtain the initial line drawing feature Feature_lineart of each target line drawing, and then superimpose random Gaussian noise Noise on these initial line drawing features to obtain the target line drawing feature Feature_lineart_noise with noise. Then, the computer device splices the target line drawing feature Feature_lineart_noise and the reference color draft feature Feature_ref_reference_unet to obtain the spliced ​​feature. For example, if the height of the target line drawing 502 is h, the width is w and the number is t frames, Figure 5 As shown, by splicing the target line draft features of each target line draft image 502, a comprehensive line draft feature 503 with dimensions of w*h*t is obtained. After copying the reference color draft feature 504 t times, the comprehensive line draft features 503 are spliced ​​together to obtain a spliced ​​feature 505 with dimensions of 2w*h*t.

[0153] Optionally, the computer device can input the splicing feature 505 into the spatial attention module, perform feature enhancement in the spatial dimension based on the spatial attention mechanism, and obtain the initial fusion feature Feature_lineart_noise_spatial corresponding to each target line drawing. Optionally, the input QKV of the spatial attention mechanism all comes from the splicing feature 505.

[0154] Optionally, the computer device can input the initial fused feature Feature_lineart_noise_spatial and the reference semantic feature Feature_ref_clip into a cross-attention module. Using the cross-attention mechanism, the semantic information represented by the reference semantic feature is integrated into the initial fused feature to obtain the target fused feature Feature_lineart_noise_cross corresponding to each target line drawing. Optionally, the input Q of the cross-attention mechanism comes from the initial fused feature, while the input KV comes from the reference semantic feature.

[0155] Optionally, the computer device can input the target fusion feature Feature_lineart_noise_cross into the temporal attention module. Using the temporal attention mechanism, the temporal information of each target line drawing in the line drawing sequence is incorporated into the target fusion feature to obtain the temporal enhancement feature Feature_lineart_noise_temporal. Optionally, during the noise addition and denoising process, the temporal attention mechanism can also calculate the correlation between different time steps and assign a weight to the features of each time step, thereby highlighting the temporal information that is more important to the current task.

[0156] Optionally, the computer device can use a denoising model to continuously reduce noise in the temporal enhancement features, and finally use a decoding module (VAE Dec) to restore the features to an image, thereby obtaining an initial color draft image 506 corresponding to each target line drawing image 502. Optionally, the denoising model can be a denoising diffusion implicit model (DDIM) or a denoising diffusion probabilistic model (DDPM), which is not limited here.

[0157] It should be noted that the above process involves three attention mechanisms, namely spatial attention, cross attention and temporal attention. The attention calculation method is the same, and the corresponding formula is:

[0158]

[0159] Where d is the dimension. They differ in their input features and in how the weight matrices are determined. Specifically, the spatial attention mechanism's QKV is derived from concatenated features; the cross-attention mechanism's input Q is derived from noisy line drawing features, while the KV is derived from reference semantic features; and the temporal attention mechanism's input QKV is derived from noisy line drawing features.

[0160] In the above embodiment, when the selected color draft image matches multiple frames of target line draft images, the selected color draft image is used as a reference, and based on the time information corresponding to the multiple frames of target line draft images and the selected line draft images in the line draft image sequence, the multiple frames of target line draft images are colored to obtain multiple frames of initial color draft images corresponding to the multiple frames of target line draft images. This can realize batch coloring processing for the target line draft images, and can improve work efficiency while ensuring the accuracy of coloring.

[0161] In one embodiment, determining a target first area and a target second area having a matching relationship from a first area and a second area includes: determining a reference color contained in a reference color draft image based on an area image of the second area; performing pixel color correction on each pixel point in the first area based on each reference color to obtain a pixel correction image corresponding to the first area; and determining a target first area and a target second area having a matching relationship from the first area and the second area based on the pixel correction image.

[0162] Specifically, the computer device can determine the colors included in the reference color draft image based on the regional image of the second region and use these colors as reference colors. The computer device can then perform pixel color correction on each pixel in the first region based on these reference colors to obtain a pixel-corrected image corresponding to the first region. After completing the pixel color correction and obtaining the pixel-corrected image, the computer device can perform image matching on the first region and the second region based on the pixel-corrected image, and determine a matching target first region and target second region from the first region and the second region.

[0163] Optionally, when it is determined based on the pixel correction map that there is a target second area matching the first area in the second area, the computer device may perform regional color correction on the pixel correction map corresponding to the first area based on the regional image of the target second area. Optionally, when there is no target second area matching the first area in the second area, the computer device may clear the color in the first area to avoid incorrect coloring. Optionally, when there is no target second area matching the first area in the second area, the computer device may also keep the pixel color in the pixel correction map of the first area unchanged. When all the first areas included in the initial color draft image have been traversed, the target color draft image corresponding to the target line draft image can be obtained.

[0164] Optionally, the pixel color of each pixel in the pixel correction image is any reference color. That is to say, through pixel color correction, the color composition and the reference color draft can be obtained. Figure 1 Consistent pixel correction map to ensure color consistency.

[0165] Optionally, for each pixel in the first area, if the pixel color of the pixel does not match any reference color, the computer device may query the initial color draft image for a reference pixel whose pixel color matches the reference color, and determine the neighboring pixel closest to the pixel from each reference pixel. The computer device may then replace the pixel color of the pixel with the pixel color of the neighboring pixel. By traversing all pixels in the first area, a pixel correction image corresponding to the first area can be obtained. Taking the case where the first area includes pixel 601 to pixel map as an example, if the pixel color of pixel 601 does not match any reference color, and pixels 602 to 604 all match the reference color, since pixels 602 and 603 are closest to pixel 601, the computer device may replace the pixel color of pixel 601 with the pixel color of pixel 602, thereby obtaining a pixel correction image A corresponding to the first area. The computer device may also replace the pixel color of pixel 601 with the pixel color of pixel 603, thereby obtaining a pixel correction image B corresponding to the first area.

[0166] Optionally, when the reference colors corresponding to the neighboring pixels closest to the pixel point are different, the computer device can determine a selected color that is closer to the pixel color of the pixel point from these reference colors, and replace the pixel color of the pixel point with the selected color. Figure 6 In the figure, the pixel color of pixel 603 is closer to the pixel color of pixel 601 than that of pixel 602. Based on this, the computer device can determine the pixel color of pixel 603 as the selected color, and replace the pixel color of pixel 601 with the selected color to obtain the pixel corrected image B corresponding to the first area.

[0167] In the above embodiment, pixel color correction is first performed according to pixel points, and then the matching relationship between areas is determined, so that regional color correction can be performed on the matching areas later. The color correction of the initial color draft image can be completed through different correction granularities, which is beneficial to improving the color correction effect and further improving the coloring accuracy.

[0168] In one embodiment, based on each reference color, pixel color correction is performed on each pixel in the first area to obtain a pixel correction map corresponding to the first area, including: determining the pixel color of each pixel in the first area; for each pixel, determining a target color that matches the pixel color of the pixel from each reference color; when the pixel color of the pixel is inconsistent with the target color, replacing the pixel color with the target color to obtain a pixel correction map corresponding to the first area.

[0169] Specifically, the computer device can determine the pixel color of each pixel in the first area by performing color recognition on the first area. Then, for each pixel, the computer device can determine the target color that matches the pixel color of the pixel from the reference colors contained in the reference color draft image. When the pixel color of the pixel is inconsistent with the target color, the computer device can replace the pixel color with the target color. If the pixel color of the pixel is consistent with the target color, no replacement is required. When all the pixels in the first area have been traversed, the computer device can obtain a pixel correction image corresponding to the first area.

[0170] Optionally, the number of target colors may be one or more. In the case of multiple target colors, if the pixel color of a pixel point is inconsistent with any of the target colors, the computer device may replace the pixel color with the target color that is closest to the pixel color.

[0171] In the above embodiment, for each pixel point, a target color matching the pixel color of the pixel point is determined from the reference colors; in the case that the pixel color of the pixel point is inconsistent with the target color, the pixel color is replaced with the target color, and a pixel correction map with the color composition consistent with the reference color sample can be obtained, which can ensure that the color information in the reference color sample can be migrated to the target color sample, and is beneficial to further improve the coloring accuracy. Figure 1

[0172] The specific manner of determining the target color matching the pixel color is not unique.

[0173] In an optional embodiment, the computer device can obtain a color specification map configured for the sequence of line drawing maps, and the color specification map includes color groups respectively configured for the component structures of the target object drawn in the line drawing map. The color specification map is a basis for the user to select colors for the line drawing map. That is, in the process of manually coloring the reference line drawing map, the user can select corresponding colors for each component structure in the color specification map, and then fill the selected colors into the corresponding regions of the line drawing map to obtain a reference color sample corresponding to the reference line drawing map. Each color group can include intrinsic colors, highlight colors, first-level shadow colors, second-level shadow colors, etc. Therefore, the colors included in the color specification map are usually consistent with the colors included in the reference color sample. In this case, the computer device can determine a selected color closest to the pixel color of the pixel point from the intrinsic colors of the color specification map, and determine each color included in the color group to which the selected color belongs as a target color matching the pixel color.

[0174] In an optional embodiment, determining the target color matching the pixel color of the pixel point from the reference colors includes: determining a color feature value of the pixel color and a reference feature value of each reference color; for each reference color, determining a color distance between the reference color and the pixel color based on the difference between the reference feature value of the reference color and the color feature value; and determining the reference color with the smallest color distance to the pixel color as the target color matching the pixel color.

[0175] The feature value of the color can represent the characteristics of the color, and for example, can include color values of each color channel, or can be obtained by performing operations on the color values of each channel. The color channel can be an RGB (Red Green Blue) color channel or an HSV (Hue Saturation Value) color channel.

[0176] ​Specifically, the computer device may determine the color characteristic value of the pixel color of the pixel point in the first area, as well as the reference characteristic value of each reference color included in the reference color draft image. Then, for each reference color, the computer device may calculate the difference between the reference characteristic value of the reference color and the color characteristic value of the pixel point, and based on the difference, determine the color distance between the reference color and the pixel color. Finally, the computer device may determine the reference color with the smallest color distance from the pixel color as the target color that matches the pixel color.

[0177] Optionally, the computer device may determine the difference or ratio between the reference characteristic value and the color characteristic value as the color distance between the reference color and the pixel color.

[0178] In a specific implementation, the characteristic value of a color includes the color values ​​corresponding to the three RGB (Red, Green, Blue) channels. For each channel, the computer device can subtract the color values ​​corresponding to the reference color and the pixel color in that channel to obtain the color value difference in that channel, and then perform a square root operation on the sum of the squares of the color value differences to obtain the color distance between the reference color and the pixel color.

[0179] In a specific implementation, for the j-th first region contained in the i-th frame initial color draft image, denoted as R_i_j, for the color of the k-th pixel in the region R_i_j, denoted as C_i_j_k, the distance between the pixel color C_i_j_k of the pixel and the reference color Colors is calculated. If there are M pixels in the region R_i_j and the number of reference colors Colors is N, then an M*N distance matrix can be obtained, and each row in the distance matrix represents the color distance between the pixel color of a certain pixel and N reference colors. Among them, the color distance can be calculated using the square difference formula, that is: the color values ​​of the three RGB channels corresponding to the pixel colors in the initial color draft image are subtracted from the color values ​​of the three RGB channels corresponding to the reference colors according to the channels to obtain the color value differences of each channel, and then the sum of the squares of the differences of each color value is calculated, and finally the square root is taken to obtain the color distance between the pixel color and the reference color.

[0180] Alternatively, in the M*N matrix, find the target color closest to pixel color C_i_j_k in the kth row. If pixel color C_i_j_k is inconsistent with the target color, replace pixel color C_i_j_k with the target color to complete the replacement correction for the kth pixel in region R_i_j. By traversing all pixels in the jth first region, the region correction map corresponding to the first region can be obtained.

[0181] In the above embodiment, for each reference color, the color distance between the reference color and the pixel color is determined based on the difference between the reference characteristic value of the reference color and the color characteristic value of the pixel color, and the reference color with the smallest color distance to the pixel color is determined as the target color that matches the pixel color. This can ensure that the replaced target color is closest to the pixel color determined by the reference coloring, and can further improve the coloring accuracy while transmitting the color information of the reference color draft.

[0182] It can be understood that pixel color correction can replace all colors in the initial color draft image with the reference colors that appear in the reference color draft image, thereby ensuring the color consistency of the color draft image. In practical applications, when the number of pixels with color differences is small or the colors are relatively similar, manual correction is difficult to distinguish. However, the pixel color correction proposed in this application can correct these pixel color differences that are difficult to distinguish with the naked eye, which is conducive to further improving coloring accuracy.

[0183] In one embodiment, there are multiple second regions. In this embodiment, determining a target first region and a target second region that have a matching relationship from the first region and the second region includes: extracting image features from the first region and the second region to obtain regional image features of the first region and the second region; determining, from each second region, a candidate region whose regional image features have the highest similarity to the first region; and determining that the candidate region matches the first region if the similarity corresponding to the candidate region meets a similarity condition, thereby obtaining the target first region and the target second region that have a matching relationship.

[0184] The similarity satisfying the similarity condition may mean that the similarity is greater than or equal to a similarity threshold.

[0185] In actual applications, when there are multiple second regions, the computer device can select a target region that matches the first region from the multiple second regions, so that the color of the first region can be subsequently corrected based on the regional image of the target region. That is, in this embodiment, the target first region is the entire first region, and the target second region is one of the multiple second regions.

[0186] Specifically, the computer device may extract image features from each of the first and second regions to obtain regional image features for each of the first and second regions. Then, for each second region, the computer device may calculate the similarity between the regional image features of the second region and the regional image features of the first region. Furthermore, the computer device may determine, from each second region, that the candidate region has the highest similarity in regional image features to the first region. Finally, if the corresponding similarity of the candidate region satisfies a similarity condition, the computer device may determine that the candidate region matches the first region, thereby obtaining a matching target first region and target second region.

[0187] Optionally, the computer device may use a feature extraction network to extract image features from the first region and the second region to obtain corresponding regional image features. Optionally, the computer device may also extract image features using edge detection, semantic analysis, or other methods to obtain corresponding regional image features.

[0188] Optionally, the similarity between image features in different regions can be calculated using cosine similarity or Euclidean distance.

[0189] In an optional embodiment, determining, from the second regions, a candidate region whose regional image features are most similar to those of the first region includes: determining the number of pixels corresponding to each second region and the number of pixels corresponding to the first region; determining, from each second region, an alternative region whose number of pixels matches that of the first region; and determining, from each alternative region, a candidate region whose regional image features are most similar to those of the first region.

[0190] Among them, the matching of the number of pixels in the two areas may mean that the difference in the number of pixels in the two areas is less than or equal to the difference threshold. The pixel difference can be represented by a difference or a ratio. For line drawings with complex lines, the number of closed areas contained is also large. If similar matching is performed for each second area, a large amount of computing resources will be consumed. Based on this, the computer device can determine the number of pixels corresponding to each second area and the number of pixels corresponding to the first area, and then determine the candidate area whose number of pixels matches the first area from each second area, and then perform regional image feature matching with the first area for each candidate area, and calculate the similarity between the regional image features of each candidate area and the regional image features of the first area, and then determine the candidate area whose regional image features are most similar to the first area from each candidate area.

[0191] In practice, the number of pixels in regions with a mapping relationship is typically close. For example, the number of pixels in the eyebrow region always remains within a certain range regardless of the subject's posture. Therefore, by first comparing the pixel counts of the first and second regions and then performing image feature matching on candidate regions with matching pixel counts, the computational complexity of the image feature matching process can be significantly reduced, ensuring the accuracy of the matching results while conserving computing resources.

[0192] In an optional embodiment, the computer device can determine the reference colors included in the reference color draft image based on the regional image of the second area; based on each reference color, perform pixel color correction on each pixel in the first area to obtain a pixel correction image corresponding to the first area. Then, by performing image feature extraction and regional image feature similarity matching on the pixel correction image corresponding to the first area, a target first area and a target second area with a matching relationship are determined from the first area and the second area. Furthermore, based on the regional image of the target second area, the computer device can perform regional color correction on the pixel correction image corresponding to the target first area to obtain a target color draft image corresponding to the target line drawing.

[0193] Optionally, the first region and the second region are both mapped based on closed regions enclosed by color lines. In other words, the first region and the second region are both closed regions that are expected to be filled with the same color. The pixel correction image obtained through pixel color correction may contain multiple colors. Based on this, regional color correction can be used to unify the colors of pixels in the same first region.

[0194] Optional, such as Figure 7 As shown, the computer device can input the reference color draft image 701 and the initial color draft image 702 after pixel color correction into the feature extraction network to obtain two feature maps of dimension h*w*C. For the i-th region obtained by the flooding method segmentation, denoted as R_i, the computer device can average all the feature values ​​belonging to R_i in the feature map corresponding to the reference color draft image to obtain the regional image features of the R_i region in the reference color draft image. Assuming that the reference line draft image includes J closed areas, the regional image features of the reference color draft image can constitute the reference image regional features of dimension J*C. Similarly, if the target line draft image includes I closed areas, the initial color draft image after pixel color correction can obtain the target image regional features of dimension I*C.

[0195] The aforementioned reference image region features and target image region features are enhanced through mask and position encoding modules, as well as a Transformer network based on a multi-head attention mechanism. The multi-head attention mechanism can include a self-attention mechanism and a cross-attention mechanism. Finally, the computer device uses matrix multiplication to obtain a correlation matrix (Correlation Matrix) that characterizes the similarity relationships between different regions. The i-th row in this matrix represents the i-th region in the initial color draft image, and the i-th row has J columns. The value of each column represents the similarity relationship between the i-th region in the initial color draft image and the J regions in the reference color draft image. The computer device can select the color of the region with the greatest similarity in the reference color draft image as the color of the i-th region in the initial color draft image. At this point, the goal of unifying the colors of all pixels in the i-th region can be achieved.

[0196] In the above embodiment, a candidate region whose regional image features have the highest similarity with the first region is determined from each second region. When the similarity corresponding to the candidate region meets the similarity condition, the candidate region is determined as the target region matching the first region. This can ensure that the target region is highly similar to the first region, thereby ensuring the accuracy of the color correction result, which is conducive to further improving the coloring accuracy.

[0197] In one embodiment, the line drawing coloring processing method further includes: when there is no target area matching the first area in each second area, clearing the color in the first area; when all first areas have been traversed, obtaining a target color drawing corresponding to the target line drawing.

[0198] Specifically, when the similarity between the i-th area in the initial color draft image and any area in the reference color draft image is relatively low, it can be considered that there is no target area matching the first area in each second area. In this case, in order to avoid incorrect coloring, the filled color in the first area can be cleared to obtain a blank area. When all the first areas in the initial color draft image have been traversed, the target color draft image corresponding to the target line drawing can be obtained. It can be understood that during the traversal process, each first area in the initial color draft image is either color-corrected based on the matching target area, or restored to an uncolored blank area through color clearing.

[0199] In the above embodiment, when there is no target area matching the first area in each second area, the color in the first area is cleared, which can avoid incorrect coloring and is conducive to further improving the coloring accuracy.

[0200] In one embodiment, the line drawing coloring processing method also includes: for each color line contained in the reference line drawing, extracting the colors on both sides of the color line from the reference color draft image to form a color pair on both sides corresponding to the color line; for the uncolored blank area in each first area, determining the target color line enclosing the blank area from the target line drawing, and determining the color of the adjacent area of ​​the blank area from the target color draft image; from the color pair on both sides corresponding to the target color line, querying the opposite side color corresponding to the adjacent area color, and filling the opposite side color into the blank area.

[0201] The uncolored blank area in the first area may be an area that was not colored during the generation of the initial color draft image, or an area that was colored during the generation of the initial color draft image but whose color was cleared during the color correction process.

[0202] Specifically, the computer device can determine the mapping position of the color line from the reference color draft image according to the mapping relationship between the reference line draft image and the reference color draft image for each color line contained in the reference line draft image, and determine the colors on both sides of the mapping position as the two-side color pairs corresponding to the color line. It can be understood that each two-side color pair contains two corresponding colors. These two colors can be the same color or different colors. Then, for the uncolored blank area in each first area, the computer device can determine the target color line enclosing the blank area from the target line draft image, and determine the mapping position of the target color line from the target color draft image according to the mapping relationship between the target line draft image and the target color draft image, and then determine the color of the adjacent area of ​​the blank area from the target color draft image. Finally, the computer device can query the opposite color corresponding to the color of the adjacent area from the two-side color pairs corresponding to the target color line, and fill the opposite color into the blank area.

[0203] For example, the color pairs corresponding to color line A include color B1-color B2 and color B3-color B4. Therefore, for the blank area enclosed by color line A in the first region, if the adjacent area to that blank area is color B1, the color B2 opposite to color B1 can be determined from the color pairs on both sides of color line A and filled into the blank area with color B2 to complete the color complementation.

[0204] In the above embodiment, for the uncolored blank areas in the target color draft image, coloring is completed through color line matching and color pair query, which can further improve the coloring accuracy.

[0205] In an optional embodiment, the colors on both sides of the color line are extracted from the reference color draft image, including: determining the color line area where the color line is located from the reference color draft image according to the mapping relationship between the reference line draft image and the reference color draft image; expanding the color line area to obtain a color line expansion area; and extracting the colors on both sides of the color line from the color line expansion area.

[0206] Specifically, the computer device can determine the color line region within the reference color draft image based on a mapping relationship between the reference line drawing and the reference color draft image. This color line region can be the region formed by the pixels corresponding to the color line in the reference color draft image. The computer device can then employ a dilation algorithm used in computer vision to dilate the color line region, obtaining a dilated color line region encompassing the color line and at least a portion of the pixels on either side of the color line. Finally, the computer device extracts the colors on either side of the color line from the dilated color line region.

[0207] In one possible implementation, the complementary color processing process of the color pair query based on color line matching may include a color pair extraction process and a complementary color process. Figure 8 As shown, the color pair extraction process may include the following steps:

[0208] Step S801, traversing the color lines in the reference line drawing;

[0209] Step S802, color line area expansion;

[0210] Step S803: extracting the color pairs on both sides.

[0211] Specifically, the computer device can traverse the color lines in the reference line drawing to obtain a color line set. Optionally, for the color line of the i-th color in the reference line drawing, it is recorded as cline_i. The computer device can determine the number Q of line colors contained in the reference line drawing based on the color value, and traverse each color to construct Q pictures, retaining only lines of one color in each picture. On this basis, the computer device can extract the color pairs on both sides of the color line according to the position of the color line cline_i in the reference color draft, recorded as color_pairs_i. Optionally, the expansion algorithm in computer vision can be used to enlarge the color line area where the color line is located. The expanded area includes the colors on both sides of the color line, so the color pairs on both sides of the color line can be extracted in the color line expansion area.

[0212] Optionally, the computer can iterate over all color lines in the reference line drawing and extract color pairs on either side, denoted as color_pairs. These color pairs represent all color combinations on either side of any color line in the reference line drawing. These combinations serve as a dictionary for subsequent color complementation. In other words, once a color line and the color of one side of it are determined, the computer can look up the corresponding color on the other side in the dictionary and fill the blank area with the changed color, completing the color complementation for that blank area.

[0213] Optional, such as Figure 9 As shown, the color complement process may include the following steps:

[0214] Step S901, traverse the outer color lines of the blank area;

[0215] Step S902, dilating the color line area;

[0216] Step S903: Find the opposite side color.

[0217] Specifically, the computer device can traverse the outer color lines of the blank area in the target color draft image. For the i-th blank area in the target color draft image, it is denoted as R_i. The blank area is expanded, and the color of the color line enclosing the blank area is found, denoted as cline_k_color, and the corresponding color line is denoted as cline_k. Then, the expansion process in computer vision is used again to enlarge the color line area of ​​the color line cline_k, and the color filled on the other side of the color line, that is, the color of the adjacent area of ​​the blank area, is extracted, and denoted as cline_k_side_region_color. Finally, all the two-side color pairs whose color line color is cline_k_color are searched in the dictionary, and then according to the obtained single-side color cline_k_side_region_color, the opposite side color of the single-side color in the two-side color pair is searched from the two-side color pair, and the opposite side color is filled into the blank area.

[0218] In the above embodiment, color extraction is performed through region dilation processing, and the accuracy of the extracted color can be ensured by controlling the dilation parameters, and color complementation can be performed based on the extracted color, which is conducive to further improving the accuracy of coloring.

[0219] In one embodiment, Figure 10 As shown, a method for coloring a line drawing is provided. The method can be executed by a computer device, which can be a terminal or a server. In this embodiment, the method includes the following steps:

[0220] Step S1001: obtaining a target line drawing to be colored and candidate line drawings corresponding to the colored drawings from a line drawing sequence corresponding to the target object;

[0221] The number of target line drawings is multiple frames; the number of candidate color drawings is at least two frames;

[0222] Step S1002, determining a first time corresponding to the target line drawing in the line drawing sequence, and a second time corresponding to the candidate line drawing in the line drawing sequence;

[0223] Step S1003, determining a matching degree between the candidate line drawing and the target line drawing based on a time interval between the second time and the first time;

[0224] Among them, the matching degree is inversely correlated with the time interval;

[0225] Step S1004: if the matching degree between the candidate line drawing and the target line drawing satisfies the matching condition, the candidate line drawing is determined as a reference line drawing corresponding to the target line drawing, and the color drawing corresponding to the candidate line drawing is determined as a reference color drawing;

[0226] Step S1005: for each frame of the reference color draft image, if the reference color draft image matches multiple frames of the target line draft images, feature extraction is performed on each target line draft image that matches the reference color draft image to obtain initial line draft features of each target line draft image;

[0227] Step S1006: superimposing noise on the initial line drawing features to obtain target line drawing features corresponding to the target line drawing image;

[0228] Step S1007: performing feature extraction on the reference color draft image to obtain reference color draft features of the reference color draft image; performing semantic analysis on the reference color draft image to obtain reference semantic features of the reference color draft image;

[0229] Step S1008, stitching the target line draft features and the reference color draft features to obtain stitching features corresponding to the target line draft image;

[0230] Step S1009: Based on the spatial attention mechanism, the splicing features are enhanced in the spatial dimension to obtain the initial fusion features corresponding to the target line drawing;

[0231] Step S1010: Based on the cross-attention mechanism, the semantic information represented by the reference semantic feature is integrated into the initial fusion feature to obtain the target fusion feature corresponding to the target line drawing;

[0232] Step S1011, determining a weight matrix corresponding to each target line draft image based on each target line draft feature and a reference line draft feature corresponding to the reference line draft image;

[0233] Step S1012: Using the weight matrix as a parameter, the target fusion features are enhanced based on the temporal attention mechanism to obtain the temporal enhancement features corresponding to the target line drawings;

[0234] Step S1013, performing denoising and decoding processing on the temporal enhancement features to obtain initial color draft images corresponding to each target line draft image;

[0235] Step S1014, based on the colored lines included in the line drawing, determining a closed area enclosed by the colored lines from the line drawing;

[0236] The line drawings include target line drawings and reference line drawings;

[0237] Step S1015, determining the coloring area included in the color draft image based on the mapping position of the closed area in the color draft image corresponding to the line draft image;

[0238] The coloring area includes a first area in the initial color draft image corresponding to the target line drawing, and a second area in the reference color draft image corresponding to the reference line drawing;

[0239] Step S1016, determining a reference color included in the reference color draft image based on the regional image of the second region;

[0240] Step S1017, determining the pixel color of each pixel in the first area;

[0241] Step S1018: for each pixel, determining the color characteristic value of the pixel color of the pixel and the reference characteristic value of each reference color;

[0242] Step S1019, for each reference color, determining a color distance between the reference color and the pixel color based on a difference between the reference characteristic value and the color characteristic value of the reference color;

[0243] Step S1020, determining the reference color with the smallest color distance to the pixel color as the target color that matches the pixel color;

[0244] Step S1021: if the pixel color of the pixel point is inconsistent with the target color, the pixel color is replaced with the target color to obtain a pixel correction image corresponding to the first area;

[0245] Optionally, if the pixel color of the pixel point is consistent with the target color, the pixel color of the pixel point is kept unchanged;

[0246] Step S1022: extracting image features of the first region based on the pixel correction image to obtain regional image features of the first region;

[0247] Step S1023, determining, from each second region, a candidate region whose regional image feature has the highest similarity with the first region;

[0248] Step S1024: if the similarity corresponding to the candidate region satisfies the similarity condition, the candidate region is determined as a target region matching the first region, and regional color correction is performed on the pixel correction image based on the regional image of the target region;

[0249] The target area refers to a second area that matches the first area among the multiple second areas included in the reference color draft image;

[0250] Step S1025: when there is no target area in the second area that matches the first area, clear the color in the first area;

[0251] Step S1026: When all first regions have been traversed, a target color draft image corresponding to the target line draft image is obtained;

[0252] Step S1027: for each color line included in the reference line drawing, according to the mapping relationship between the reference line drawing and the reference color drawing, determine the color line area where the color line is located in the reference color drawing;

[0253] Step S1028, dilating the color line region to obtain a color line dilation region, and extracting the colors on both sides of the color line from the color line dilation region to form a color pair corresponding to both sides of the color line;

[0254] Step S1029: for the uncolored blank area in each first area, determine a target color line enclosing the blank area from the target line drawing, and determine the color of the adjacent area of ​​the blank area from the target color drawing;

[0255] Step S1030 , searching for the opposite color corresponding to the color of the adjacent area from the color pairs on both sides of the target color line, and filling the blank area with the opposite color.

[0256] In the above-mentioned line drawing coloring processing method, the coloring process refers to the reference color drawing corresponding to the reference line drawing in the line drawing sequence of the same target object, which can ensure the semantic consistency and detail correspondence between the reference color drawing and the target line drawing, so that the initial color drawing and the reference color drawing can maintain a high degree of consistency in color and style, which is conducive to improving the coloring accuracy. In addition, the time information of the target line drawing and the reference line drawing in the line drawing sequence is also taken into account in the coloring process, which can better understand the semantic relationship between the line drawings, thereby more accurately transferring the color information in the reference color drawing to the target line drawing, which is conducive to further improving the coloring accuracy. On the basis of this, the first area included in the corresponding initial color draft image is determined based on the closed area in the target line draft image, and the second area included in the corresponding reference color draft image is determined based on the closed area in the reference line draft image. Then, based on the regional image of the target area matching the first area in the second area, the matching first area is color corrected to obtain the target color draft image corresponding to the target line draft image. On the basis of the initial color draft image, based on the regional matching relationship between the reference color draft image and the initial color draft image, the first area in the initial color draft image matching the target area is color corrected, which can further ensure the consistency in color and style, and is conducive to further improving the coloring accuracy.

[0257] The coloring method for coloring a color draft provided in this application can be applied to scenes such as illustration, game design, and animation coloring. The following uses the animation coloring scene as an example to introduce the coloring method for coloring a color draft provided in this application in detail.

[0258] In a specific embodiment, for a given sequence of line drawings, one or more frames can be selected for manual coloring. In practical applications, the sequence of line drawings to be colored is called a card line drawing. A card line drawing usually contains dozens of frames of pictures. These pictures constitute a complete set of actions. The movement amplitude of the characters between adjacent pictures is not large, but the movement amplitude between pictures that are farther apart is large. If the movement amplitude between the reference color drawing used as a coloring reference and the target line drawing to be colored in the automatic coloring process is too large, it will affect the accuracy of coloring. Based on this, multiple frames of line drawings with more complex lines in the whole card and a certain number of frames spaced apart can be selected for manual coloring. It can be understood that complex line drawings mean that the colors of the characters are more comprehensive and suitable as a coloring reference. In the field of animation coloring, the colors that each component of the character should have are configured through a color designation map, and the color of each component is further subdivided into highlights, shadows and inherent colors. The colorist can refer to the color designation map to manually color the selected line drawing to obtain a reference color drawing. As shown in FIG. Figure 11 As shown, from the 30 frames of line drawings, the 14th frame is manually colored based on the pre-configured color designation map 1101 to obtain the color draft map 1103 corresponding to the 14th frame of line drawing 1102.

[0259] On this basis, the line drawing coloring processing method provided in this application can be applied to color the target line drawing to be colored in the line drawing sequence, and obtain a color drawing sequence 1104 corresponding to the line drawing sequence, which is used to present the color animation effect of the cartoon character. Optionally, after automatic coloring, manual detection and modification are also supported. For example, dynamic playback inspection and highlighting of uncolored areas are supported for quickly locating coloring problems; lasso, brush, eraser and other tools are supported for quick correction; detailed modification by region and layer is supported; multiple export formats are supported, etc.

[0260] In an optional embodiment, the overall workflow of the line drawing coloring process is as follows: Figure 12 Shown, including:

[0261] Step S1201: Initialize the image generation algorithm.

[0262] The image generation algorithm is implemented based on the extended model, and image generation is completed through expansion and denoising. By initializing the image generation algorithm, the generation network model can be loaded. The specific structure of the generation network model is as follows Figure 5 As shown, it is used to color the target line drawing based on the time information of the target line drawing and the reference line drawing in the line drawing sequence with reference to the reference color drawing, so as to obtain the initial color drawing corresponding to the target line drawing.

[0263] Step S1202: pre-process the reference color draft image and the reference line draft image.

[0264] Specifically, based on the closed area enclosed by the color lines in the reference line drawing, the color and number of pixels in each area of ​​the reference color drawing are counted; based on the color lines in the reference line drawing, the colors on both sides of the color lines in the reference color drawing are counted to form color pairs on both sides.

[0265] Step S1203: unify the resolutions of the reference color draft image and the target line draft image.

[0266] Specifically, the reference color image and target line image to be input into the generative network model are cropped and edge-filled so that the image resolution is unified into a fixed size (e.g. 768*768 or 1024*1024). Figure 13 As shown, after unifying the resolution, a reference color draft image 1301 and a target line draft image 1302 can be obtained.

[0267] Step S1204: Generate an initial color draft image corresponding to the target line draft image.

[0268] Specifically, based on the generative network model loaded in step S1201, given a plurality of different random noises, an initial color draft corresponding to the target line draft is generated by diffusion and noise addition. Figure 13 In the process of generating the color draft image, an initial color draft image 1303 corresponding to the target line draft image 1302 can be generated. Furthermore, through the temporal attention mechanism, the temporal information of the target line draft image and the reference line draft image in the line draft image sequence can be considered, which is conducive to improving the coloring accuracy.

[0269] Step S1205: post-processing the initial color draft image.

[0270] Specifically, the image resolution of the initial color draft is restored to its original size, and the Anime Production Inspired Real-World Anime Super-Resolution (APISR) algorithm is used to improve the image quality.

[0271] Step S1206: color correction.

[0272] Specifically, the image generation algorithm may cause differences in pixel colors within the same area to be colored. In order to meet the coloring requirements, that is, the colors of any closed area are the same, color correction can be used to correct the colors of all pixels in the same closed area to be consistent and correct. Figure 13 In the embodiment, through color correction, the pixel point A whose color is inconsistent with other pixels in the closed area is updated to the correct color of the closed area.

[0273] Step S1207, color line complementation.

[0274] Specifically, some color blocks with low confidence will be filtered out through color correction, such as Figure 13 The color confidence of area B is low, so the color of this area is filtered out, forming a color draft image 1304 containing a blank area. By using color line complementation, the blank area with the filtered color can be recolored using the prior information of the colors on both sides of the color line in the reference line drawing. Figure 13 As shown, by coloring the blank area, a color draft image 1305 can be obtained.

[0275] Step S1208: Output the colored animation image.

[0276] Input the color draft image sequence corresponding to the line draft image sequence to obtain a color animation.

[0277] During the coloring process of the above-mentioned color draft image, a temporal attention mechanism is added based on the animation image generation framework to integrate the time information between the reference line draft and the line draft to be colored into the generation process, which is conducive to improving the coloring accuracy. In the animation coloring industry, the closed area of ​​the line draft is used as the basic unit for coloring, that is, a certain area needs to be filled with one color. The color of each pixel in the same area of ​​the initial color draft image generated based on the diffusion model may be different. Although the colors of these pixels are very close, and even cannot be distinguished by the naked eye, strictly speaking, it is necessary to ensure that the colors of all pixels in the area are consistent. Based on this, the present application solves the common problems faced by coloring schemes based on image generation through color correction. Furthermore, some wrong colors may be deleted during the color correction process, which inevitably leads to blank areas in the image. Therefore, using the colors on both sides of the color line as a priori information to complement the blank areas can further improve the coloring accuracy.

[0278] It should be understood that, although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence according to the instructions of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments described above can include multiple steps or multiple stages, and these steps or stages are not necessarily executed and completed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0279] Based on the same inventive concept, embodiments of the present application also provide a color draft image coloring processing device for implementing the aforementioned color draft image coloring processing method. The solution provided by this device is similar to the solution described in the aforementioned method. Therefore, the specific limitations of one or more embodiments of the color draft image coloring processing device provided below can be found in the aforementioned limitations of the color draft image coloring processing method, and will not be further elaborated here.

[0280] In one embodiment, Figure 14 As shown, a color draft image coloring processing device is provided, comprising:

[0281] The acquisition module 1401 is used to acquire a target line drawing to be colored and a reference line drawing corresponding to the reference color drawing from a line drawing sequence corresponding to the target object;

[0282] A coloring module 1402 is configured to color the target line drawing based on the time information of the target line drawing and the reference line drawing in the line drawing sequence, using the reference color drawing as a reference, to obtain an initial color drawing corresponding to the target line drawing;

[0283] The region division module 1403 is configured to determine a first region included in the corresponding initial color draft image based on the closed region in the target line draft image, and to determine a second region included in the corresponding reference color draft image based on the closed region in the reference line draft image;

[0284] The color correction module 1404 is used to determine a target first area and a target second area that have a matching relationship from the first area and the second area, and perform color correction on the target first area based on the area image of the target second area to obtain a target color draft image corresponding to the target line draft image.

[0285] In one embodiment, the coloring module 1402 includes: a line draft feature extraction unit, used to extract features of a target line draft image to obtain target line draft features of the target line draft image; a color draft feature extraction unit, used to extract features of a reference color draft image to obtain reference color draft features of the reference color draft image; a feature fusion unit, used to determine a target fusion feature corresponding to the target line draft image based on the target line draft features and the reference color draft features; a time information enhancement unit, used to integrate the time information of the target line draft image and the reference line draft image in the line draft image sequence into the target fusion feature to obtain a time enhancement feature corresponding to the target line draft image; a decoding unit, used to decode the time enhancement feature to obtain an initial color draft image corresponding to the target line draft image.

[0286] In one embodiment, the line drawing coloring processing apparatus further includes a semantic analysis module for performing semantic analysis on a reference color drawing to obtain reference semantic features of the reference color drawing. In this embodiment, the feature fusion unit includes: a splicing component for obtaining initial fusion features corresponding to the target line drawing based on the splicing results of the target line drawing features and the reference color drawing features; and a feature fusion component for integrating the semantic information represented by the reference semantic features into the initial fusion features to obtain target fusion features corresponding to the target line drawing.

[0287] In one embodiment, the stitching component is specifically used to: stitch the target line draft features and the reference color draft features to obtain stitching features corresponding to the target line draft image; and enhance the stitching features in the spatial dimension to obtain initial fusion features corresponding to the target line draft image.

[0288] In one embodiment, the line drawing feature extraction unit is specifically configured to: extract features from a target line drawing image to obtain initial line drawing features of the target line drawing image; and superimpose noise on the initial line drawing features to obtain target line drawing features corresponding to the target line drawing image. In this embodiment, the decoding unit is specifically configured to: perform denoising and decoding processing on the temporal enhancement features to obtain an initial color draft image corresponding to the target line drawing image.

[0289] In one embodiment, the temporal information enhancement unit is specifically used to: determine a weight matrix corresponding to the target line draft image based on the target line draft features and the reference line draft features corresponding to the reference line draft image; use the weight matrix as a parameter, perform feature enhancement processing on the target fusion features based on the temporal attention mechanism, and obtain the temporal enhancement features corresponding to the target line draft image.

[0290] In one embodiment, the number of target line drawings is multiple frames, and the number of reference color drawings is at least two frames. In this embodiment, the coloring module 1402 is specifically configured to: for each frame of the target line drawing, determine a selected color drawing from each reference line drawing that matches the target line drawing; if the selected color drawing matches multiple frames of the target line drawing, color the multiple frames of the target line drawing based on the time information corresponding to the multiple frames of the target line drawing and the selected line drawing in the line drawing sequence, thereby obtaining multiple frames of initial color drawings corresponding to the multiple frames of the target line drawing.

[0291] In one embodiment, the acquisition module 1401 includes: a line drawing acquisition unit, used to obtain a target line drawing to be colored and a candidate line drawing corresponding to a colored drawing from a line drawing sequence corresponding to the target object; a reference drawing determination unit, used to determine the candidate line drawing as a reference line drawing and the color drawing corresponding to the candidate line drawing as a reference color drawing when the candidate line drawing matches the target line drawing.

[0292] In one embodiment, the acquisition module 1401 also includes a line drawing matching unit, which is used to: determine a first time corresponding to the target line drawing in the line drawing sequence, and a second time corresponding to the candidate line drawing in the line drawing sequence; based on the time interval between the second time and the first time, determine the matching degree between the candidate line drawing and the target line drawing; the matching degree is inversely correlated with the time interval.

[0293] In one embodiment, the acquisition module 1401 also includes a line drawing matching unit, which is used to: determine a first object posture of the target object in the target line drawing, and a second object posture of the target object in the candidate line drawing; when the posture similarity between the second object posture and the first object posture meets the similarity condition, determine that the candidate line drawing matches the target line drawing.

[0294] In one embodiment, the area division module 1403 is specifically used to: determine a closed area enclosed by the color lines in the line drawing based on the color lines contained in the line drawing; the line drawing includes a target line drawing and a reference line drawing; based on the mapping position of the closed area in the color drawing corresponding to the line drawing, determine the colored area included in the color drawing; the colored area includes a first area in the initial color drawing corresponding to the target line drawing, and a second area in the reference color drawing corresponding to the reference line drawing.

[0295] In one embodiment, the color correction module 1404 includes: a reference color determination unit, which is used to determine the reference color contained in the reference color draft image based on the regional image of the second area; a pixel color correction unit, which is used to perform pixel color correction on each pixel point in the first area based on each reference color to obtain a pixel correction image corresponding to the first area; and an area matching unit, which is used to determine a target first area and a target second area with a matching relationship from the first area and the second area based on the pixel correction image.

[0296] In one embodiment, the pixel color correction unit includes: a pixel color determination component for determining the pixel color of each pixel point in the first area; a pixel color matching component for determining, for each pixel point, a target color that matches the pixel color of the pixel point from each reference color; and a pixel color correction component for replacing the pixel color with the target color when the pixel color of the pixel point is inconsistent with the target color, thereby obtaining a pixel correction image corresponding to the first area.

[0297] In one embodiment, the pixel color matching component is specifically used to: determine the color characteristic value of the pixel color and the reference characteristic value of each reference color; for each reference color, determine the color distance between the reference color and the pixel color based on the difference between the reference characteristic value and the color characteristic value of the reference color; and determine the reference color with the smallest color distance to the pixel color as the target color that matches the pixel color.

[0298] In one embodiment, there are multiple second regions. In this embodiment, the color correction module 1404 includes: a region feature extraction unit for extracting image features from the first region and the second region, respectively, to obtain regional image features of the first region and the second region; a similarity matching unit for determining, from each second region, a candidate region having the highest similarity in regional image features with the first region; and a region matching unit for determining that the candidate region matches the first region when the similarities corresponding to the candidate regions meet a similarity condition, thereby obtaining a target first region and a target second region having a matching relationship.

[0299] In one embodiment, the similarity matching unit is specifically used to: determine the number of pixels corresponding to each second area and the number of pixels corresponding to the first area; determine, from each second area, an alternative area whose number of pixels matches the first area; and determine, from each alternative area, a candidate area whose regional image features have the highest similarity with the first area.

[0300] In one embodiment, the color correction module 1404 is further used to: clear the color in the first area when there is no target area matching the first area in each second area; and obtain a target color draft image corresponding to the target line drawing when all first areas have been traversed.

[0301] In one embodiment, the color correction module 1404 is specifically used to: determine an image correspondence between the first area and the second area based on the image information respectively contained in the first area and the second area; determine a target second area that matches at least a portion of the image content of the first area from the second area based on the image correspondence; and determine a target first area corresponding to at least a portion of the image content from the first area.

[0302] In one embodiment, the line drawing coloring processing device also includes: a color pair extraction module, which is used to extract the colors on both sides of the color line from the reference color draft image for each color line included in the reference line drawing, and form a color pair on both sides corresponding to the color line; a color matching module, which is used to determine the target color line enclosing the blank area from the target line drawing for the uncolored blank area in each first area, and determine the color of the adjacent area of ​​the blank area from the target color draft image; a complementary color module, which is used to query the opposite color corresponding to the color of the adjacent area from the color pair on both sides corresponding to the target color line, and fill the opposite color into the blank area.

[0303] In one embodiment, the color pair extraction module is specifically used to: determine the color line area where the color line is located from the reference color draft image based on the mapping relationship between the reference line draft image and the reference color draft image; expand the color line area to obtain a color line expansion area; and extract the colors on both sides of the color line from the color line expansion area.

[0304] Each module in the above-mentioned color draft image coloring processing device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in the computer device in the form of software, so that the processor can call and execute the corresponding operations of each module.

[0305] The present application also provides a computer device, which can be a server or a terminal.

[0306] In one embodiment, the computer device may be a server, and its internal structure diagram may be as follows: Figure 15 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, abbreviated as I / O) and a communication interface. The processor, memory and input / output interface are connected via a system bus, and the communication interface is connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The database of the computer device is used to store data involved in the above-mentioned color draft image coloring process. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a color draft image coloring method is implemented.

[0307] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 16As shown. The computer device includes a processor, memory, an input / output interface, a communication interface, a display unit, and an input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface, display unit, and input device are connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal via wired or wireless communication, and the wireless communication can be achieved via Wi-Fi, a mobile cellular network, NFC (near field communication), or other technologies. When executed by the processor, the computer program implements a method for coloring a color draft image. The display unit of the computer device is used to form a visually visible image, and can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device casing, or an external keyboard, touchpad or mouse, etc.

[0308] Those skilled in the art will understand that Figure 15 The structure shown in or 16 is merely a block diagram of a portion of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0309] In one embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps of the above method embodiment when executing the computer program.

[0310] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above method embodiment are implemented.

[0311] In one embodiment, a computer program product is provided, comprising a computer program, which implements the steps of the above method embodiment when executed by a processor.

[0312] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, storage, and display, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. The collection, use, and processing of such data must comply with the relevant laws, regulations, and standards of the relevant regions and areas. Furthermore, the user may choose not to authorize the use of such information and related data, or may refuse or conveniently refuse to receive push notifications.

[0313] In this application, when collecting and processing relevant data in actual applications, the requirements of relevant local laws and regulations should be strictly followed to obtain the informed consent or separate consent of the personal information subject, and subsequent data use and processing should be carried out within the scope of authorization of laws and regulations and the personal information subject.

[0314] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments. In particular, any reference to memory, database, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The databases involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the various embodiments provided herein may be, but are not limited to, general-purpose processors, central processing units (CPUs), graphics processors (GPUs), digital signal processors (DSPs), programmable logic devices (PLCs), and the like.

[0315] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0316] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A method for coloring a line drawing, characterized in that: The method comprises: Obtaining the target line drawing to be colored and the reference line drawing corresponding to the reference color drawing from the line drawing sequence corresponding to the target object; Taking the reference color draft image as a reference, based on the time information of the target line draft image and the reference line draft image in the line draft image sequence, coloring the target line draft image to obtain an initial color draft image corresponding to the target line draft image; Determine a first area included in the corresponding initial color draft image based on the closed area in the target line draft image, and determine a second area included in the corresponding reference color draft image based on the closed area in the reference line draft image; A target first area and a target second area having a matching relationship are determined from the first area and the second area, and color correction is performed on the target first area based on the area image of the target second area to obtain a target color draft image corresponding to the target line draft image.

2. The method according to claim 1, characterized in that The step of coloring the target line drawing based on the reference color drawing and time information of the target line drawing and the reference line drawing in the line drawing sequence to obtain an initial color drawing corresponding to the target line drawing includes: Performing feature extraction on the target line drawing to obtain target line drawing features of the target line drawing; Performing feature extraction on the reference color draft image to obtain reference color draft features of the reference color draft image; Determining a target fusion feature corresponding to the target line draft image based on the target line draft feature and the reference color draft feature; Integrating the time information of the target line drawing and the reference line drawing in the line drawing sequence into the target fusion feature to obtain a time enhancement feature corresponding to the target line drawing; The time enhancement feature is decoded to obtain an initial color draft image corresponding to the target line draft image.

3. The method according to claim 2, characterized in that The method further comprises: Performing semantic analysis on the reference color draft image to obtain reference semantic features of the reference color draft image; The determining, based on the target line draft feature and the reference color draft feature, a target fusion feature corresponding to the target line draft image includes: Based on the splicing result of the target line draft feature and the reference color draft feature, an initial fusion feature corresponding to the target line draft image is obtained; The semantic information represented by the reference semantic feature is integrated into the initial fusion feature to obtain a target fusion feature corresponding to the target line drawing.

4. The method according to claim 3, characterized in that The obtaining of the initial fusion feature corresponding to the target line drawing image based on the splicing result of the target line drawing feature and the reference color drawing feature includes: Splicing the target line draft feature and the reference color draft feature to obtain a splicing feature corresponding to the target line draft image; The splicing features are enhanced in the spatial dimension to obtain initial fusion features corresponding to the target line drawing.

5. The method according to claim 2, characterized in that The step of extracting features from the target line drawing to obtain target line features of the target line drawing includes: Performing feature extraction on the target line drawing to obtain initial line drawing features of the target line drawing; Superimposing noise on the initial line draft features to obtain target line draft features corresponding to the target line draft image; The decoding process of the time enhancement feature to obtain an initial color draft image corresponding to the target line draft image includes: The temporal enhancement features are subjected to denoising and decoding processing to obtain an initial color draft image corresponding to the target line draft image.

6. The method according to claim 2, characterized in that The step of integrating the time information of the target line drawing and the reference line drawing in the line drawing sequence into the target fusion feature to obtain the time enhancement feature corresponding to the target line drawing includes: Determining a weight matrix corresponding to the target line draft image based on the target line draft features and the reference line draft features corresponding to the reference line draft image; Taking the weight matrix as a parameter, the target fusion feature is enhanced based on the temporal attention mechanism to obtain the temporal enhancement feature corresponding to the target line drawing.

7. The method according to any one of claims 1 to 6, characterized in that The number of the target line drawing is multiple frames; the number of the reference color drawing is at least two frames; and the step of coloring the target line drawing based on the time information of the target line drawing and the reference line drawing in the line drawing sequence to obtain an initial color drawing corresponding to the target line drawing includes: For each frame of the target line drawing, determining a selected color drawing that matches the target line drawing from each of the reference line drawings; In the case where the selected color draft image matches multiple frames of target line draft images, the selected color draft image is used as a reference, and based on the time information corresponding to the multiple frames of target line draft images and the selected line draft images in the line draft image sequence, the multiple frames of target line draft images are colored to obtain multiple frames of initial color draft images corresponding to the multiple frames of target line draft images.

8. The method according to any one of claims 1 to 6, characterized in that The step of obtaining a target line drawing to be colored and a reference line drawing corresponding to the reference color drawing from a line drawing sequence corresponding to the target object includes: Obtaining, from the line drawing sequence corresponding to the target object, a target line drawing to be colored and candidate line drawings corresponding to the colored drawings; In a case where the candidate line drawing matches the target line drawing, the candidate line drawing is determined as a reference line drawing, and the color drawing corresponding to the candidate line drawing is determined as a reference color drawing.

9. The method according to any one of claims 1 to 6, characterized in that The process of determining the first area and the second area includes: Based on the colored lines contained in the line drawing, determining a closed area enclosed by the colored lines from the line drawing; the line drawing includes a target line drawing and a reference line drawing; Based on the mapping position of the closed area in the color draft image corresponding to the line draft image, the colored area included in the color draft image is determined; the colored area includes a first area in the initial color draft image corresponding to the target line draft image, and a second area in the reference color draft image corresponding to the reference line draft image.

10. The method according to any one of claims 1 to 6, characterized in that The determining, from the first area and the second area, a target first area and a target second area having a matching relationship includes: determining a reference color included in the reference color draft image according to the regional image of the second region; Based on each of the reference colors, pixel color correction is performed on each pixel point in the first area to obtain a pixel correction image corresponding to the first area; According to the pixel correction map, a target first area and a target second area having a matching relationship are determined from the first area and the second area.

11. The method according to claim 10, characterized in that The step of performing pixel color correction on each pixel point in the first area based on each reference color to obtain a pixel correction image corresponding to the first area includes: Determine the pixel color of each pixel in the first area; For each of the pixel points, determining a target color that matches the pixel color of the pixel point from the reference colors; When the pixel color of the pixel point is inconsistent with the target color, the pixel color is replaced with the target color to obtain a pixel correction image corresponding to the first area.

12. The method according to claim 11, characterized in that The determining of a target color matching the pixel color of the pixel point from each of the reference colors includes: Determining a color characteristic value of the pixel color and a reference characteristic value of each of the reference colors; For each of the reference colors, determining a color distance between the reference color and the pixel color based on a difference between a reference characteristic value of the reference color and the color characteristic value; A reference color having the smallest color distance from the pixel color is determined as a target color that matches the pixel color.

13. The method according to any one of claims 1 to 6, 11 to 12, characterized in that There are multiple second areas; and determining a target first area and a target second area having a matching relationship from the first area and the second area includes: Performing image feature extraction on the first region and the second region respectively to obtain regional image features of the first region and the second region; Determine, from each of the second regions, a candidate region whose regional image features are most similar to those of the first region; When the similarity corresponding to the candidate region satisfies a similarity condition, it is determined that the candidate region matches the first region, and a target first region and a target second region having a matching relationship are obtained.

14. The method according to claim 13, characterized in that The method further comprises: When there is no target area matching the first area in each of the second areas, clearing the color in the first area; When all the first areas have been traversed, a target color draft image corresponding to the target line draft image is obtained.

15. The method according to any one of claims 1 to 6, 11 to 12, characterized in that The method further comprises: For each color line included in the reference line drawing, extract the colors on both sides of the color line from the reference color drawing to form a color pair on both sides corresponding to the color line; For each uncolored blank area in the first area, determining a target color line enclosing the blank area from the target line drawing, and determining the color of an adjacent area of ​​the blank area from the target color drawing; From the color pairs on both sides corresponding to the target color line, query the opposite color corresponding to the color of the adjacent area, and fill the blank area with the opposite color.

16. The method according to claim 15, characterized in that The step of extracting the colors on both sides of the color line from the reference color draft image includes: Determining the color line area where the color line is located from the reference color draft image according to a mapping relationship between the reference line draft image and the reference color draft image; performing expansion processing on the color line region to obtain a color line expansion region; Colors on both sides of the color line are extracted from the color line expansion region.

17. A line drawing coloring processing device, characterized in that: The device comprises: An acquisition module is used to acquire a target line drawing to be colored and a reference line drawing corresponding to the reference color drawing from a sequence of line drawings corresponding to the target object; a coloring module, configured to color the target line drawing based on the time information of the target line drawing and the reference line drawing in the line drawing sequence, using the reference color drawing as a reference, to obtain an initial color drawing corresponding to the target line drawing; an area division module, configured to determine a first area included in the corresponding initial color draft image based on the closed area in the target line draft image, and to determine a second area included in the corresponding reference color draft image based on the closed area in the reference line draft image; The color correction module is used to determine a target first area and a target second area that have a matching relationship from the first area and the second area, and based on the area image of the target second area, perform color correction on the target first area to obtain a target color draft image corresponding to the target line draft image.

18. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 16 are implemented.

19. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 16 are implemented.

20. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 16 are implemented.

Images