Image processing method and apparatus, and computer device and computer-readable storage medium

Abstract

An image processing method, which is executed by a computer device. The method comprises: acquiring a sketch image, and extracting sketch texture features of multiple scales from the sketch image (S202); acquiring a preset noise image, and extracting noise features of multiple scales from the preset noise image (S204); determining color guidance information corresponding to the sketch image (S206); for each scale among the multiple scales, with reference to a sketch texture feature of the scale and the color guidance information, encoding a noise feature of the scale, so as to obtain image features of multiple scales (S208); and performing multi-scale decoding on the image features of multiple scales, so as to obtain a colored image, wherein the colored image has sketch textures of the sketch image and has a color indicated by the color guidance information (S210).

Description

Image processing method, device, computer equipment, and computer-readable storage medium

[0001] Related applications

[0002] This application claims priority to Chinese patent application number 202310687711X, filed on June 12, 2023, entitled “Image processing method, device, computer equipment and computer-readable storage medium,” the entire text of which is hereby incorporated by reference. Technical Field

[0003] The present application relates to the field of computer technology, and in particular to an image processing method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Art

[0004] With the development of computer technology, more and more users are using electronic devices to draw line art. Coloring these line art drawings is crucial in the drawing process, but manual coloring is often time-consuming. Therefore, technologies for automatically coloring line art have emerged.

[0005] Traditional methods for automatically coloring line drawings typically require the user to select a color and specify an image area within the line drawing, automatically filling the selected color into the image area. However, this method simply fills the image area with color, resulting in distinct color differences between image areas, resulting in visually inconsistent colors in the final image.

[0006] Summary of the Invention

[0007] According to various embodiments provided in the present application, an image processing method, apparatus, computer device, computer-readable storage medium, and computer program product are provided.

[0008] In one aspect, the present application provides an image processing method, performed by a computer device, the method comprising:

[0009] Acquire a line drawing image, and extract multi-scale line drawing texture features of the line drawing image;

[0010] Acquire a preset noise image, and extract the multi-scale noise features from the preset noise image;

[0011] Determining color guide information corresponding to the line drawing image;

[0012] For each scale of the multi-scales, referring to the line drawing texture feature and the color guide information of the scale, encoding the noise feature of the scale to obtain a multi-scale image feature; and

[0013] The multi-scale image features are multi-scale decoded to obtain a colored image, where the colored image has the line draft texture of the line draft image and has the color indicated by the color guide information.

[0014] On the other hand, the present application also provides an image processing device, comprising:

[0015] A texture feature extraction module is used to obtain a line drawing image and extract multi-scale line drawing texture features of the line drawing image;

[0016] A noise feature extraction module is used to preset a noise image and extract the multi-scale noise features from the preset noise image;

[0017] A color acquisition module, used to determine color guidance information corresponding to the line drawing image;

[0018] an encoding module, configured to, for each scale of the multi-scale image, encode the noise feature of the scale with reference to the line drawing texture feature and the color guide information of the scale to obtain a multi-scale image feature; and

[0019] A decoding module is used to perform multi-scale decoding on the multi-scale image features to obtain a colored image, wherein the colored image has the line draft texture of the line draft image and has the color indicated by the color guide information.

[0020] On the other hand, the present application further provides a computer device, which includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the steps of the method embodiments of the present application when executing the computer program.

[0021] On the other hand, the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the various method embodiments of the present application are performed.

[0022] On the other hand, the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, it performs the steps of the various method embodiments of the present application.

[0023] On the other hand, the present application also provides a method for training an image processing model, which is performed by a computer device, and the method includes:

[0024] Acquire a sample line draft image, and extract multi-scale sample line draft texture features of the sample line draft image;

[0025] Acquire a sample noise image, and extract the multi-scale sample noise features from the sample noise image;

[0026] Determining sample color guide information corresponding to the sample line drawing image;

[0027] For each scale of the multi-scales, referring to the sample line drawing texture feature and the sample color guide information of the scale, the sample noise feature of the scale is encoded to obtain a multi-scale sample image feature;

[0028] Performing multi-scale decoding on the multi-scale sample image features to obtain a predicted colored image;

[0029] Obtain a labeled colored image corresponding to the sample line drawing image, and train an initial image processing model based on the predicted colored image and the labeled colored image to obtain an image processing model, wherein the labeled colored image has the line drawing texture of the sample line drawing image and the color indicated by the sample color guide information.

[0030] On the other hand, the present application also provides a training device for an image processing model, the device comprising:

[0031] A sample texture feature extraction module is used to obtain a sample line draft image and extract multi-scale sample line draft texture features of the sample line draft image;

[0032] A sample noise feature extraction module, configured to obtain a sample noise image and extract the multi-scale sample noise features from the sample noise image;

[0033] A sample color acquisition module, configured to determine sample color guidance information corresponding to the sample line drawing image;

[0034] a sample encoding module, configured to, for each scale of the multi-scales, encode the sample noise feature of the scale with reference to the sample line drawing texture feature and the sample color guidance information of the scale, so as to obtain a multi-scale sample image feature;

[0035] A sample decoding module, configured to perform multi-scale decoding on the multi-scale sample image features to obtain a predicted colored image;

[0036] A training module is used to obtain a labeled colored image corresponding to the sample line drawing image, and to train an initial image processing model based on the predicted colored image and the labeled colored image to obtain an image processing model. The labeled colored image has the line drawing texture of the sample line drawing image and the color indicated by the sample color guide information.

[0037] On the other hand, the present application also provides a computer device, which includes a memory and a processor, wherein the memory stores computer-readable instructions, and the processor executes the steps of each method embodiment of the present application when executing the computer-readable instructions.

[0038] On the other hand, the present application also provides a computer-readable storage medium, on which computer-readable instructions are stored. When the computer-readable instructions are executed by a processor, the steps of the various method embodiments of the present application are executed.

[0039] On the other hand, the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, it performs the steps of the various method embodiments of the present application.

[0040] The details of one or more embodiments of the present application are set forth in the accompanying drawings and the description below. Other features and advantages of the present application will become apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the conventional technology, the following briefly introduces the drawings required for use in the embodiments or the conventional technology descriptions. Obviously, the drawings described below are merely embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the disclosed drawings without any creative work.

[0042] FIG1 is a diagram illustrating an application environment of an image processing method according to an embodiment;

[0043] FIG2 is a schematic flow chart of an image processing method according to an embodiment;

[0044] FIG3 is a schematic diagram of a framework of an image processing model in one embodiment;

[0045] FIG4 is a diagram illustrating the architecture of a color control unit according to one embodiment;

[0046] FIG5 is a schematic diagram of a line drawing image and a grid color block image in one embodiment;

[0047] FIG6 is a flow chart of the steps of obtaining a layer color block image in one embodiment;

[0048] FIG7 is a schematic diagram of a line drawing image and an initial layer color block image in one embodiment;

[0049] FIG8 is a schematic diagram of a layer color block image in one embodiment;

[0050] FIG9 is a flow chart of a method for training an image processing model according to an embodiment;

[0051] FIG10 is a schematic diagram illustrating the difference between a traditional diffusion model and an initial image processing model in one embodiment;

[0052] FIG11 is a schematic diagram of a network structure with an added line drawing control branch in one embodiment;

[0053] FIG12 is a schematic diagram of a colored image generated when there is no custom color palette image in one embodiment;

[0054] FIG13 is a schematic diagram of a colored image generated by combining a grid color block image in one embodiment;

[0055] FIG14 is a schematic diagram of a colored image generated by combining layer color block images in one embodiment;

[0056] FIG15 is a schematic diagram of an application scenario of an image processing method according to an embodiment;

[0057] FIG16 is a block diagram of the structure of an image processing device according to an embodiment;

[0058] FIG17 is a block diagram of a training apparatus for an image processing model according to one embodiment;

[0059] FIG18 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0060] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0061] The embodiments of the present application can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, assisted driving, etc. For example, it can be applied to the field of artificial intelligence (AI) technology, wherein artificial intelligence is the theory, method, technology and application system of using digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology of computer science, which attempts to understand the essence of intelligence and produce a new intelligent machine that can react in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that the machine has the functions of perception, reasoning and decision-making. The solution provided in the embodiments of the present application relates to an image processing method of artificial intelligence and a training method of an image processing model, which is specifically described by the following embodiments.

[0062] The image processing method provided in the embodiments of the present application can be applied in the application environment shown in FIG1 . In particular, terminal 102 communicates with server 104 via a network. A data storage system can store data that server 104 needs to process. The data storage system can be integrated with server 104, or located in the cloud or on another server. Terminal 102 and server 104 can each independently execute the image processing method provided in the embodiments of the present application. Terminal 102 and server 104 can also be used together to execute the image processing method provided in the embodiments of the present application. When terminal 102 and server 104 are used together to execute the image processing method provided in the embodiments of the present application, terminal 102 obtains a line drawing image and sends it to server 104. Server 104 extracts multi-scale line drawing texture features from the line drawing image. Server 104 obtains a preset noise image and extracts multi-scale noise features from the preset noise image. Terminal 102 determines color guidance information corresponding to the line drawing image and sends it to server 104. For each multi-scale scale, server 104 encodes the noise features of the scale by referring to the line drawing texture features and color guidance information of the scale to obtain multi-scale image features. Server 104 performs multi-scale decoding on the multi-scale image features to obtain a colored image. The colored image has the line drawing texture of the line drawing image and the color indicated by the color guidance information. Server 104 returns the colored image to terminal 102. Terminal 102 may be, but is not limited to, various desktop computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices may include smart speakers, smart TVs, smart air conditioners, smart car devices, etc. Portable wearable devices may include smart watches, smart bracelets, head-mounted devices, etc. Server 104 may be implemented as a standalone server or a server cluster consisting of multiple servers.

[0063] The image processing model training method provided in the embodiments of the present application can be applied in the application environment shown in FIG1 . Terminal 102 and server 104 can each independently perform the image processing model training method provided in the embodiments of the present application. Terminal 102 and server 104 can also be used together to perform the image processing model training method provided in the embodiments of the present application. When terminal 102 and server 104 are used together to perform the image processing model training method provided in the embodiments of the present application, terminal 102 obtains a sample line drawing image and a sample noise image from server 104, and extracts multi-scale sample line drawing texture features from the sample line drawing image. Terminal 102 extracts multi-scale sample noise features from the sample noise image. Terminal 102 obtains sample color guidance information corresponding to the sample line drawing image from server 104. For each multi-scale scale, terminal 102 encodes the sample noise features for that scale by referring to the sample line drawing texture features and sample color guidance information for that scale, thereby obtaining multi-scale sample image features. Terminal 102 performs multi-scale decoding on the multi-scale sample image features to obtain a predicted color image. Terminal 102 obtains a labeled colored image corresponding to the sample line drawing image and trains an initial image processing model based on the predicted colored image and the labeled colored image to obtain an image processing model. The labeled colored image has the line drawing texture of the sample line drawing image and the color indicated by the sample color guidance information. The image processing model can be deployed on terminal 102 or server 104.

[0064] In one embodiment, as shown in FIG2 , an image processing method is provided. The method is described by taking the computer device in FIG1 (the computer device may be a terminal or a server in FIG1 ) as an example, and includes the following steps:

[0065] Step S202 : obtaining a line drawing image and extracting multi-scale line drawing texture features of the line drawing image.

[0066] A line drawing image is an image composed of lines and is not colored. Line drawing texture features refer to features in a line drawing image that can represent the texture structure.

[0067] In this embodiment, a computer device obtains a line drawing image to be colored, performs feature extraction on the line drawing image, and obtains multi-scale line drawing texture features.

[0068] In this embodiment, feature extraction is performed on the line drawing image through an image processing model to obtain multi-scale line drawing texture features.

[0069] In this embodiment, the image processing model includes a line drawing control unit and a color control unit. The computer device inputs the line drawing image into the line drawing control unit, and the line drawing control unit extracts features of the line drawing image to obtain multi-scale line drawing texture features.

[0070] Step S204: obtaining a preset noise image, and extracting multi-scale noise features from the preset noise image.

[0071] The preset noise image is a pre-set pure noise image, such as a noise image formed by Gaussian noise, but not limited thereto. Multi-scale refers to multiple different scales, such as 64×64, 32×32, 16×16, 8×8, etc.

[0072] In this embodiment, the computer device obtains a preset noise image, performs feature extraction on the preset noise image, and obtains multi-scale noise features.

[0073] In this embodiment, the preset noise image may be a noise image pre-stored in an image processing model. The image processing model performs feature extraction on the preset noise image to obtain multi-scale noise features. Furthermore, the number of scales corresponding to the multi-scale line drawing texture features is the same as the number of scales corresponding to the multi-scale noise features. For example, if line drawing texture features at five scales are extracted from the line drawing image, and noise features at five scales are extracted from the preset noise image, the number of scales is five.

[0074] In one embodiment, a computer device obtains noise features extracted from a preset noise image; for each scale of the multi-scale, the noise features are encoded at the scale with reference to the line drawing texture features and color guidance information of the scale to obtain multi-scale image features.

[0075] Furthermore, noise features extracted from a preset noise image are obtained, including: obtaining multi-scale noise features extracted from the preset noise image; for each scale of the multi-scale, referring to the line draft texture features and color guidance information of the targeted scale, encoding the noise features at the targeted scale to obtain multi-scale image features, including: for each scale of the multi-scale, referring to the line draft texture features and color guidance information of the targeted scale, encoding the noise features at the targeted scale to obtain multi-scale image features.

[0076] Step S206: Determine the color guide information corresponding to the line drawing image.

[0077] Color guidance information is information used to guide the coloring of a line drawing image. This color guidance information is used to define the color of the line drawing image and can specifically be user-defined color information for the line drawing image. This user-defined color information for the line drawing image can be a customized color for at least one grid area within the line drawing image. This user-defined color information for the line drawing image can include a customized color palette image formed by custom color fills, or text describing the color of the line drawing image.

[0078] In this embodiment, the computer device obtains color guidance information customized by the user for the line drawing image, or randomly obtains a color random seed from a plurality of preset color random seeds as the color guidance information. The color random seed may include one or more colors, and the color random seed may also be set to correspond to different content in the image, such as green for plants and blue for the sky and sea, but is not limited to this.

[0079] In this embodiment, the computer device may obtain color description text as color guidance information. The color description text is used to describe the color of the line drawing image. Specifically, it may describe the color of at least a portion of the content in the line drawing image. In other words, the color description text may be text customized by the user to define the color of the line drawing image. For example, if the line drawing image depicts a cat and a dog, and the color description text describes the cat as white and the dog as yellow, the subsequently generated colored image will include a white cat and a yellow dog.

[0080] In this embodiment, the computer device may obtain a custom color palette image corresponding to the line drawing image as color guide information. The custom color palette image is an image obtained by a user customizing the color of the line drawing image.

[0081] Step S208 : For each scale of the multi-scale, refer to the line drawing texture features and color guidance information of the scale, encode the noise features of the scale, and obtain multi-scale image features.

[0082] The image features are formed by fusing the line drawing texture features and the colors indicated by the color guidance information. The image features at each scale in the multi-scale image features are fusing the image features at the previous scale.

[0083] In this embodiment, the computer device determines the color indicated by the color guidance information. For each of the multiple scales, the computer device encodes the noise features of the scale by referring to the line drawing texture features of the scale and the color indicated by the color guidance information to obtain the multi-scale image features.

[0084] Furthermore, for each scale of the multi-scale image, the noise features of the scale are encoded with reference to the line drawing texture features and color guidance information of the scale, thereby obtaining multi-scale image features, including:

[0085] For each scale of the multiple scales, when the targeted scale is the first scale among the multiple scales, referencing the line drawing texture features and color guidance information of the targeted scale, the noise features of the targeted scale are encoded to obtain the image features of the targeted scale;

[0086] When the targeted scale is not the first scale among the multiple scales, the noise features of the targeted scale are encoded with reference to the line drawing texture features, color guidance information, and image features of the previous scale of the targeted scale to obtain the image features of the targeted scale.

[0087] In this embodiment, for each scale of the multiple scales, when the targeted scale is the first scale among the multiple scales, the line drawing texture features of the targeted scale, the color features of the color guide information, and the noise features of the targeted scale are fused to obtain the image features of the targeted scale;

[0088] When the targeted scale is not the first scale among the multiple scales, the line drawing texture features of the targeted scale, the color features of the color guide information, the image features of the previous scale of the targeted scale, and the noise features of the targeted scale are fused to obtain the image features of the targeted scale.

[0089] In this embodiment, for each scale of the multi-scale, the noise features of the targeted scale are encoded with reference to the line draft texture features and color guidance information of the targeted scale to obtain multi-scale image features, including: extracting color features from the color guidance information; for each scale of the multi-scale, the color features, the line draft texture features of the targeted scale, and the noise features of the targeted scale are fused to obtain the image features of the targeted scale, thereby obtaining multi-scale image features.

[0090] In step S210 , multi-scale decoding is performed on the multi-scale image features to obtain a colored image. The colored image has the line draft texture of the line draft image and has the color indicated by the color guide information.

[0091] In this embodiment, the computer device performs multi-scale decoding on the multi-scale image features, and after completing the decoding of the last scale, a colored image is obtained. The colored image has the line drawing texture of the line drawing image and has the color indicated by the color guide information.

[0092] In this embodiment, for each scale in the multi-scale decoding, the decoded features of the previous scale are decoded with reference to the image features of the targeted scale to obtain the decoded features of the targeted scale. The decoding of the next scale is then started, and the obtained decoded features of the targeted scale are used as the decoded features of the previous scale in the decoding of the next scale. The process then returns to the step of decoding the decoded features of the previous scale and continues until the last scale is decoded to obtain a colored image.

[0093] In this embodiment, the encoding order of the multiple scales in encoding is opposite to the decoding order of the multiple scales in decoding.

[0094] In this embodiment, a line drawing image is acquired, and multi-scale line drawing texture features are extracted based on the line drawing image. The extracted large-scale line drawing texture features contain more low-level semantic features, while the small-scale line drawing texture features contain more high-level semantic features. Multi-scale noise features are extracted from a preset noise image, and multiple feature extractions at the same scale are performed on the line drawing image and the preset noise image. The multi-scale line drawing texture features and color guidance information corresponding to the line drawing image are used as guidance information for noise feature encoding. For each multi-scale scale, the noise features at that scale are encoded with reference to the line drawing texture features and color guidance information of the targeted scale. This enables the line drawing texture features and the colors indicated by the color guidance information to be integrated into the encoding, resulting in image features having both the line drawing texture features and the indicated colors. Furthermore, the integration of line drawing texture and color helps generate an image with color variations that more closely match the direction of the line drawing texture.

[0095] This multi-scale encoding process can generate image features that fuse line art texture features at different scales with the indicated color. This allows for multi-scale decoding of these features, focusing on textures at different levels during the decoding process. This allows for the generation of colored images with clearer line art textures and finer edge transition details. Furthermore, this allows for a more harmonious blend of color and line art texture in the colored image, improving image quality.

[0096] In one embodiment, the color guidance information includes a custom palette image, which is used to customize the color of the line drawing image. For each scale of the multi-scale image, the noise features of the scale are encoded with reference to the line drawing texture features and the color guidance information of the scale to obtain the multi-scale image features, including:

[0097] Extract the palette features of the custom palette image; for each scale in the multi-scale image, refer to the line drawing texture features and palette features of the scale, encode the noise features of the scale, and obtain the multi-scale image features.

[0098] The custom palette image is a palette image formed by the user customizing the color of the line drawing image. That is, the custom palette image is used for the user to customize the color of the line drawing image.

[0099] In this embodiment, the user can customize the color of the line drawing image to obtain a customized color palette image corresponding to the line drawing image. The computer device performs feature extraction on the customized color palette image to obtain color palette features corresponding to the customized color palette image.

[0100] For each scale, the computer fuses the line drawing texture features and color plate features at that scale, along with the noise features at that scale, to obtain image features at that scale. Similar processing can be used to obtain image features at multiple scales.

[0101] In this embodiment, for each scale of the multiple scales, when the targeted scale is the first scale among the multiple scales, the line drawing texture features and color plate features of the targeted scale and the noise features at the targeted scale are fused to obtain the image features of the targeted scale;

[0102] When the targeted scale is not the first scale among multiple scales, the line drawing texture features, color plate features, image features of the previous scale of the targeted scale, and noise features of the targeted scale are fused to obtain the image features of the targeted scale, thereby obtaining multi-scale image features.

[0103] In this embodiment, the custom palette image is a palette image of the color of the user-defined line drawing image. By performing feature extraction on the user-defined palette image, palette features corresponding to the custom palette image are obtained. For each of the multiple scales, the noise features at that scale are encoded with reference to the line drawing texture features and palette features of the targeted scale. The noise features can be encoded into image features that combine the line drawing texture features and palette features. This allows the line drawing image and corresponding color information to be used as guidance information, enabling the generation of an image with the user's desired line drawing texture and customized color from the preset noise image. Furthermore, by performing encoding at multiple scales, the multi-scale image features obtained can contain both shallow and deep information about the line drawing texture and color, resulting in a decoded image with clearer and more delicate texture and more harmonious colors.

[0104] In one embodiment, extracting palette features of a custom palette image includes: extracting palette features of at least two different scales from the custom palette image, wherein the number of scales corresponding to the at least two different scales is less than the number of scales corresponding to the multiple scales;

[0105] For each scale in the multi-scale image, the noise features of the scale are encoded with reference to the line drawing texture features and color plate features of the scale, and the multi-scale image features are obtained, including:

[0106] For each scale of the multi-scale, when the color plate feature of the scale exists, the noise feature at the scale is encoded with reference to the line draft texture feature and color plate feature of the scale to obtain the image feature of the scale; when the color plate feature of the scale does not exist, the noise feature of the scale is encoded with reference to the line draft texture feature of the scale to obtain the image feature of the scale.

[0107] The number of scales represented by the at least two different scales is less than the number of scales represented by the multi-scale representation. For example, the at least two different scales may specifically include three scales, and the multi-scale representation may specifically include five scales. The number of scales corresponding to the multi-scale line drawing texture features is the same as the number of scales corresponding to the multi-scale noise features.

[0108] In this embodiment, the computer device performs feature extraction on the custom palette image to obtain palette features at at least two different scales corresponding to the custom palette image. Furthermore, the computer device performs feature extraction on the custom palette image at a first scale to obtain palette features at the first scale. Feature processing is performed on the palette features at the first scale to obtain feature processing at the next scale. Similar processing can be used to obtain palette features at different scales.

[0109] For image features at multiple scales, noise features at multiple scales, and color plate features at different scales, for each of the multiple scales, the computer device determines whether there is a color plate feature for the targeted scale. When there is a color plate feature for the targeted scale, the noise feature for the targeted scale is encoded with reference to the line drawing texture feature and color plate feature for the targeted scale to obtain the image feature for the targeted scale, thereby obtaining an image feature that integrates the color plate feature and the line drawing texture feature.

[0110] When there is no color plate feature of the targeted scale, the noise feature of the targeted scale is encoded with reference to the line drawing texture feature of the targeted scale to obtain the image feature of the targeted scale, thereby obtaining the image feature integrated with the line drawing texture feature.

[0111] In this embodiment, color and texture belong to different levels of semantics: color belongs to a lower level of semantics, while texture belongs to a higher level of semantic information. Therefore, features of fewer scales can be extracted for color, while features of more scales can be extracted for the line art texture of the line art image. For each of the multiple scales, when palette features exist for that scale, the noise features for that scale are encoded with reference to the line art texture features and palette features for that scale. These features serve as guidance information, indicating that the line art texture features and palette features should be fused during encoding based on the noise features, thereby obtaining image features for that scale that incorporate the line art texture features and palette features at that scale. When palette features do not exist for that scale, the noise features for that scale are encoded with reference to the line art texture features for that scale, using the line art texture features as guidance information, indicating that the line art texture features should be fused during encoding based on the noise features, thereby obtaining image features for that scale that incorporate the line art texture features at that scale. This time, the image features that integrate different levels of line drawing texture and color can reconstruct the image more accurately, making the line drawing texture in the reconstructed colored image clearer and more delicate, and the color more harmonious and natural, thereby improving the image quality.

[0112] In one embodiment, the image processing method is performed by an image processing model, which includes a line drawing control unit, a color control unit, and an image generation unit. The framework of the image processing model is shown in Figure 3. The line drawing control unit extracts multi-scale line drawing texture features from the line drawing image, and the color control unit extracts color plate features of different scales from the custom color plate image. The image generation unit extracts multi-scale noise features of the preset noise image, inputs the multi-scale line drawing texture features and color plate features of different scales into the image generation unit, and encodes them with the noise features of the corresponding scales to obtain multi-scale image features. The multi-scale image features are decoded in the image generation unit to obtain a colored image.

[0113] FIG4 is a diagram illustrating the architecture of a color control unit in one embodiment. The color control unit includes multiple residual blocks, which can be used to extract color palette features of different scales for a custom color palette image. It is understood that the number of residual blocks in the color control unit can be set as needed.

[0114] During the training of the image processing model, the input of the color control unit also includes the temporal feature time embedding.

[0115] In one embodiment, the custom color palette image includes a grid color block image corresponding to the line drawing image, and the method further includes:

[0116] Obtain a color palette template divided according to grid color blocks, and divide the line drawing image into grids according to the grid color blocks divided by the color palette template to obtain grid areas in the line drawing image. Each grid color block in the line drawing image corresponds one-to-one to each grid area in the line drawing image. Obtain a grid color block image corresponding to the color palette template. The grid color block image is obtained by filling each grid color block in the color palette template with a custom color. The grid color block is used to define the color of the corresponding grid area in the line drawing image.

[0117] The colored image has the line draft texture of the line draft image, and each grid area in the colored image has the color filled by the corresponding grid color block in the grid color block image.

[0118] In this embodiment, the computer device obtains a color plate template, and the color plate image is divided according to grid color blocks. The size of the color plate template is the same as the size of the line draft image. The computer device divides the line draft image into grids according to the grid color blocks divided by the color plate template, and obtains each grid area in the line draft image, so that each grid area represents an image content in the line draft image, and each grid color block corresponds to a grid area in the line draft image. That is, each grid color block in the line draft image corresponds one-to-one to each grid area in the line draft image. The line draft image and the corresponding grid color block image are shown in Figure 5, which also shows the correspondence between each grid area in the line draft image and each grid color block in the grid color block image.

[0119] Users can fill color in each grid color block of the color palette template to customize the color of the grid area corresponding to the grid color block, so that the image content represented by the grid area in the subsequently generated colored image can be displayed as the color filled in the grid color block.

[0120] The user fills each grid block in the palette template with a custom color, forming a grid block image. A computer device acquires this grid block image and performs feature extraction on it, obtaining grid block features corresponding to the grid block image. These grid block features serve as palette features. For each scale in the multi-scale image, the noise features at that scale are encoded by referring to the line drawing texture features and palette features at that scale, yielding multi-scale image features.

[0121] The computer device performs multi-scale decoding on the multi-scale image features to obtain a colored image, wherein the colored image has the line draft texture of the line draft image, and each grid area in the colored image has the color filled by the corresponding grid color block in the grid color block image.

[0122] In this embodiment, when the size of the line drawing image is different from the size of the color plate template, the line drawing image and the color plate template are adjusted to the same size. Under the same size, the line drawing image is divided according to the grid divided by the line drawing image to obtain each grid area in the line drawing image.

[0123] The resizing may include adjusting the size of the line drawing image to be the same as the size of the color palette template, adjusting the size of the color palette template to be the same as the size of the line drawing image, or adjusting the sizes of the line drawing image and the color palette template simultaneously.

[0124] In this embodiment, a color palette template divided into grid color blocks is obtained, and the line drawing image is grid-divided according to the grid color blocks divided by the color palette template. This results in grid areas representing image regions in the line drawing image. This ensures a one-to-one correspondence between each grid area of ​​the line drawing image and each grid color block of the color palette template. This allows the user to define the color of each image area in the line drawing image by filling the grid color blocks with color, effectively enabling user customization of the color of the line drawing image. Furthermore, the user-defined color is subsequently used as guidance information in feature encoding and decoding, ensuring that the resulting colored image not only possesses the line drawing texture of the line drawing image, but also that each grid area in the colored image also has the color that the user filled in the corresponding grid color block, effectively enabling the generation of a line drawing image with a customized color.

[0125] In one embodiment, the custom color palette image includes a layer color block image corresponding to the line drawing image, and the method further includes:

[0126] The line drawing image is randomly filled with colors to obtain a colored image; the colored image is segmented into layer color blocks according to each image area in the colored image to obtain a layer color block image corresponding to the line drawing image, the layer color block image including a layer color block corresponding to each image area; wherein the colored image has the line drawing texture of the line drawing image, and each image area in the colored image has the color indicated by the corresponding layer color block in the layer color block image.

[0127] In this embodiment, the computer device can automatically fill the line drawing image with random colors to obtain a colored image. The colored image includes at least two colors. Furthermore, the user can customize the random color filling of the line drawing image to obtain a colored image.

[0128] The computer device determines each image region in the colored image and performs layer color block segmentation on the colored image according to each image region in the colored image to obtain a layer color block image corresponding to the line drawing image. The layer color block image includes multiple layer color blocks, and the multiple layer color blocks in the layer color block image correspond one-to-one to the multiple image regions in the colored image.

[0129] The layer color blocks represent the colors of the corresponding image areas. The colors of adjacent layer color blocks in each layer color block are different.

[0130] The computer device can extract layer color block features from the layer color block image. These layer color block features can be used as color plate features. For each of the multiple scales, the noise features of the targeted scale are encoded with reference to the line drawing texture features and color plate features of the targeted scale to obtain multi-scale image features.

[0131] In this embodiment, random color filling is performed on the line drawing image to obtain a colored image. This allows for layer color block segmentation of the colored image according to each image region within the colored image, accurately segmenting each layer color block within the colored image. Each layer color block forms a layer color block image corresponding to the colored image. Extracting features from the layer color block image yields a palette feature, which is then used as color guidance information for feature encoding and decoding. This ensures that the resulting colored image not only possesses the line drawing texture of the line drawing image, but also that each image region within the colored image has the color defined by the corresponding layer color block, enabling rapid coloring of the line drawing image.

[0132] In one embodiment, the colored image is segmented into layer color blocks according to each image area in the colored image to obtain a layer color block image corresponding to the line drawing image, including:

[0133] According to each image area in the colored image, the colored image is segmented into layer color blocks to obtain an initial layer color block image, which includes layer color blocks corresponding to each image area; based on the customized color adjustment of at least one layer color block in the initial layer color block image, the layer color block image is obtained.

[0134] In this embodiment, the computer device determines each image region in the colored image and performs layer color block segmentation on the colored image according to each image region in the colored image to obtain an initial layer color block image corresponding to the line drawing image. The initial layer color block image includes multiple layer color blocks, and the multiple layer color blocks in the initial layer color block image correspond one-to-one to the multiple image regions in the colored image.

[0135] The user can adjust the color of each layer color block in the initial layer color block image to a desired color. The computer device obtains the adjusted layer color block image based on the user's customized color adjustment.

[0136] In this embodiment, the layer color blocks represent the colors of the corresponding image areas, and the colors of adjacent layer color blocks in each layer color block are different.

[0137] In this embodiment, a colored image is segmented into layers of color blocks, accurately segmenting each layer of color blocks within the colored image. Each layer of color blocks forms an initial layer of color block image corresponding to the colored image. This initial layer of color block image includes the layer of color blocks corresponding to each image region in the line drawing image. The user can adjust the color of each layer of color block to redefine the desired color, thereby obtaining a user-defined layer of color block image. Furthermore, the user-defined color is subsequently used as guidance information in feature encoding and decoding, resulting in a final colored image that not only possesses the line drawing texture of the line drawing image, but also incorporates the color specified by the user in the corresponding layer of color blocks within each image region, effectively achieving the generation of a line drawing image with a customized color.

[0138] In one embodiment, as shown in FIG6 , the custom color palette image includes a layer color block image corresponding to the line drawing image, and the method further includes:

[0139] Step S602: Fill the line drawing image with random colors to obtain a colored image.

[0140] In this embodiment, the computer device can automatically fill the line drawing image with random colors to obtain a colored image.

[0141] The colored image includes at least two colors.

[0142] Step S604 , performing layer color block segmentation on the colored image according to each image area in the colored image, and obtaining an initial layer color block image corresponding to the line drawing image, wherein the initial layer color block image includes the layer color blocks corresponding to each image area in the line drawing image.

[0143] In this embodiment, the computer device determines each image region in the colored image, and performs layer color block segmentation on the colored image according to each image region in the colored image to obtain an initial layer color block image corresponding to the colored image. The initial layer color block image represents the color of each image region in the colored image.

[0144] The initial layer color block image includes layer color blocks corresponding to each image area in the line drawing image, and the layer color blocks represent the color of the corresponding image area. The colors of adjacent layer color blocks in each layer color block are different.

[0145] The initial layer color block image corresponding to the colored image is the initial layer color block image corresponding to the line drawing image. The initial layer color block image represents the color of each image area in the colored image, that is, the initial layer color block image includes the layer color block corresponding to each image area in the line drawing image.

[0146] Step S606 : obtaining a layer color block image based on the customized color adjustment of at least one layer color block in the initial layer color block image.

[0147] The colored image has the line draft texture of the line draft image, and each image area in the colored image has the color indicated by the corresponding layer color block in the layer color block image.

[0148] In this embodiment, the user can adjust the color of each layer color block in the initial layer color block image to a desired color. The computer device obtains the adjusted layer color block image based on the user's customized color adjustment.

[0149] For example, image segmentation is performed on colored image A to obtain an initial layer color block image B. This initial layer color block image B includes layer color blocks 1, 2, and 3, each of which represents the color of a corresponding image region of colored image A. The user adjusts the color of at least one of layer color blocks 1, 2, and 3 to obtain color-adjusted layer color blocks 1, 2, and 3. The color-adjusted layer color blocks 1, 2, and 3 form layer color block image C.

[0150] As shown in FIG7 , a line drawing image and an initial layer color block image corresponding to the line drawing image are provided. After the user adjusts the color of the layer color block in the initial layer color block image, a layer color block image as shown in FIG8 is obtained.

[0151] The computer extracts features from the layer color block image to obtain corresponding layer color block features. These layer color block features serve as palette features. For each scale in the multi-scale image, the noise features at that scale are encoded by referring to the line drawing texture features and palette features at that scale to obtain multi-scale image features.

[0152] In this embodiment, the line drawing image is randomly filled with colors to obtain a colored image, so that the colored image can be segmented into layer color blocks according to each image area in the colored image, and each layer color block in the colored image can be accurately segmented. Each layer color block forms an initial layer color block image corresponding to the colored image. The initial layer color block image includes the layer color blocks corresponding to each image area in the line drawing image. The user can adjust the color of each layer color block to redefine the color they want, thereby obtaining a user-defined layer color block image. In addition, the user-defined color is subsequently used as guidance information in feature encoding and decoding, so that the final generated colored image not only has the line drawing texture of the line drawing image, but also each image area in the colored image has the color filled by the user in the corresponding layer color block, effectively realizing the generation of a line drawing image with a user-defined color.

[0153] In one embodiment, the method further comprises:

[0154] Obtain image description text for the line drawing image; extract image description features from the image description text;

[0155] For each scale in the multi-scale image, the noise features of the scale are encoded with reference to the line texture features and color guidance information of the scale, and the multi-scale image features are obtained, including:

[0156] For each scale in the multi-scale approach, the noise features of the scale are encoded with reference to the line drawing texture features, color guidance information, and image description features of the scale to obtain the multi-scale image features.

[0157] Image description text is used to describe the features of a line drawing image. For example, if a line drawing image contains a sun but its lines are blurry, the image description text "There is a sun in the upper left corner of the image" can describe the sun and its position within the line drawing image. Image description text can provide descriptive information for unclear line drawings, supplementing at least part of the line drawing image and making the subsequently generated colored image clearer.

[0158] The image description text can also define the content in the line drawing image. For example, there is no sun in the line drawing image, and the image description text is "there is a sun in the upper left corner of the image", so that the sun is present in the upper left corner of the subsequently generated colored image.

[0159] In this embodiment, the computer device obtains image description text for the line drawing image, performs feature extraction on the image description text, and obtains image description features corresponding to the image description text. For each of the multiple scales, the computer device references the line drawing texture features, color guidance information, and image description features at that scale to encode the noise features at that scale. This is then combined with the line drawing texture features, the colors indicated by the color guidance information, and the image description features to obtain image features that combine these features, thereby obtaining multi-scale image features.

[0160] In this embodiment, image description text is obtained for a line drawing image, and the image description text is used to describe the features of the line drawing image as supplementary information for the line drawing image. Image description features are extracted from the image description text, and for each of the multiple scales, the noise features of the targeted scale are encoded with reference to the line drawing texture features, color guidance information, and image description features of the targeted scale. This allows the encoded image features to incorporate the line drawing texture and the color indicated by the line drawing texture features at the corresponding scale, and also to include feature information described by the image description features. This allows the encoding to capture image content not present in the line drawing image, or to convert previously unclear features in the line drawing image into clear features by combining the image description features. Furthermore, encoding at multiple scales can yield multi-scale image features. Image features at larger scales contain more deep image information, while image features at smaller scales contain more shallow image information, thereby enabling the acquisition of image information at different levels.

[0161] In one embodiment, performing multi-scale decoding on multi-scale image features to obtain a colored image includes:

[0162] For each scale in the multi-scale decoding, when the targeted scale is the first scale in the multi-scale decoding, the image features of the last scale obtained by encoding are decoded with reference to the image features of the targeted scale to obtain the decoded features of the targeted scale;

[0163] When the targeted scale is not the first scale or the last scale in the multi-scale decoding, the decoded features of the previous scale of the targeted scale are decoded with reference to the image features of the targeted scale to obtain the decoded features of the targeted scale;

[0164] When the targeted scale is the last scale, the decoded features of the previous scale of the last scale are decoded with reference to the image features of the targeted scale to obtain a colored image.

[0165] In this embodiment, for each scale in the multi-scale decoding, when the targeted scale is the first scale in the multi-scale decoding, the computer device decodes the image features of the last scale obtained by encoding with reference to the image features of the targeted scale to obtain decoded features of the targeted scale. When the targeted scale is not the first or last scale in the multi-scale decoding, the computer device decodes the decoded features of the previous scale with reference to the image features of the targeted scale to obtain decoded features of the targeted scale. When the targeted scale is the last scale, the computer device decodes the decoded features of the previous scale with reference to the image features of the targeted scale to obtain a colored image.

[0166] In this embodiment, during the multi-scale decoding process, the image features at each scale are used as a reference for decoding at that scale. This allows the encoded features to be used as a reference for decoding the decoded features, effectively avoiding decoding errors and improving decoding accuracy. Furthermore, during decoding at different scales, the features decoded at each scale are the same as those obtained at the previous scale. This allows for progressive decoding of the decoded features to gradually reconstruct the image's texture details and color information, accurately reconstructing a colored image with the texture and color of the line drawing.

[0167] In one embodiment, the method further comprises:

[0168] A sample line draft image is obtained, and multi-scale sample line draft texture features of the sample line draft image are extracted; a sample noise image is obtained, and multi-scale sample noise features are extracted from the sample noise image; sample color guidance information corresponding to the sample line draft image is determined; for each scale of the multi-scale, the sample noise features of the scale are encoded with reference to the sample line draft texture features and the sample color guidance information of the scale to obtain multi-scale sample image features; the multi-scale sample image features are multi-scale decoded to obtain a predicted colored image, and a label colored image corresponding to the sample line draft image is obtained. An initial image processing model is trained based on the predicted colored image and the label colored image to obtain an image processing model, wherein the label colored image has the line draft texture of the sample line draft image and has the color indicated by the sample color guidance information.

[0169] In this embodiment, a sample line drawing image is obtained and multi-scale sample line drawing texture features are extracted from the sample line drawing image. The multi-scale includes both large and small scales. The large-scale sample line drawing texture features contain more low-level semantic features, while the small-scale sample line drawing texture features contain more high-level semantic features. Multi-scale sample noise features are extracted from a sample noise image, and multiple feature extractions are performed on the line drawing image and a preset noise image at the same scale. The multi-scale sample line drawing texture features and sample color guidance information corresponding to the sample line drawing image are used as guidance information for encoding the sample noise features. For each scale in the multi-scale, the sample noise features at that scale are encoded with reference to the sample line drawing texture features and sample color guidance information at that scale. This encoding generates a sample image feature that combines the sample line drawing texture features with the colors indicated by the sample color guidance information. Furthermore, the fusion of the sample line drawing texture and color helps ensure that color changes in the subsequently generated image align with the direction of the line drawing texture.

[0170] The encoding is multi-scale and can obtain sample image features that fuse sample line drawing texture features at different scales and the indicated color. The multi-scale sample image features are then decoded at multiple scales to obtain a predicted colored image, and a labeled colored image corresponding to the sample line drawing image is obtained. The initial image processing model is then trained based on the difference between the predicted colored image and the labeled colored image to adjust the model parameters so that the predicted colored image generated by the trained image processing model has the line drawing texture of the sample line drawing image and the color indicated by the sample color guidance information. Furthermore, the trained image processing model can generate colored images with clearer line drawing textures and finer edge transition details, and the combination of color and line drawing texture in the colored image is more harmonious, thereby improving the image quality of the colored image.

[0171] The present application also provides a training method for an image processing model, as shown in FIG9 , which is illustrated by applying the method to the computer device in FIG1 (the computer device may be a terminal or a server in FIG1 ) as an example, and includes the following steps:

[0172] Step S902 : obtaining a sample line drawing image, and extracting multi-scale sample line drawing texture features of the sample line drawing image.

[0173] The sample line drawing image is an image composed of lines and is not colored. The sample line drawing texture feature refers to a feature in the sample line drawing image that can represent the texture structure.

[0174] In this embodiment, a computer device obtains a sample line drawing image to be colored, performs feature extraction on the sample line drawing image, and obtains multi-scale texture features of the sample line drawing.

[0175] In this embodiment, feature extraction is performed on the sample line draft image using an initial image processing model to obtain multi-scale texture features of the sample line draft.

[0176] Step S904: Acquire a sample noise image, and extract multi-scale sample noise features from the sample noise image.

[0177] The sample noise image is a preset pure noise image, for example, a noise image formed by Gaussian noise, but not limited thereto.

[0178] In this embodiment, the computer device obtains a sample noise image, performs feature extraction on the sample noise image, and obtains corresponding sample noise features. Furthermore, multi-scale sample noise features can be obtained through feature extraction.

[0179] In this embodiment, the sample noise image is a noise image pre-stored in the initial image processing model. The image processing model performs feature extraction on the sample noise image to obtain multi-scale sample noise features. Furthermore, the multi-scale sample line drawing texture features correspond to the same number of scales as the multi-scale sample noise features, for example, both including five scales.

[0180] In one embodiment, a computer device obtains sample noise features extracted from a sample noise image; for each scale of the multi-scale, the sample noise features are encoded at the scale with reference to the sample line drawing texture features and sample color guidance information of the scale, thereby obtaining multi-scale sample image features.

[0181] Furthermore, obtaining sample noise features extracted from the sample noise image includes: obtaining multi-scale sample noise features extracted from the sample noise image;

[0182] For each scale of the multi-scale image, the sample noise features are encoded for the scale with reference to the sample line draft texture features and sample color guidance information of the scale, so as to obtain the multi-scale sample image features, including: for each scale of the multi-scale image, the sample noise features are encoded for the scale with reference to the sample line draft texture features and sample color guidance information of the scale, so as to obtain the multi-scale sample image features.

[0183] Step S906: Determine sample color guide information corresponding to the sample line drawing image.

[0184] The sample color guidance information is used to guide the coloring of the sample line drawing image. This sample color guidance information is used to define the color of the sample line drawing image. Specifically, it may be user-defined color information for the sample line drawing image. The user-defined color information for the sample line drawing image may be a customized color for at least one area within the sample line drawing image. The user-defined color information for the sample line drawing image may be represented by a sample customized color palette image filled with customized colors, or by text describing the sample colors of the sample line drawing image.

[0185] In this embodiment, the computer device obtains sample color guidance information customized by the user for the sample line drawing image, or randomly obtains a sample color random seed from a plurality of preset sample color random seeds as the sample color guidance information. The sample color random seed may include one or more colors, and the sample color random seed may also be set to correspond to different content in the image, such as green for plants and blue for the sky and sea, but is not limited to this.

[0186] In this embodiment, the computer device may obtain sample color description text as sample color guidance information. The sample color description text is used to describe the color of the line drawing image, and specifically may describe the color of at least a portion of the content in the sample line drawing image.

[0187] In this embodiment, the computer device may obtain a sample custom color palette image corresponding to the sample line drawing image as sample color guide information. The sample custom color palette image is an image obtained by a user customizing the color of the sample line drawing image.

[0188] Step S908 : For each scale of the multi-scale, refer to the sample line drawing texture feature and sample color guidance information of the scale, encode the sample noise feature of the scale, and obtain the multi-scale sample image feature.

[0189] The sample image feature is formed by fusing the sample line texture feature and the color indicated by the sample color guide information. The sample image feature at each scale in the multi-scale sample image feature is a fusion of the sample image feature at the previous scale.

[0190] In this embodiment, the computer device determines the color indicated by the sample color guidance information. For each of the multiple scales, the computer device encodes the sample noise characteristics at that scale by referring to the sample line drawing texture characteristics at that scale and the color indicated by the sample color guidance information, thereby obtaining the multi-scale sample image characteristics.

[0191] In this embodiment, the sample color guidance information includes a sample custom color palette image, which is used to customize the color of the sample line drawing image. For each scale of the multi-scale image, the sample noise features of the scale are encoded with reference to the sample line drawing texture features and the sample color guidance information of the scale, to obtain the multi-scale sample image features, including:

[0192] Perform feature extraction on the sample custom color palette image to obtain sample color palette features corresponding to the sample custom color palette image;

[0193] For each scale of the multi-scale image, the sample noise features at the scale are encoded with reference to the sample line draft texture features and sample color plate features of the scale, thereby obtaining the multi-scale sample image features.

[0194] In this embodiment, the sample custom palette image corresponds to sample palette features of at least two different scales, and the number of scales corresponding to the sample palette features of different scales is less than the number of scales corresponding to the multi-scale sample line drawing texture features; the number of scales corresponding to the multi-scale sample line drawing texture features is the same as the number of scales corresponding to the multi-scale sample noise features;

[0195] For each scale in the multi-scale image, the sample noise features of the scale are encoded with reference to the sample line texture features and sample color plate features of the scale, and the multi-scale sample image features are obtained, including:

[0196] For each scale of the multi-scale, when there is a sample color plate feature for the scale, the sample noise feature of the scale is encoded with reference to the sample line draft texture feature and the sample color plate feature of the scale to obtain the sample image feature at the scale;

[0197] When there is no sample color plate feature of the targeted scale, the sample noise feature of the targeted scale is encoded with reference to the sample line draft texture feature of the targeted scale to obtain the sample image feature of the targeted scale.

[0198] In other embodiments, the sample custom color palette image includes a sample grid color block image corresponding to the sample line drawing image, and the method further includes:

[0199] Obtain a sample color palette template divided according to the sample grid color blocks, grid-divide the sample line draft image according to the grid color blocks divided by the sample color palette template, and obtain each sample grid area in the sample line draft image. Each sample grid color block in the sample line draft image corresponds one-to-one to each sample grid area in the sample line draft image; obtain a sample grid color block image corresponding to the sample color palette template. The sample grid color block image is obtained by filling each sample grid color block in the sample color palette template with a custom color. Each sample grid color block is used to define the color of the corresponding sample grid area in the sample line draft image.

[0200] The label colored image has the line draft texture of the sample line draft image, and each grid area in the label colored image has the color filled by the corresponding sample grid color block in the sample grid color block image.

[0201] In this embodiment, the sample custom color palette image includes a sample layer color block image corresponding to the sample line drawing image. The method further includes:

[0202] The sample line drawing image is randomly filled with colors to obtain a sample colored image. The sample colored image is then segmented into layer color blocks according to each sample image area in the sample colored image to obtain a sample layer color block image corresponding to the sample line drawing image. The colored image has the line drawing texture of the line drawing image, and each image area in the colored image has the color indicated by the corresponding layer color block in the layer color block image.

[0203] In this embodiment, the sample colored image is segmented into layer color blocks according to each sample image area in the sample colored image to obtain a sample layer color block image corresponding to the sample line draft image, including: segmenting the sample colored image into layer color blocks according to each sample image area in the sample colored image to obtain a sample initial layer color block image corresponding to the sample line draft image, the sample initial layer color block image including the sample layer color blocks corresponding to each sample image area in the sample line draft image; and obtaining the sample layer color block image based on a custom color adjustment of at least one sample layer color block in the sample initial layer color block image;

[0204] The label colored image has the line draft texture of the sample line draft image, and each image area in the label colored image has the color indicated by the corresponding sample layer color block in the sample layer color block image.

[0205] Step S910 : performing multi-scale decoding on the multi-scale sample image features to obtain a predicted colored image.

[0206] In this embodiment, the computer device performs multi-scale decoding on the multi-scale sample image features and obtains a predicted color image after completing the decoding of the last scale. The predicted color image may have the line drawing texture of the sample line drawing image and may have the color indicated by the sample color guidance information.

[0207] In this embodiment, for each scale in the multi-scale decoding, the sample decoding features of the previous scale are decoded with reference to the sample image features of the targeted scale to obtain the sample decoding features of the targeted scale. Then, the decoding of the next scale is started, and the obtained sample decoding features of the targeted scale are used as the sample decoding features of the previous scale in the decoding of the next scale. The decoding step of returning to the step of decoding the sample decoding features of the previous scale is continued until the decoding of the last scale is completed to obtain a predicted colored image.

[0208] In this embodiment, the encoding order of the multiple scales in encoding is opposite to the decoding order of the multiple scales in decoding.

[0209] In this embodiment, multi-scale decoding is performed on multi-scale sample image features to obtain a predicted color image, including:

[0210] For each scale in the multi-scale decoding, when the targeted scale is the first scale in the multi-scale decoding, the sample image features of the last scale obtained by encoding are decoded with reference to the sample image features of the targeted scale to obtain sample decoding features of the targeted scale;

[0211] When the targeted scale is not the first scale or the last scale in the multi-scale decoding, refer to the sample image features of the targeted scale, decode the sample decoding features of the previous scale of the targeted scale, and obtain the sample decoding features of the targeted scale;

[0212] When the targeted scale is the last scale, the sample decoding features of the last scale are decoded with reference to the sample image features of the targeted scale to obtain a predicted colored image.

[0213] Step S912, obtain the label colored image corresponding to the sample line drawing image, train the initial image processing model based on the predicted colored image and the label colored image, and obtain the image processing model. The label colored image has the line drawing texture of the sample line drawing image and has the color indicated by the sample color guide information.

[0214] Among them, the label colored image is the label in the model training, the label colored image has the line draft texture of the sample line draft image, and has the color indicated by the sample color guide information.

[0215] In this embodiment, the initial image processing model is trained based on the predicted colored image and the labeled colored image to obtain the image processing model, including:

[0216] At least one of a texture loss and a color loss between the predicted colored image and the labeled colored image is determined, and an initial image processing model is trained based on at least one of the texture loss and the color loss to obtain an image processing model.

[0217] In other embodiments, the label colored image may be a predicted colored image output in the previous iteration of the current iteration.

[0218] In this embodiment, a sample line drawing image is obtained and multi-scale sample line drawing texture features are extracted from the sample line drawing image. The multi-scale includes both large and small scales. The large-scale sample line drawing texture features contain more low-level semantic features, while the small-scale sample line drawing texture features contain more high-level semantic features. Multi-scale sample noise features are extracted from a sample noise image, and multiple feature extractions are performed on the line drawing image and a preset noise image at the same scale. The multi-scale sample line drawing texture features and sample color guidance information corresponding to the sample line drawing image are used as guidance information for encoding the sample noise features. For each scale in the multi-scale, the sample noise features at that scale are encoded with reference to the sample line drawing texture features and sample color guidance information at that scale. This encoding generates a sample image feature that combines the sample line drawing texture features with the colors indicated by the sample color guidance information. Furthermore, the fusion of the sample line drawing texture and color helps ensure that color changes in the subsequently generated image align with the direction of the line drawing texture.

[0219] The encoding is multi-scale and can obtain sample image features that fuse sample line drawing texture features at different scales and the indicated color. The multi-scale sample image features are then decoded at multiple scales to obtain a predicted colored image, and a labeled colored image corresponding to the sample line drawing image is obtained. The initial image processing model is then trained based on the difference between the predicted colored image and the labeled colored image to adjust the model parameters so that the predicted colored image generated by the trained image processing model has the line drawing texture of the sample line drawing image and the color indicated by the sample color guidance information. Furthermore, the trained image processing model can generate colored images with clearer line drawing textures and finer edge transition details, and the combination of color and line drawing texture in the colored image is more harmonious, thereby improving the image quality of the colored image.

[0220] In one embodiment, the method further comprises:

[0221] Obtaining a sample image description text for the sample line draft image, where the sample image description text is used to describe features of the sample line draft image; extracting sample image description features from the sample image description text;

[0222] For each scale in the multi-scale image, the sample noise features of the scale are encoded with reference to the sample line texture features and sample color guidance information of the scale, and the multi-scale sample image features are obtained, including:

[0223] For each scale of the multi-scale, the sample noise features of the scale are encoded with reference to the sample line draft texture features, sample color guidance information and sample image description features of the scale, so as to obtain the multi-scale sample image features.

[0224] In one embodiment, training an initial image processing model based on the predicted colored image and the labeled colored image to obtain an image processing model includes:

[0225] Determine the texture loss between the line drawing texture of the predicted colored image and the line drawing texture of the labeled colored image; determine the color loss between the color of the predicted colored image and the color of the labeled colored image; train the initial image processing model according to the texture loss and the color loss to obtain the image processing model.

[0226] In this embodiment, the computer device may calculate the texture loss between the texture of the line art of the predicted colored image and the texture of the line art of the labeled colored image. The computer device may also calculate the color loss between the color of the predicted colored image and the color of the labeled colored image. A target loss is calculated based on the texture loss and the color loss. The parameters of the initial image processing model are adjusted based on the target loss, and training continues until a training stop condition is met, thereby obtaining an image processing model.

[0227] In this embodiment, the initial image processing model is trained based on texture loss and color loss to obtain an image processing model, including:

[0228] Obtain a target loss function, calculate the target loss based on the target loss function, texture loss, and color loss; train the initial image processing model based on the target loss to obtain an image processing model.

[0229] In this embodiment, the texture loss and color loss between the predicted colored image and the labeled colored image generated by the model are taken into account to adjust the model parameters for generating the line drawing texture and the model parameters for generating the color, thereby gradually reducing the texture loss and color loss during training, and then obtaining an image processing model that can accurately generate the line drawing texture and color.

[0230] In one embodiment, the initial image processing model is iteratively trained multiple times based on the predicted color image to obtain the predicted color image corresponding to each iteration. When the image difference between the predicted color image obtained in the current iteration and the predicted color image obtained in the previous iteration of the current iteration meets the iteration stopping condition, the image processing model is obtained.

[0231] In one embodiment, the image processing initial model includes a line drawing control initial unit, which is used to extract multi-scale sample line drawing texture features; the image processing initial model is trained based on the predicted color image and the labeled color image to obtain the image processing model, including:

[0232] Determine the texture loss between the line draft texture of the predicted colored image and the line draft texture of the labeled colored image; train the line draft control initial unit of the image processing initial model based on the texture loss to obtain an image processing model including a line draft control unit.

[0233] Among them, texture loss represents the difference between the line draft texture of the predicted colored image and the line draft texture of the labeled colored image, that is, the difference in line draft texture between the colored image generated by the model and the real colored image.

[0234] In this embodiment, the image processing initial model includes a line draft control initial unit. The computer device inputs the sample line draft image into the image processing initial model, and extracts multi-scale sample line draft texture features from the sample line draft image through the line draft control initial unit.

[0235] The computer device obtains a sample noise image and extracts multi-scale sample noise features from the sample noise image using an initial image processing model. In other embodiments, the sample noise image may be stored in the initial image processing model.

[0236] The sample color guidance information corresponding to the sample line drawing image is obtained and input into the initial image processing model. For each multi-scale scale, the initial image processing model encodes the sample noise characteristics at that scale by referring to the sample line drawing texture features and the sample color guidance information at that scale, thereby obtaining multi-scale sample image features. The initial image processing model then decodes the multi-scale sample image features at multiple scales to obtain a predicted color image generated by the initial image processing model.

[0237] The computer device calculates the texture loss between the line drawing texture of the predicted colored image and the line drawing texture of the labeled colored image, adjusts the parameters of the line drawing control initial unit based on the texture loss, and continues training until the training stop condition is met, thereby obtaining an image processing model including the line drawing control unit.

[0238] In this embodiment, the image processing initialization model includes a line drawing control initialization unit and an image generation initialization unit. The computer device inputs a sample line drawing image into the line drawing control initialization unit and inputs sample color guidance information and a sample noise image into the image generation initialization unit. The line drawing control initialization unit then extracts multi-scale texture features of the sample line drawing based on the sample line drawing image.

[0239] The multi-scale sample noise features extracted from the sample noise image are obtained through the image generation initialization unit, and the multi-scale sample line draft texture features are input into the image generation initialization unit.

[0240] For each multi-scale image, the initial image generation unit encodes the sample noise characteristics at that scale by referencing the sample line texture features and sample color guidance information at that scale, obtaining multi-scale sample image features. The initial image generation unit then decodes the multi-scale sample image features to obtain a predicted colorized image. The initial line control unit and the initial image generation unit are trained based on texture loss to obtain an image processing model consisting of a line control unit and an image generation unit.

[0241] Furthermore, the initial image generation unit includes an encoder and a decoder. The encoder performs encoding to obtain multi-scale sample image features, and the decoder performs multi-scale decoding to obtain a predicted color image.

[0242] In this embodiment, the image processing initial model includes a line draft control initial unit, which is used to extract multi-scale sample line draft texture features through the line draft control initial unit. The line draft control initial unit of the image processing initial model is trained based on texture loss, so that the unit that controls the generation of line draft texture can be trained separately, so as to gradually reduce the texture difference between the predicted colored image and the label image output by the model during training, thereby improving the accuracy of the line draft texture generated by the line draft control unit, so that the line draft texture of the obtained colored image is more accurate.

[0243] In one embodiment, the image processing initial model includes a line drawing control initial unit and a color control initial unit. The line drawing control initial unit is used to extract multi-scale sample line drawing texture features. The sample color guidance information includes a sample color palette image for customizing the color of the sample line drawing image.

[0244] For each of the multiple scales, referring to the sample line draft texture features and sample color guide information of the targeted scale, encoding the sample noise features of the targeted scale to obtain the multi-scale sample image features, including: extracting the sample color plate features of the sample color plate image through the color control initial unit; for each of the multiple scales, referring to the sample line draft texture features and sample color plate features of the targeted scale, encoding the sample noise features of the targeted scale to obtain the multi-scale sample image features;

[0245] The initial image processing model is trained based on texture loss and color loss to obtain an image processing model, including:

[0246] The line drawing control initial unit is trained based on texture loss, and the color control initial unit is trained based on color loss, so as to obtain an image processing model including a line drawing control unit and a color control unit.

[0247] In this embodiment, the image processing initial model includes a line drawing control initialization unit and a color control initialization unit. The computer device extracts multi-scale sample line drawing texture features of the sample line drawing image through the line drawing control initialization unit, and extracts sample color plate features of the sample color plate image through the color control initialization unit.

[0248] For each scale of the multi-scale image processing, the sample line drawing texture features of the targeted scale are extracted by the line drawing control initial unit, and the sample color plate features are extracted by the color control initial unit. The sample noise features of the targeted scale are encoded to obtain the multi-scale sample image features.

[0249] The initial line control unit is trained based on texture loss to obtain a line control unit. Furthermore, the initial color control unit is trained based on color loss to obtain a color control unit, thereby obtaining an image processing model including the line control unit and the color control unit.

[0250] In this embodiment, the image processing initial model includes a line draft control initial unit and a color control initial unit. The line draft control initial unit is trained based on texture loss, and the color control initial unit is trained based on color loss, so that the training of the two units is independent of each other. The line draft control initial unit is trained based on texture loss to gradually reduce the difference in texture between the predicted colored image output by the model and the label image during training, thereby improving the accuracy of the line draft texture generated by the line draft control unit, making the line draft texture of the obtained colored image more accurate. Based on the color loss between the predicted colored image and the label colored image, the color control initial unit is trained to gradually reduce the difference in color between the predicted colored image output by the model and the label image during training, thereby improving the accuracy of the color control unit in generating the color of the line draft image.

[0251] In this embodiment, the training order of the line control initialization unit and the color control initialization unit is not limited. They can be trained simultaneously, or the line control initialization unit can be trained first, and then the color control initialization unit can be trained after the line control unit is obtained. Alternatively, the color control initialization unit can be trained first, and then the line control initialization unit can be trained after the color control unit is obtained.

[0252] In this embodiment, the image processing initial model includes a line draft control initial unit and a color control initial unit. The computer device trains the line draft control initial unit based on texture loss to obtain a trained line draft control unit. After obtaining the line draft control unit, the multi-scale sample line draft texture features of the sample line draft image are extracted by the line draft control unit, and the sample color plate features of the sample color plate image are extracted by the color control initial unit. For each scale of the multi-scale, the sample noise features of the scale are encoded with reference to the sample line draft texture features of the scale and the sample color plate features extracted by the color control initial unit to obtain multi-scale sample image features. The multi-scale sample image features are multi-scale decoded to obtain a predicted colored image. The color loss between the predicted colored image and the labeled colored image is determined, and the color control initial unit is trained based on the texture loss to obtain an image processing model including the line draft control unit and the color control unit.

[0253] In this embodiment, the line control initialization unit is trained based on texture loss to obtain the line control unit. After the line control unit is trained, the color control initialization unit is trained, making the training of the two units independent of each other. The line control unit extracts multi-scale sample line texture features. For each multi-scale scale, the sample noise features at the targeted scale are encoded with reference to the sample line texture features and sample color palette features of the targeted scale to obtain multi-scale sample image features. The multi-scale sample image features are multi-scale decoded to obtain a predicted colored image. The color loss between the predicted colored image and the labeled colored image is determined. The color control initialization unit is trained based on the color loss to obtain an image processing model including the line control unit and the color control unit. The trained line control unit and the color control unit are independent of each other. In the application scenario of the image processing model, the line control unit can be called separately to accurately generate a colored image with line texture and random color. In conjunction with the color control unit, user-defined color information can be combined to accurately generate a colored image with line texture and customized color, making the image processing model adaptable to different usage scenarios.

[0254] In one embodiment, the initial image processing model used during training is based on an improved version of the traditional stable diffusion model. Figure 10 shows the difference between the traditional diffusion model and the initial image processing model. The upper half of Figure 10 shows the architecture of the traditional diffusion model during training, while the lower half shows the main architecture of the initial image processing model. The initial image processing model is obtained by replacing the denoising unit in the upper half with the lower half. Specifically, the initial image processing model adds two branches: a line drawing control unit and a color control unit. The structures of these two branches are similar to the encoder in the denoising unit U-Net, but with different parameter sizes. The branch controlling the line drawing has approximately 400 megabytes of parameters, while the branch controlling the color has approximately 40 megabytes of parameters, making the two branches asymmetric. This is primarily due to the different levels of semantic information in color and texture data. While high-level line drawing texture requires a deeper network structure for recognition and understanding, low-level color guidance information only requires a lightweight network. Color guidance information can also take into account time embeddings, and texture information in the line drawing image can also include image description text prompts. That is, the image processing initial model is input into the line drawing control unit as an additional condition.

[0255] The line art control branch, or line art control unit, refers to the ControlNet network structure, as shown in Figure 11. The network structure on the left is the pre-trained portion of the diffusion model, which provides stable image generation results. The weights in this portion do not need to be updated. The network structure on the right is the line art control branch, ControlNet. Its function is to further incorporate customized line art texture information into the pre-trained network to guide the network to generate images with specific line art textures.

[0256] To train the line drawing control branch, we collected a large amount of high-quality, open-source image data. We then used the line drawing extraction model anime2sketch to extract the corresponding line drawings from these image data as input for the line drawing control branch, thereby constructing paired data for supervised training. To make the line drawing model training more robust, we performed data augmentation on the line drawing output of the Anime2Sketch model. This ensures that the model truly learns the line drawing information during training, rather than other irrelevant factors such as image noise and image orientation.

[0257] The first step of data enhancement is image quantization, that is, normalization, which is achieved through zero convolution network. The role of quantization is to convert continuous values ​​into discrete values ​​and discard redundant information, especially to avoid some grayscale information of the original image in the line drawing input. The network may use this grayscale information to reconstruct the original image. In this way, what the network learns is not the true line drawing generation. Therefore, it is very necessary to quantize the line drawing input. Let I be the original line drawing image, that is, the line drawing image before quantization, whose pixel value range is 0 to 1. The quantization operation maps each continuous pixel value into a discrete value. Let Q be the number of intervals used for quantization, and the sample line drawing image I after quantization q It can be expressed as: q (x,y)=floor(I(x,y)*Q) / Q

[0258] Where floor is the floor function and (x, y) is the coordinate of the pixel being quantized. q The pixel values ​​in will only have Q possible discrete values, rather than a continuous value from 0 to 1. The default setting is Q=8.

[0259] Other data enhancement operations include commonly used image flipping upside down, flipping left and right, adding random white noise, etc.

[0260] When training the line branch, the color branch does not exist. Training parameters: batch size equals 64, learning rate equals 1×10 -5 , the training steps are 5000, and the optimizer is Adam. The objective function of the training is the image reconstruction loss function L DM :

[0261] Among them, x is the label colored image, ε is the predicted colored image, which represents the intermediate result of image generation; ε θ It refers to the predicted colored image when the model parameters are θ in the t-th iteration, which is also the intermediate result of image generation; θ refers to the model parameters; N(0, 1) represents a Gaussian distribution with mean 0 and variance 1; t∈{1,2,…T} represents the number of sampling steps.

[0262] The color control branch has two inputs: rectangular color blocks and SAM layer color blocks. To train the color control branch network, we need to construct training data for the corresponding rectangular color blocks and SAM layer color blocks.

[0263] Get a rectangular color block:

[0264] Assume that the original line drawing image X is a two-dimensional matrix of size H×W. Use bicubic interpolation to downsample X by a factor of d: X resized=bicubic_resize(X,(W / d),(H / d))

[0265] Among them, bicubic_resize is a function used to perform bicubic interpolation on the rectangular color block image and resize it to the specified size. Then, the resized rectangular color block image is upsampled to an H×W image using nearest neighbor interpolation: color = nearest_neighbor_resize(X resize ,W,H)

[0266] The nearest_neighbor_resize function performs nearest neighbor interpolation on a resized rectangular block image to the specified size. The default setting is d = 32, and the upsampling and downsampling functions can directly call the pillow library.

[0267] During the inference phase, the rectangular color block image is initialized to a blank space. Users need to fill in the colors of the rectangular blocks in the rectangular color block image and then input the rectangular color block image into the network to obtain a colored image with the custom colors. The inference phase is also the stage for model application.

[0268] Get the SAM layer color block:

[0269] Layer blocks are divided into blocks based on the image segmentation results. By default, each block has the same color. The segmentation model uses SAM (sagment anything). To obtain full image segmentation results, a prompt is input to the SAM model. The prompt here is a uniformly distributed grid of points. The model automatically filters out low-quality or duplicate segmentation layers. The results of layer block segmentation can be seen in Figures 7 and 8.

[0270] During training, you can segment a layer to obtain the original color values ​​of the image within that layer. However, when applying the model, users can only modify and specify colors for the entire layer, so the colors within the layer are consistent. To minimize this discrepancy between training and application, preprocessing the training data is necessary. This involves performing a local average based on the layer segmentation results.

[0271] Assume that the image is segmented to obtain layer information: m i,j ∈{0, 1, ..., N}, where N represents the number of layers. For the pixel value m in the layer i,j Performing local averaging, it can be expressed as:

[0272] After layer pixel preprocessing, the layer color blocks obtained in the training phase are consistent with the layer color blocks input by the user in the inference phase, thus ensuring the stability of the model output results.

[0273] After constructing the corresponding training data for rectangular color patches and SAM layer color patches, the color branch network can be trained. The color branch network structure uses a more lightweight network compared to the control network. The main reason is that color information is easier to understand than network information and does not require overly complex parameters. Referring to Figure 4 above, the input of the color control network is the sum of the time embedding and the downsampled custom palette image (i.e., the rectangular color patch image or the SAM layer color patch image). Downsampling is generally achieved through the pixel shuffle operation, the basic principle of which is to change the spatial pixel distribution to the channel direction. Assume that the original size of the custom palette image is (1, 3, 512, 512), and the downsampled palette image is (1, 129, 64, 64). The palette features of each dimension obtained by the color control branch are added to the vectors of the corresponding dimensions in the image generation unit to control and guide the generated results.

[0274] For the training of the color control branch network, it is necessary to fix the weights of the line drawing control branch and then perform joint training.

[0275] The training parameters of the color control branch are: batch size = 64, learning rate = 1×10 -5 The training steps are 5000 and the optimizer is Adam. The training objective function also uses the image reconstruction loss function mentioned above.

[0276] In the image processing model trained in this embodiment, the line drawing control branch, rectangular color blocks, and SAM layer color blocks can be used individually or in combination. For example, the line drawing control branch can be used alone to generate a random automatic coloring effect. Using the line drawing control branch + rectangular color blocks, custom color control can be achieved by editing the color of the rectangular box. Using the line drawing control branch + SAM layer color blocks, color control of the corresponding image area can be achieved by specifically modifying the color of a layer color block.

[0277] It is understood that the line extraction model is not limited to the anime2sketch method. Other edge extraction models such as canny and pidinet are also applicable. The layer segmentation method is not limited to the SAM model. Other stable image segmentation methods can also be used.

[0278] In one embodiment, an image processing method is provided, which is applied to a computer device and includes:

[0279] Acquire a line drawing image and extract multi-scale line drawing texture features of the line drawing image.

[0280] And, obtaining a preset noise image, and extracting multi-scale noise features from the preset noise image.

[0281] Furthermore, a custom color palette image corresponding to the line drawing image is obtained, and at least two color palette features of different scales are extracted from the custom color palette image.

[0282] Furthermore, for each scale of the multiple scales, when there is a color plate feature at the targeted scale, the noise feature of the targeted scale is encoded with reference to the line drawing texture feature and the color plate feature of the targeted scale to obtain the image feature of the targeted scale.

[0283] Next, when there is no color plate feature of the targeted scale, the noise feature of the targeted scale is encoded with reference to the line drawing texture feature of the targeted scale to obtain the image feature of the targeted scale.

[0284] Furthermore, for each scale in the multi-scale decoding, when the targeted scale is the first scale in the multi-scale decoding, the image features of the last scale obtained by encoding are decoded with reference to the image features of the targeted scale to obtain the decoded features of the targeted scale.

[0285] Optionally, when the targeted scale is not the first scale or the last scale in the multi-scale decoding, the decoded features of the previous scale of the targeted scale are decoded with reference to the image features of the targeted scale to obtain the decoded features of the targeted scale.

[0286] Furthermore, when the targeted scale is the last scale, the decoded features of the previous scale of the last scale are decoded with reference to the image features of the targeted scale to obtain a colored image. The colored image has the line draft texture of the line draft image and has the color indicated by the custom color palette image.

[0287] In one embodiment, as shown in Figures 12, 13, and 14, the image processing model automatically colors line drawings, effectively adapting to diverse scenes such as landscapes and people, with good spatial color correspondence and tonal consistency. Without the color control unit, the generated image colors are random; if color control is added, the generated image is laid out according to spatial color blocks.

[0288] As shown in Figure 12, there is no color palette image as input, that is, there is no color control, and only line drawing image input can generate high-quality colored images, but the colors are random.

[0289] As shown in FIG13 , adding a rectangular color block image, that is, a grid color block image, can stably generate a customized color image, and the result is not affected by the random seed.

[0290] As shown in FIG14 , adding a SAM color block image, that is, a layer color block image, can also stably generate a customized color image.

[0291] The results generated by the rectangular color block and the SAM color block will have color differences, which is caused by the fact that the color values ​​of the two inputs are not completely consistent.

[0292] In one embodiment, as shown in FIG15 , an application scenario of an image processing method is provided. After a user uploads a line drawing image, a custom color palette image is obtained by any of the following methods:

[0293] Fill the mask with color one by one, that is, fill the grid color blocks in a custom way to form a grid color block image;

[0294] Alternatively, you can select a satisfactory layer color block image from the initial layer color block images recommended by the system. If there is an unsatisfactory layer color block, you can directly modify the color of the layer color block to form a layer color block image.

[0295] A grid or layer color block image can be used as a custom palette image. The custom palette image and line drawing image are input into an image processing model. The image processing model includes a line drawing control unit, a color control unit, and an image generation unit. The image processing model stores a preset noise image.

[0296] The image processing model processes the line drawing image, the custom color palette image and the preset noise image to generate a colored image corresponding to the line drawing image.

[0297] After the image processing model generates a colored image, if the user is still not satisfied with the colors in the colored image, the user can continue to adjust the corresponding grid color block image or grid color block image to generate a new colored image.

[0298] In this embodiment, automatic coloring of line drawing input is possible. The designed custom color palette image enables precise spatial control of the generated image, ensuring consistency in spatial elements and color control. At the same time, network structures of varying sizes are designed for the line drawing control unit and color control unit based on the semantic attributes of the line drawing texture structure and color. This ensures the generation of high-quality colored images when user-defined colors are used, and also ensures the stable generation of high-quality colored images based on line drawing images even without color input.

[0299] It should be understood that, although the steps in the flowcharts of the above embodiments are shown in sequence as indicated by the arrows, these steps are not necessarily performed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be performed in other orders. Moreover, at least a portion of the steps in the flowcharts of the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily performed at the same time, but can be performed at different times. The execution order of these steps or stages is not necessarily to be performed in sequence, but can be performed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0300] Based on the same inventive concept, embodiments of the present application also provide an image processing device for implementing the aforementioned image processing method. The solution provided by this device is similar to the solution described in the aforementioned method. Therefore, the specific limitations in one or more of the following image processing device embodiments can be found in the above-described limitations on the image processing method and will not be further elaborated here.

[0301] In one embodiment, as shown in FIG16 , an image processing apparatus 1600 is provided, comprising: a texture feature extraction module 1602 , a noise feature extraction module 1604 , a color acquisition module 1606 , an encoding module 1608 , and a decoding module 1610 , wherein:

[0302] The texture feature extraction module 1602 is used to obtain a line drawing image and extract multi-scale line drawing texture features of the line drawing image.

[0303] The noise feature extraction module 1604 is configured to obtain a preset noise image and extract multi-scale noise features from the preset noise image.

[0304] The color acquisition module 1606 is used to determine the color guide information corresponding to the line drawing image.

[0305] The encoding module 1608 is configured to encode the noise features of each scale of the multi-scale image with reference to the line drawing texture features and color guidance information of the scale, thereby obtaining multi-scale image features.

[0306] The decoding module 1610 is configured to perform multi-scale decoding on the multi-scale image features to obtain a colored image. The colored image has the line draft texture of the line draft image and has the color indicated by the color guide information.

[0307] In this embodiment, a line drawing image is acquired, and multi-scale line drawing texture features are extracted based on the line drawing image. The extracted large-scale line drawing texture features contain more low-level semantic features, while the small-scale line drawing texture features contain more high-level semantic features. Multi-scale noise features are extracted from a preset noise image, and multiple feature extractions at the same scale are performed on the line drawing image and the preset noise image. The multi-scale line drawing texture features and color guidance information corresponding to the line drawing image are used as guidance information for noise feature encoding. For each multi-scale scale, the noise features at that scale are encoded with reference to the line drawing texture features and color guidance information of the targeted scale. This encoding generates image features that fuse the line drawing texture features and the colors indicated by the color guidance information. Furthermore, the line drawing texture and color are integrated, so that the colors in subsequently generated images can align with the direction of the line drawing texture. This multi-scale encoding process fuses line art texture features at different scales with color image features, enabling multi-scale decoding of these features. This process focuses on textures at different levels, resulting in a colored image with clearer line art textures and finer edge transition details. Furthermore, this results in a more harmonious blend of color and line art texture in the colored image, improving the quality of the resulting colored image.

[0308] In one embodiment, the color guidance information includes a custom palette image, which is used to customize the color of the line drawing image; the encoding module 1608 is also used to extract the palette features of the custom palette image; for each scale of the multi-scale, the noise features of the scale are encoded with reference to the line drawing texture features and palette features of the scale to obtain the multi-scale image features.

[0309] In this embodiment, the custom palette image is a palette image of the color of the user-defined line drawing image. By performing feature extraction on the user-defined palette image, palette features corresponding to the custom palette image are obtained. For each of the multiple scales, the noise features at that scale are encoded with reference to the line drawing texture features and palette features of the targeted scale. The noise features can be encoded into image features that combine the line drawing texture features and palette features. This allows the line drawing image and corresponding color information to be used as guidance information, enabling the generation of an image with the user's desired line drawing texture and customized color from the preset noise image. Furthermore, by performing encoding at multiple scales, the multi-scale image features obtained can contain both shallow and deep information about the line drawing texture and color, resulting in a decoded image with clearer and more delicate texture and more harmonious colors.

[0310] In one embodiment, the encoding module 1608 is further used to extract palette features of at least two different scales from the custom palette image, and the number of scales corresponding to the at least two different scales is less than the number of scales corresponding to the multiple scales; for each scale of the multiple scales, when the palette features at the targeted scale exist, the noise features at the targeted scale are encoded with reference to the line drawing texture features and palette features at the targeted scale to obtain the image features at the targeted scale; when the palette features at the targeted scale do not exist, the noise features at the targeted scale are encoded with reference to the line drawing texture features at the targeted scale to obtain the image features at the targeted scale.

[0311] In this embodiment, color and texture belong to different levels of semantics: color belongs to a lower level of semantics, while texture belongs to a higher level of semantic information. Therefore, features of fewer scales are extracted for color, while features of more scales are extracted for the line art texture of the line art image. For each of the multiple scales, when palette features exist for the targeted scale, the noise features of the targeted scale are encoded with reference to the line art texture features and palette features of the targeted scale. This allows the line art texture features and palette features to be used as guidance information, indicating that the line art texture features and palette features should be fused during encoding based on the noise features, thereby obtaining image features for the targeted scale that are fused with the line art texture features and palette features of the corresponding scale. When palette features do not exist for the targeted scale, the noise features of the targeted scale are encoded with reference to the line art texture features of the targeted scale, using the line art texture features as guidance information, indicating that the line art texture features should be fused during encoding based on the noise features, thereby obtaining image features for the targeted scale that are fused with the line art texture features of the corresponding scale. This time, the image features that integrate different levels of line drawing texture and color can reconstruct the image more accurately, making the line drawing texture in the reconstructed colored image clearer and more delicate, and the color more harmonious and natural, thereby improving the image quality.

[0312] In one embodiment, the color acquisition module 1606 is further used to obtain a color palette template divided according to grid color blocks, and to grid-divide the line draft image according to the grids divided by the color palette template to obtain grid areas in the line draft image, where each grid color block in the line draft image corresponds one-to-one to each grid area in the line draft image; obtain a grid color block image corresponding to the color palette template, where the grid color block image is obtained by filling each grid color block in the color palette template with a custom color, and the grid color block is used to define the color of the corresponding grid area in the line draft image; wherein the colored image has the line draft texture of the line draft image, and each grid area in the colored image has the color filled with the corresponding grid color block in the grid color block image.

[0313] In this embodiment, a color palette template divided into grid color blocks is obtained, and the line drawing image is grid-divided according to the grid color blocks divided by the color palette template. This results in grid areas representing image regions in the line drawing image. This ensures a one-to-one correspondence between each grid area of ​​the line drawing image and each grid color block of the color palette template. This allows the user to define the color of each image area in the line drawing image by filling the grid color blocks with color, effectively enabling user customization of the color of the line drawing image. Furthermore, the user-defined color is subsequently used as guidance information in feature encoding and decoding, ensuring that the resulting colored image not only possesses the line drawing texture of the line drawing image, but also that each grid area in the colored image also has the color that the user filled in the corresponding grid color block, effectively enabling the generation of a line drawing image with a customized color.

[0314] In one embodiment, the color acquisition module 1606 is further used to randomly fill the line drawing image with colors to obtain a colored image; the colored image is segmented into layer color blocks according to each image area in the colored image to obtain a layer color block image corresponding to the line drawing image, and the layer color block image includes a layer color block corresponding to each image area; wherein the colored image has the line drawing texture of the line drawing image, and each image area in the colored image has the color indicated by the corresponding layer color block in the layer color block image.

[0315] In this embodiment, random color filling is performed on the line drawing image to obtain a colored image. This allows for layer color block segmentation of the colored image according to each image region within the colored image, accurately segmenting each layer color block within the colored image. Each layer color block forms a layer color block image corresponding to the colored image. Extracting features from the layer color block image yields a palette feature, which is then used as color guidance information for feature encoding and decoding. This ensures that the resulting colored image not only possesses the line drawing texture of the line drawing image, but also that each image region within the colored image has the color defined by the corresponding layer color block, enabling rapid coloring of the line drawing image.

[0316] In one embodiment, the color acquisition module 1606 is further used to segment the colored image into layer color blocks according to each image area in the colored image to obtain an initial layer color block image, where the initial layer color block image includes layer color blocks corresponding to each image area; and to obtain a layer color block image based on a customized color adjustment of at least one layer color block in the initial layer color block image.

[0317] In this embodiment, the line drawing image is randomly filled with colors to obtain a colored image, so that the colored image can be segmented into layer color blocks according to each image area in the colored image, and each layer color block in the colored image can be accurately segmented. Each layer color block forms an initial layer color block image corresponding to the colored image. The initial layer color block image includes the layer color blocks corresponding to each image area in the line drawing image. The user can adjust the color of each layer color block to redefine the color they want, thereby obtaining a user-defined layer color block image. In addition, the user-defined color is subsequently used as guidance information in feature encoding and decoding, so that the final generated colored image not only has the line drawing texture of the line drawing image, but also each image area in the colored image has the color filled by the user in the corresponding layer color block, effectively realizing the generation of a line drawing image with a customized color.

[0318] In one embodiment, the device further includes a description feature extraction module; the description feature extraction module is configured to obtain image description text for the line drawing image; and extract image description features from the image description text;

[0319] The encoding module 1608 is configured to encode the noise features of each of the multi-scales with reference to the line drawing texture features, color guidance information, and image description features of the scale to obtain multi-scale image features.

[0320] In this embodiment, image description text is obtained for a line drawing image, and features of the line drawing image are described in the image description text as supplementary information for the line drawing image. Image description features are extracted from the image description text, and for each of the multiple scales, the noise features of the targeted scale are encoded with reference to the line drawing texture features, color guidance information, and image description features of the targeted scale. This allows the encoded image features to incorporate the line drawing texture and the color indicated by the line drawing texture features at the corresponding scale, and also includes feature information described by the image description features. This allows the encoding to capture image content not present in the line drawing image, or to convert previously unclear features in the line drawing image into clear features by combining the image description features. Furthermore, encoding at multiple scales can yield multi-scale image features. Image features at larger scales contain more deep image information, while image features at smaller scales contain more shallow image information, thereby enabling the acquisition of image information at different levels.

[0321] In one embodiment, the decoding module 1610 is further configured to, for each scale in the multi-scale decoding, when the targeted scale is the first scale in the multi-scale decoding, decode the image features of the last scale obtained by encoding with reference to the image features of the targeted scale to obtain the decoded features of the targeted scale; when the targeted scale is not the first or last scale in the multi-scale decoding, decode the decoded features of the previous scale of the targeted scale with reference to the image features of the targeted scale to obtain the decoded features of the targeted scale; and when the targeted scale is the last scale, decode the decoded features of the previous scale of the last scale with reference to the image features of the targeted scale to obtain the colored image.

[0322] In this embodiment, during the multi-scale decoding process, the image features at each scale are used as a reference for decoding at that scale. This allows the encoded features to be used as a reference for decoding the decoded features, effectively avoiding decoding errors and improving decoding accuracy. Furthermore, during decoding at different scales, the features decoded at each scale are the same as those obtained at the previous scale. This allows for progressive decoding of the decoded features to gradually reconstruct the image's texture details and color information, accurately reconstructing a colored image with the texture and color of the line drawing.

[0323] Based on the same inventive concept, the present application also provides an image processing model training device for implementing the image processing model training method described above. The solution to the problem provided by this device is similar to the solution described in the above method. Therefore, the specific limitations of the embodiments of the training device for one or more image processing models provided below can be found in the above-mentioned limitations of the image processing model training method, and will not be repeated here.

[0324] In one embodiment, as shown in FIG17 , a training apparatus 1700 for an image processing model is provided, comprising:

[0325] The sample texture feature extraction module 1702 is used to obtain a sample line drawing image and extract multi-scale sample line drawing texture features of the sample line drawing image.

[0326] The sample noise feature extraction module 1704 is used to obtain a sample noise image and extract multi-scale sample noise features from the sample noise image.

[0327] The sample color acquisition module 1706 is used to determine the sample color guide information corresponding to the sample line drawing image;

[0328] The sample encoding module 1708 is used to encode the sample noise features of each scale of the multi-scale image by referring to the sample line drawing texture features and sample color guidance information of the scale, so as to obtain the multi-scale sample image features.

[0329] The sample decoding module 1710 is used to perform multi-scale decoding on multi-scale sample image features to obtain a predicted colored image.

[0330] Training module 1712 is used to obtain the label colored image corresponding to the sample line drawing image, train the initial image processing model based on the predicted colored image and the label colored image, and obtain the image processing model. The label colored image has the line drawing texture of the sample line drawing image and has the color indicated by the sample color guide information.

[0331] In this embodiment, a sample line drawing image is obtained, and multi-scale sample line drawing texture features are extracted based on the sample line drawing image. Large-scale sample line drawing texture features contain more low-level semantic features, while small-scale sample line drawing texture features contain more high-level semantic features. Multi-scale sample noise features are extracted from a sample noise image, and the multi-scale sample line drawing texture features and sample color guidance information corresponding to the sample line drawing image are used as guidance information for encoding the sample noise features. For each scale in the multi-scale image, the sample noise features at the targeted scale are encoded with reference to the sample line drawing texture features and sample color guidance information at the targeted scale. This encoding generates a sample image feature that combines the sample line drawing texture features and the color indicated by the sample color guidance information. Furthermore, the fusion of the sample line drawing texture and color helps ensure that the colors in subsequently generated images align with the direction of the line drawing texture.

[0332] The encoding is multi-scale and can obtain a fusion of sample line drawing texture features at different scales and sample image features of the color. This allows for multi-scale decoding of the multi-scale sample image features to obtain a predicted colored image, and a labeled colored image corresponding to the sample line drawing image. The initial image processing model is then trained based on the difference between the predicted colored image and the labeled colored image to adjust the model parameters so that the predicted colored image generated by the trained image processing model has the line drawing texture of the sample line drawing image and the color indicated by the sample color guidance information. Furthermore, the trained image processing model can generate colored images with clearer line drawing textures and finer edge transition details, and the combination of color and line drawing texture in the colored image is more harmonious, thereby improving the image quality of the colored image.

[0333] In one embodiment, the image processing initial model includes a line draft control initial unit, which is used to extract multi-scale sample line draft texture features; the training module 1712 is also used to determine the texture loss between the line draft texture of the predicted colored image and the line draft texture of the labeled colored image; the line draft control initial unit of the image processing initial model is trained based on the texture loss to obtain an image processing model including a line draft control unit.

[0334] In this embodiment, the image processing initial model includes a line draft control initial unit, which is used to extract multi-scale sample line draft texture features through the line draft control initial unit. The line draft control initial unit of the image processing initial model is trained based on texture loss, so that the unit that controls the generation of line draft texture can be trained separately, so as to gradually reduce the texture difference between the predicted colored image and the label image output by the model during training, thereby improving the accuracy of the line draft texture generated by the line draft control unit, so that the line draft texture of the obtained colored image is more accurate.

[0335] In one embodiment, the image processing initial model includes a line drawing control initial unit and a color control initial unit. The line drawing control initial unit is used to extract multi-scale sample line drawing texture features. The sample color guidance information includes a sample color palette image for customizing the color of the sample line drawing image.

[0336] The sample color acquisition module 1706 is configured to extract the sample color plate features of the sample color plate image through the color control initial unit.

[0337] The sample encoding module 1708 is further configured to encode the sample noise features of each scale of the multi-scale image with reference to the sample line drawing texture features and sample color plate features of the scale, thereby obtaining the multi-scale sample image features.

[0338] The training module 1712 is further configured to train the line drawing control initial unit based on texture loss, and to train the color control initial unit based on color loss, to obtain an image processing model including a line drawing control unit and a color control unit.

[0339] In this embodiment, the image processing initial model includes a line draft control initial unit and a color control initial unit. The line draft control initial unit is trained based on texture loss, and the color control initial unit is trained based on color loss, so that the training of the two units is independent of each other. The line draft control initial unit is trained based on texture loss to gradually reduce the difference in texture between the predicted colored image output by the model and the label image during training, thereby improving the accuracy of the line draft texture generated by the line draft control unit, making the line draft texture of the obtained colored image more accurate. Based on the color loss between the predicted colored image and the label colored image, the color control initial unit is trained to gradually reduce the difference in color between the predicted colored image output by the model and the label image during training, thereby improving the accuracy of the color control unit in generating the color of the line draft image.

[0340] In one embodiment, the image processing initial model further includes a color control initial unit, the sample color guide information includes a sample color palette image for customizing the color of the sample line drawing image, and the color control initial unit is used to extract sample color palette features of the sample color palette image;

[0341] The training module 1712 is also used to train the line draft control initial unit based on texture loss to obtain a line draft control unit; the line draft control unit is used to extract multi-scale sample line draft texture features; for each scale of the multi-scale, with reference to the sample line draft texture features and sample color plate features of the targeted scale, the sample noise features at the targeted scale are encoded to obtain multi-scale sample image features; the multi-scale sample image features are multi-scale decoded to obtain a predicted colored image; the color loss between the predicted colored image and the labeled colored image is determined, and the color control initial unit is trained based on the texture loss to obtain an image processing model including the line draft control unit and the color control unit.

[0342] In this embodiment, the line control initialization unit is trained based on texture loss to obtain the line control unit. After the line control unit is trained, the color control initialization unit is trained, making the training of the two units independent of each other. The line control unit extracts multi-scale sample line texture features. For each multi-scale scale, the sample noise features at the targeted scale are encoded with reference to the sample line texture features and sample color palette features of the targeted scale to obtain multi-scale sample image features. The multi-scale sample image features are multi-scale decoded to obtain a predicted colored image. The color loss between the predicted colored image and the labeled colored image is determined. The color control initialization unit is trained based on texture loss to obtain an image processing model including the line control unit and the color control unit. The trained line control unit and the color control unit are independent of each other. In the application scenario of the image processing model, the line control unit can be called separately to accurately generate a colored image with line texture and random color. In conjunction with the color control unit, user-defined color information can be combined to accurately generate a colored image with line texture and customized color, making the image processing model adaptable to different usage scenarios.

[0343] In one embodiment, the training module 1712 is also used to determine the line drawing texture of the predicted colored image, and determine the texture loss between the line drawing texture of the predicted colored image and the line drawing texture of the labeled colored image; determine the color loss between the color of the predicted colored image and the color of the labeled colored image; and train the initial image processing model based on the texture loss and color loss to obtain the image processing model.

[0344] In this embodiment, the texture loss and color loss between the predicted colored image and the labeled colored image generated by the model are taken into account to adjust the model parameters for generating the line drawing texture and the model parameters for generating the color, thereby gradually reducing the texture loss and color loss during training, and then obtaining an image processing model that can accurately generate the line drawing texture and color.

[0345] Each module in the aforementioned image processing device and image processing model training device may be implemented in whole or in part via software, hardware, or a combination thereof. Each module may be embedded in or independent of a processor in a computer device in the form of hardware, or may be stored in a memory in the computer device in the form of software, so that the processor can call and execute the corresponding operations of each module.

[0346] In one embodiment, a computer device is provided. The computer device may be a terminal or a server. Taking a terminal as an example, its internal structure diagram may be shown in FIG18 . The computer device includes a processor, memory, an input / output interface, a communication interface, a display unit, and an input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface, display unit, and input device are connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal via wired or wireless communication. The wireless communication may be achieved via Wi-Fi, a mobile cellular network, NFC (near field communication), or other technologies. When executed by the processor, the computer program implements an image processing method and may also implement a method for training an image processing model. The display unit of the computer device is used to form a visually visible image, and can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device casing, or an external keyboard, touchpad or mouse, etc.

[0347] Those skilled in the art will understand that the structure shown in Figure 18 is merely a block diagram of a portion of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0348] In one embodiment, a computer device is further provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps in the above method embodiments when executing the computer program.

[0349] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0350] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0351] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions.

[0352] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, database or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processor involved in the various embodiments provided herein may be, but are not limited to, a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic unit, a data processing logic unit based on quantum computing, and the like.

[0353] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0354] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. An image processing method, executed by a computer device, comprising: Acquire a line draft image, and extract multi-scale line draft texture features of the line draft image; Acquire a preset noise image, and extract the multi-scale noise features from the preset noise image; Determining color guide information corresponding to the line drawing image; For each scale of the multi-scales, refer to the line drawing texture feature and the color guide information of the scale, encode the noise feature of the scale, and obtain a multi-scale image feature; and The multi-scale image features are multi-scale decoded to obtain a colored image, wherein the colored image has the line draft texture of the line draft image and has the color indicated by the color guide information.

2. The method according to claim 1, wherein the color guide information includes a custom palette image, and the custom palette image is used to customize the color of the line drawing image; for each scale of the multi-scale, referring to the line drawing texture feature and the color guide information of the scale, encoding the noise feature of the scale to obtain the multi-scale image feature, comprising: Extracting the palette features of the custom palette image; and For each scale of the multi-scales, the noise features of the scale are encoded with reference to the line drawing texture features and the color plate features of the scale to obtain multi-scale image features.

3. The method according to claim 2, wherein extracting the palette features of the custom palette image comprises: Extracting at least two different scales of palette features from the custom palette image, wherein the number of scales corresponding to the at least two different scales is less than the number of scales corresponding to the multiple scales; For each scale of the multi-scales, referring to the line drawing texture feature and the color plate feature of the scale, encoding the noise feature of the scale to obtain the multi-scale image feature, includes: For each scale of the multiple scales, when there is a color plate feature of the targeted scale, refer to the line drawing texture feature and the color plate feature of the targeted scale, encode the noise feature of the targeted scale, and obtain the image feature of the targeted scale; and When there is no color plate feature at the targeted scale, the noise feature at the targeted scale is encoded with reference to the line draft texture feature at the targeted scale to obtain the image feature at the targeted scale.

4. According to the method of claim 2 or 3, the custom color palette image includes a grid color block image corresponding to the line drawing image, and the method further includes: Obtaining a color palette template divided according to grid color blocks, and dividing the line draft image into grids according to the grid color blocks divided by the color palette template to obtain grid areas in the line draft image, wherein each grid color block in the line draft image corresponds one-to-one to each grid area in the line draft image; and Obtaining a grid color block image corresponding to the color palette template, wherein the grid color block image is obtained by filling each grid color block in the color palette template with a custom color, and the grid color block is used to define the color of the corresponding grid area in the line drawing image; The colored image has the line drawing texture of the line drawing image, and each grid area in the colored image has the color defined by the corresponding grid color block in the grid color block image.

5. The method according to any one of claims 2 to 4, wherein the custom color palette image comprises a layer color block image corresponding to the line drawing image, and the method further comprises: Filling the line draft image with random colors to obtain a colored image; and According to each image area in the colored image, the colored image is segmented into layer color blocks to obtain a layer color block image corresponding to the line draft image, wherein the layer color block image includes a layer color block corresponding to each of the image areas; The colored image has the line drawing texture of the line drawing image, and each image area in the colored image has the color indicated by the corresponding layer color block in the layer color block image.

6. The method according to claim 5, wherein the colored image is segmented into layer color blocks according to each image area in the colored image to obtain a layer color block image corresponding to the line drawing image, comprising: According to each image area in the colored image, the colored image is segmented into layer color blocks to obtain an initial layer color block image, wherein the initial layer color block image includes the layer color blocks corresponding to each of the image areas; and Based on the customized color adjustment of at least one of the layer color blocks in the initial layer color block image, a layer color block image is obtained.

7. The method according to any one of claims 1 to 6, further comprising: Obtaining image description text for the line drawing image; Extracting image description features from the image description text; and For each scale of the multi-scales, referring to the line drawing texture feature and the color guide information of the scale, encoding the noise feature of the scale to obtain the multi-scale image feature, includes: For each scale of the multi-scale, the noise feature of the scale is encoded with reference to the line drawing texture feature, the color guide information and the image description feature of the scale, so as to obtain the multi-scale image feature.

8. The method according to any one of claims 1 to 7, wherein the multi-scale decoding of the multi-scale image features to obtain a colored image comprises: For each scale in the multi-scale decoding, when the targeted scale is the first scale in the multi-scale decoding, refer to the image features of the targeted scale, decode the image features of the last scale obtained by encoding, and obtain the decoding features of the targeted scale; When the targeted scale is not the first scale or the last scale in the multi-scale decoding, refer to the image features of the targeted scale, decode the decoding features of the previous scale of the targeted scale, and obtain the decoding features of the targeted scale; and When the targeted scale is the last scale, the decoded features of the previous scale of the last scale are decoded with reference to the image features of the targeted scale to obtain a colored image.

9. A method for training an image processing model, performed by a computer device, the method comprising: Acquire a sample line draft image, and extract multi-scale sample line draft texture features of the sample line draft image; Acquire a sample noise image, and extract the multi-scale sample noise features from the sample noise image; Determining sample color guide information corresponding to the sample line drawing image; For each scale of the multi-scales, refer to the sample line draft texture feature and the sample color guide information of the scale, encode the sample noise feature of the scale, and obtain the multi-scale sample image feature; Performing multi-scale decoding on the multi-scale sample image features to obtain a predicted colored image; Obtain a labeled colored image corresponding to the sample line drawing image, and train an initial image processing model based on the predicted colored image and the labeled colored image to obtain an image processing model, wherein the labeled colored image has the line drawing texture of the sample line drawing image and has the color indicated by the sample color guide information.

10. The method according to claim 9, wherein the initial image processing model is trained based on the predicted colored image and the labeled colored image to obtain the image processing model, comprising: Determining a line draft texture of the predicted colored image, and determining a texture loss between the line draft texture of the predicted colored image and the line draft texture of the label colored image; determining a color loss between a color of the predicted colored image and a color of the labeled colored image; The initial image processing model is trained according to the texture loss and the color loss to obtain an image processing model.

11. The method according to claim 10, wherein the image processing initial model comprises a line draft control initial unit and a color control initial unit, the line draft control initial unit is used to extract the multi-scale sample line draft texture features, and the sample color guide information comprises a sample color palette image for customizing the color of the sample line draft image; For each scale of the multi-scales, referring to the sample line draft texture feature and the sample color guide information of the scale, encoding the sample noise feature of the scale to obtain the multi-scale sample image feature, including: Extracting the sample color palette features of the sample color palette image by the color control initialization unit; For each scale of the multi-scales, refer to the sample line draft texture feature and the sample color plate feature of the scale, encode the sample noise feature of the scale, and obtain the multi-scale sample image feature; The step of training the initial image processing model according to the texture loss and the color loss to obtain the image processing model comprises: The line drawing control initial unit is trained based on the texture loss, and the color control initial unit is trained based on the color loss, so as to obtain an image processing model including a line drawing control unit and a color control unit.

12. An image processing device, comprising: A texture feature extraction module, used to obtain a line draft image and extract multi-scale line draft texture features of the line draft image; A noise feature extraction module, used to obtain a preset noise image and extract the multi-scale noise features from the preset noise image; A color acquisition module, used to determine the color guide information corresponding to the line drawing image; an encoding module, configured to encode the noise feature of each scale of the multi-scales with reference to the line drawing texture feature and the color guide information of the scale, so as to obtain a multi-scale image feature; and A decoding module is used to perform multi-scale decoding on the multi-scale image features to obtain a colored image, wherein the colored image has the line draft texture of the line draft image and has the color indicated by the color guide information.

13. A training device for an image processing model, the device comprising: A sample texture feature extraction module is used to obtain a sample line draft image and extract multi-scale sample line draft texture features of the sample line draft image; A sample noise feature extraction module, used to obtain a sample noise image and extract the multi-scale sample noise features from the sample noise image; A sample color acquisition module, used to determine sample color guide information corresponding to the sample line draft image; A sample encoding module, for encoding the sample noise feature of each scale of the multi-scales with reference to the sample line draft texture feature and the sample color guide information of the scale, to obtain a multi-scale sample image feature; A sample decoding module, used for performing multi-scale decoding on the multi-scale sample image features to obtain a predicted color image; A training module is used to obtain a labeled colored image corresponding to the sample line draft image, and to train an initial image processing model based on the predicted colored image and the labeled colored image to obtain an image processing model. The labeled colored image has the line draft texture of the sample line draft image and the color indicated by the sample color guide information.

14. A computer device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method according to any one of claims 1 to 11 when executing the computer program.

15. A computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the method according to any one of claims 1 to 11.

16. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 11 are implemented.

Images