Local image condition coloring method and system and electronic equipment

Abstract

The invention discloses a local image condition coloring method and system and electronic equipment, and the method comprises the steps: carrying out the processing of an original image according to a local region, and obtaining a first mask; determining a second image according to a preset color and the prompt point set of the first mask, and determining a first image according to the prompt point set; and determining a third image according to the original image, the first image and the second image. According to the embodiment of the invention, the local target coloring area can be accurately colored, and the method can be widely applied to the technical field of image processing.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a local image conditional coloring method, system and electronic equipment. Background Art

[0003] In related technologies, point-based prompting methods are used for local conditional coloring of images, hoping to rely on the good receptive field of the transformer to improve the diffusion ability of local area colors. This method of relying entirely on network capabilities for local color diffusion is difficult to implement well because network capabilities are ultimately limited. In addition, current conditional coloring methods mainly use point prompts, that is, manually selecting a few points in the target area, hoping that the color can diffuse around these points to achieve coloring of the entire local area. Therefore, there is a problem of insufficient diffusion in local conditional coloring of images. Summary of the invention

[0004] In view of this, in order to solve one of the above problems, an object of an embodiment of the present invention is to provide a local image conditional coloring method, system and electronic device to achieve accurate coloring of a local target coloring area.

[0005] On the one hand, an embodiment of the present invention provides a local image conditional coloring method, comprising:

[0006] Processing the original image according to the local area to obtain a first mask;

[0007] Determine the second image according to the preset color and the cue point set of the first mask, and determine the first image according to the cue point set;

[0008] A third image is determined according to the original image, the first image and the second image.

[0009] In an optional embodiment, determining the third image according to the original image, the first image and the second image includes:

[0010] Determining a transformation coefficient according to the original image and the preset color;

[0011] A third image after coloring a target coloring area is determined according to the original image, the first image, the second image and the transformation coefficient.

[0012] In an optional embodiment, determining a third image after coloring the target coloring area according to the original image, the first image, the second image and the transformation coefficient includes:

[0013] Determine a fourth image containing chromaticity information according to the brightness image of the original image, the first image, and the second image;

[0014] Adjust the original image according to the transformation coefficient to obtain a fifth image containing brightness information;

[0015] The fourth image and the fifth image are combined to determine a third image after the target coloring area is colored.

[0016] In an optional embodiment, the method further includes:

[0017] The first mask, the original image and the third image are fused to obtain a colored sixth image.

[0018] In an optional embodiment, the processing of the original image according to the local area to obtain the first mask includes:

[0019] Segmenting the original image according to the local area to obtain a local image of a preset size;

[0020] The local image is input into a semantic segmentation model to obtain a first mask.

[0021] In an optional embodiment, the cue point set of the first mask is determined by the following method, including:

[0022] Traversing the first mask according to a preset step size and a preset order;

[0023] The cue point set is determined by using pixel points in the first mask that meet preset requirements.

[0024] In an optional embodiment, determining the first image according to the cue point set includes:

[0025] Initialize an initial image with the same pixel value;

[0026] The pixel values ​​of the cue point set at corresponding positions of the initial image are reset to determine the first image.

[0027] In an optional embodiment, determining the second image according to the preset color and the cue point set of the first mask includes:

[0028] Constructing a seventh image according to the preset color, and performing channel separation on the seventh image to obtain an eighth image containing chromaticity information;

[0029] The color values ​​of the prompt point set of the first mask at the corresponding positions of the eighth image are retained to determine the second image.

[0030] The determining of the transformation coefficient according to the original image and the preset color includes:

[0031] A first operation is performed according to the brightness mean of the original image and the brightness mean of the preset color to determine a transformation coefficient.

[0032] In an optional embodiment, determining a fourth image containing chrominance information from the brightness image of the original image, the first image, and the second image includes:

[0033] splicing the brightness image of the original image, the first image, and the second image to obtain a ninth image;

[0034] The ninth image is input into a coloring model to determine a fourth image containing chromaticity information.

[0035] In an optional embodiment, adjusting the original image according to the transformation coefficient to obtain a fifth image containing brightness information includes:

[0036] The brightness image of the original image is adjusted according to the transformation coefficient to obtain a fifth image containing brightness information.

[0037] In an optional embodiment, the step of fusing the first mask, the original image, and the third image to obtain a colored sixth image includes:

[0038] determining a tenth image within a target shading area based on the first mask and the third image;

[0039] determining an eleventh image outside the target coloring area according to the first mask and the original image;

[0040] The tenth image and the eleventh image are fused to obtain a colored sixth image.

[0041] On the other hand, an embodiment of the present invention provides a local image conditional colorization system, including:

[0042] The first module is used to process the original image according to the local area to obtain a first mask;

[0043] A second module is used to determine the second image according to the preset color and the cue point set of the first mask, and determine the first image according to the cue point set;

[0044] The third module is used to determine a third image according to the original image, the first image and the second image.

[0045] In an optional embodiment, the third module is used to:

[0046] Determining a transformation coefficient according to the original image and the preset color;

[0047] A third image after coloring a target coloring area is determined according to the original image, the first image, the second image and the transformation coefficient.

[0048] In an optional embodiment, the third module is used to:

[0049] Determine a fourth image containing chromaticity information according to the brightness image of the original image, the first image, and the second image;

[0050] Adjust the original image according to the transformation coefficient to obtain a fifth image containing brightness information;

[0051] The fourth image and the fifth image are combined to determine a third image after the target coloring area is colored.

[0052] In an optional embodiment, the local image conditional coloring system further includes a fourth module, and the fourth module is used to:

[0053] The first mask, the original image and the third image are fused to obtain a colored sixth image.

[0054] In an optional embodiment, the first module is used to:

[0055] Segmenting the original image according to the local area to obtain a local image of a preset size;

[0056] The local image is input into a semantic segmentation model to obtain a first mask.

[0057] In an optional embodiment, the second module is used to:

[0058] Traversing the first mask according to a preset step size and a preset order;

[0059] The cue point set is determined by using pixel points in the first mask that meet preset requirements.

[0060] In an optional embodiment, the second module is used to:

[0061] Initialize an initial image with the same pixel value;

[0062] The pixel values ​​of the cue point set at corresponding positions of the initial image are reset to determine the first image.

[0063] In an optional embodiment, the second module is used to:

[0064] Constructing a seventh image according to the preset color, and performing channel separation on the seventh image to obtain an eighth image containing chromaticity information;

[0065] The color values ​​of the prompt point set of the first mask at the corresponding positions of the eighth image are retained to determine the second image.

[0066] In an optional embodiment, the third module is used to:

[0067] A first operation is performed according to the brightness mean of the original image and the brightness mean of the preset color to determine a transformation coefficient.

[0068] In an optional embodiment, the third module is used to:

[0069] splicing the brightness image of the original image, the first image, and the second image to obtain a ninth image;

[0070] The ninth image is input into a coloring model to determine a fourth image containing chromaticity information.

[0071] In an optional embodiment, the third module is used to:

[0072] The brightness image of the original image is adjusted according to the transformation coefficient to obtain a fifth image containing brightness information.

[0073] In an optional embodiment, the fourth module is used to:

[0074] determining a tenth image within a target shading area based on the first mask and the third image;

[0075] determining an eleventh image outside the target coloring area according to the first mask and the original image;

[0076] The tenth image and the eleventh image are fused to obtain a colored sixth image.

[0077] In another aspect, an embodiment of the present invention provides an electronic device, including:

[0078] at least one processor;

[0079] at least one memory for storing at least one program;

[0080] When the at least one program is executed by the at least one processor, the at least one processor implements the above method.

[0081] Implementation of the embodiments of the present invention includes the following beneficial effects: In the present embodiment, first, the original image is processed according to the local area to obtain a first mask, thereby obtaining a first mask including a target coloring area; then, a cue point set is determined according to the first mask, and a first image is determined according to the cue point set to obtain an image including an area formed by the cue point set; then, a second image is determined according to a preset color and a cue point set, so that the preset color is completely diffused and covers the local area; finally, a third image after coloring the target coloring area is determined according to the original image, the first image, and the second image, thereby completing coloring with the preset color in the target coloring area of ​​the first mask, reducing color overflow, and achieving precise coloring of the local target coloring area. BRIEF DESCRIPTION OF THE DRAWINGS

[0082] Figure 1 It is a schematic diagram of the steps of a local image conditional coloring method provided by an embodiment of the present invention;

[0083] Figure 2 is a schematic flow chart of a step of determining a third image provided by an embodiment of the present invention;

[0084] Figure 3 is another schematic flow chart of steps for determining a third image provided by an embodiment of the present invention;

[0085] Figure 4 is a schematic flow chart of a step of determining a first mask provided by an embodiment of the present invention;

[0086] Figure 5 is a schematic flow chart of steps for determining a cue point set provided by an embodiment of the present invention;

[0087] Figure 6 is a schematic flow chart of a step of determining a first image provided by an embodiment of the present invention;

[0088] Figure 7 is a schematic flow chart of a step of determining a second image provided by an embodiment of the present invention;

[0089] Figure 8 is a schematic flow chart of a step of determining a fourth image provided by an embodiment of the present invention;

[0090] Fig. 9 It is a structural schematic diagram of a transfomer network model provided by an embodiment of the present invention;

[0091] Fig.10 is a schematic flow chart of a step of determining a fifth image provided by an embodiment of the present invention;

[0092] Fig.11 is a schematic flow chart of a step of determining a sixth image provided by an embodiment of the present invention;

[0093] Fig.12 This is a schematic diagram of the steps of a method for conditional coloring of a local image in a specific embodiment provided by an embodiment of the present invention;

[0094] Fig.13 is another schematic flow chart of steps for determining a first mask provided by an embodiment of the present invention;

[0095] Fig.14 is an image before and after scaling of a first mask provided by an embodiment of the present invention;

[0096] Fig.15 is a schematic diagram of a prompt point set provided by an embodiment of the present invention;

[0097] Fig.16 is an original image and a multi-channel image provided by an embodiment of the present invention;

[0098] Fig.17 is another schematic flow chart of steps for determining a fourth image provided by an embodiment of the present invention;

[0099] Fig.18 An image before and after brightness adjustment of a transform coefficient provided by an embodiment of the present invention;

[0100] Fig.19 It is a schematic diagram of the steps of another local image conditional coloring method provided by an embodiment of the present invention;

[0101] Fig. 20 It is an image colored by multiple local image conditions provided by an embodiment of the present invention;

[0102] Fig.21 is a structural block diagram of a local image conditional coloring system provided by an embodiment of the present invention;

[0103] Fig. 22 It is a structural block diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0104] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. The step numbers in the following embodiments are only provided for the convenience of explanation and description, and the order between the steps is not limited in any way. The execution order of each step in the embodiment can be adaptively adjusted according to the understanding of those skilled in the art.

[0105] A mask is usually a binary or Boolean image of the same size as the original image, in which the selected area is marked as 1 and the other areas are marked as 0. When applying an operation (such as filtering, edge detection, region extraction, image enhancement, etc.) to an image, a mask can be used to restrict the operation to occur only in a specific area of ​​the image without affecting other parts of the image. For example, in feature extraction applications, masks can be used to extract features only from certain specific areas of the image; in image fusion applications, masks can be used to specify which parts of the pixels should be taken from one image and which parts of the pixels should be taken from another image; in image segmentation applications, masks can be used to mark and distinguish different areas. In practical applications, a mask is usually an array or matrix of the same size as the original image. When using a mask, only the corresponding pixels with a value of 1 in the mask array are processed, and pixels with a value of 0 are ignored.

[0106] The RGB color model is a color standard in the industry. It obtains a variety of colors by changing the three color channels of red (R), green (G), and blue (B) and superimposing them on each other. RGB represents the colors of the three channels: red, green, and blue.

[0107] The Lab color space is a color space model based on people's perception of color. It contains three channel parameters, namely L, a, and b, where L represents brightness, and a and b represent the hue and saturation of the color. The value range of the brightness value L of the Lab color space is [0, 100], and the value range of a and b is [-128, 127].

[0108] like Figure 1 As shown, an embodiment of the present invention provides a local image conditional coloring method, including:

[0109] S100: Process the original image according to the local area to obtain a first mask.

[0110] The local area refers to a part of the original image that contains the target coloring area, the target coloring area refers to the area in the image that needs to be colored, the original image refers to the color before the local conditional coloring, and the first mask refers to the mask of the original image.

[0111] Specifically, an original image and a local area are obtained. The original image can be obtained through external input or direct shooting; the local area can be obtained through interaction, such as through a box selection operation in an interactive interface; the original image is intercepted and processed according to the local area to obtain a local image corresponding to the local area, and the local image is processed to obtain a first mask corresponding to the local area.

[0112] S200: Determine a second image according to a preset color and a cue point set of a first mask, and determine a first image according to the cue point set.

[0113] The preset color refers to the color that requires local conditional coloring, the cue point set refers to the coordinate set formed by multiple cue points in the target coloring area, the first image refers to the mask image corresponding to the local area formed by the cue point set, and the second image refers to the chromaticity image corresponding to the local area formed by the preset color at the cue point set.

[0114] Specifically, first, a cue point set is determined according to the first mask, then a preset color is obtained, and the second image is determined according to the preset color and the cue point set of the first mask, and the first image is determined according to the cue point set. It should be noted that the preset color can be a color in the RGB color space or a color in the Lab color space; the preset color can be obtained through interaction or default settings, such as obtaining the preset color through color selection in an interactive interface. The first mask contains pixel values ​​0 and 1, and the pixel points of the first mask are classified according to the size of the pixel values ​​in the first mask to determine the cue point set, and then the first image is constructed according to the cue point set. If the preset color is a color in the RGB color space, the preset color is converted from the RGB color space to the Lab color space, and then the preset color and the cue point set in the Lab color space are processed to obtain the second image.

[0115] S300: Determine a third image according to the original image, the first image and the second image.

[0116] The original image can be decomposed into a brightness image and a chromaticity image. The brightness image of the original image refers to an image containing the brightness information of the original image, and the chromaticity image of the original image refers to an image containing the chromaticity information of the original image. If the original image is an image in the Lab color space, the brightness image of the original image refers to the image of the original image in the L channel. If the original image is an image in the RGB color space, the original image is converted to the Lab color space to obtain the brightness image of the original image. The third image refers to an image after the target coloring area in the original image is colored with a preset color.

[0117] Specifically, first, a chromaticity image after coloring the target coloring area of ​​the original image is determined based on the brightness image, the first image and the second image of the original image, and then a third image after coloring the target coloring area is synthesized based on the chromaticity image after coloring the target coloring area of ​​the original image and the brightness image of the original image.

[0118] It should be noted that, in the above processing, according to actual application needs, some images or masks can be scaled during the local coloring of the original image. The specific methods of scaling include but are not limited to upsampling or downsampling to further improve the accuracy of local coloring.

[0119] Implementation of the embodiments of the present invention includes the following beneficial effects: In the present embodiment, first, the original image is processed according to the local area to obtain a first mask, thereby obtaining a first mask including a target coloring area; then, a cue point set is determined according to the first mask, and a first image is determined according to the cue point set to obtain an image including an area formed by the cue point set; then, a second image is determined according to a preset color and a cue point set, so that the preset color is completely diffused and covers the local area; finally, a third image after coloring the target coloring area is determined according to the original image, the first image, and the second image, thereby completing coloring with the preset color in the target coloring area of ​​the first mask, reducing color overflow, and achieving precise coloring of the local target coloring area.

[0120] When actually applied, when the preset color is an RGB value with brightness information, the brightness information of the preset color cannot be transferred to the target coloring area, resulting in the final colored color being inconsistent with the preset color RGB.

[0121] In an alternative embodiment, see Figure 2 , determining a third image after the target coloring area is colored according to the original image, the first image, the second image and the transformation coefficient, including:

[0122] S310, determining a transformation coefficient according to the original image and a preset color;

[0123] S320, determining a third image after the target coloring area is colored according to the original image, the first image, the second image and the transformation coefficient.

[0124] The transformation coefficient refers to the correlation coefficient that needs to be adjusted for the brightness information in the original image. Specifically, first, the transformation coefficient is determined based on the brightness information of the original image and the brightness information of the preset color, then, the chromaticity image of the original image after coloring in the target coloring area is determined based on the original image, the first image, and the second image, and the brightness information of the original image is adjusted according to the transformation coefficient to obtain a brightness image after the brightness is adjusted according to the preset color, and finally, the third image after coloring the target coloring area is determined based on the chromaticity image after coloring the target coloring area and the brightness image after the brightness is adjusted, so as to migrate the brightness information of the preset color to the target coloring area, so that the final color after coloring does not look consistent with the preset color RGB, and the accuracy of coloring is improved.

[0125] It should be noted that the brightness information of the L channel of the original image in the Lab color space is calculated. If the original image is an image in the RGB color space, the original image is converted to the Lab color space and the brightness information of the original image is calculated; the brightness information of the L channel of the preset color in the Lab color space is calculated. If the preset color is a color in the RGB color space, the preset color is converted to the Lab color space and the brightness information of the preset color is calculated; then, the transformation coefficient is determined according to the brightness information of the original image and the brightness information of the preset color.

[0126] In an alternative embodiment, see Figure 3 , determining a third image after the target coloring area is colored according to the original image, the first image, the second image and the transformation coefficient, including:

[0127] S410: Determine a fourth image containing chromaticity information according to the brightness image of the original image, the first image, and the second image.

[0128] The fourth image refers to a chromaticity image after the target colored area in the original image is colored with a preset color. If the fourth image is an image in the Lab color space, the fourth image refers to an ab channel image after the target colored area in the original image is colored with a preset color.

[0129] Specifically, the first image contains a mask image corresponding to the local area, and the second image contains a chromaticity image corresponding to the preset color in the local area. The brightness image, the first image and the second image of the original image are fused, so that the target colored area is colored in the chromaticity space according to the preset color, and is fused with the area outside the target colored area of ​​the original image to determine a fourth image containing chromaticity information.

[0130] S420: Adjust the original image according to the transformation coefficient to obtain a fifth image containing brightness information.

[0131] The fifth image refers to the brightness image after the brightness information of the original image is adjusted, that is, the image of the L channel in the Lab color space after the brightness information of the original image is adjusted. Specifically, the transformation coefficient carries the brightness information of the preset color, and the brightness information of the original image is adjusted according to the transformation coefficient to obtain the fifth image containing the brightness information, thereby integrating the brightness information of the preset color into the fifth image through the transformation coefficient. It should be noted that the method for calculating the transformation coefficient is determined according to the actual application, and this embodiment does not make any specific restrictions.

[0132] S430: Combine the fourth image and the fifth image to determine a third image after the target coloring area is colored.

[0133] The fourth image includes the chromaticity information of the target coloring area of ​​the original image after being colored with a preset color, and the fifth image includes the brightness information of the original image after the brightness information is adjusted.

[0134] Specifically, the fourth image of the ab channel of the original image in the Lab color space and the fifth image of the L channel of the original image in the Lab color space are merged to determine the third image of the original image in the Lab color space after the target coloring area is colored. If the original image is an image in the RGB color space, the third image is converted from the Lab color space to the RGB color space.

[0135] In an embodiment of the present invention, the brightness information of a preset color is transferred to a target coloring area through transformation coefficients, thereby optimizing the color inconsistency problem and making the brightness information of a third image after coloring the target coloring area more similar to the brightness information of the preset color, thereby further improving the color accuracy of the target coloring area.

[0136] During the process of coloring the target coloring area of ​​the original image with a preset color, the color of the image outside the target coloring area in the original image may also be changed, causing color difference in the image outside the target coloring area of ​​the original image.

[0137] In an optional embodiment, the method further comprises:

[0138] S500: Fusing the first mask, the original image and the third image to obtain a colored sixth image.

[0139] The sixth image refers to an image obtained by merging the third image after the target coloring area is colored with a preset color and the original image. Fusion refers to weighted processing of images in different areas of the original image to obtain an image of the entire area of ​​the original image.

[0140] Specifically, by processing the first mask and the original image, the color of the original image outside the target coloring area can be retained; by processing the first mask and the third image, the image after coloring the target coloring area can be retained; by fusing the first mask, the original image and the third image, both the image after coloring the target coloring area and the image outside the target coloring area of ​​the original image can be retained, thereby obtaining a colored sixth image.

[0141] In an embodiment of the present invention, by fusing the first mask, the original image and the third image, the image color of the original image outside the target coloring area and the image after the target coloring area is colored are retained, thereby reducing the image color difference of the original image outside the target coloring area and improving the color accuracy of the original image after the target coloring area is colored.

[0142] In an alternative embodiment, see Figure 4 , performing image processing on the original image according to the local area to obtain a first mask corresponding to the local area, including:

[0143] S110, segmenting the original image according to the local area to obtain a local image of a preset size;

[0144] S120: Input the local image into a semantic segmentation model to obtain a first mask.

[0145] A local image refers to an image segmented from the original image based on a local region. It should be noted that the shape of the local region is determined according to the actual application, such as a rectangular or circular local region, etc. The size of the local image is determined according to the input requirements of the semantic segmentation model, and the size of the local image can be different from the size of the image obtained directly based on the local region segmentation. The image obtained directly based on the local region segmentation can be scaled to obtain the size of the local image; the specific structure of the semantic segmentation model is determined according to the actual application, such as a neural network model.

[0146] Specifically, the original image is segmented according to the local area, and if necessary, the segmented image is scaled to obtain a local image of a preset size, and the local image is input into the semantic segmentation model to obtain a binary image, and if necessary, the binary image is scaled to obtain a first mask.

[0147] In a specific embodiment, the local area is a rectangular frame, and the semantic segmentation model is an image semantic segmentation model whose backbone network is a U-net structure. First, the original image is segmented by an external rectangular frame to obtain an image of size w1*h1*c1, wherein w1, h1, and c1 represent the width, height, and number of channels of the image respectively, and the image of size w1*h1*c1 contains the target colored area, and the image of size w1*h1*c1 is scaled to a local image of size 256*256*3, and the local image is input into the semantic segmentation model to obtain a probability map of size 256*256*1, wherein the numerical range of the probability is [0, 1], and the probability map is scaled to the size of w1*h1, and it is binarized by a threshold, and the threshold is set to 0.6 to obtain a binary map of w1*h1*1, and the binary map is expanded to the size of the original image according to the coordinate information of the external rectangular frame to obtain the first mask corresponding to the local area.

[0148] The U-net structure is a bilaterally symmetrical structure, consisting of three parts: the encoder path on the left side of the network, the decoder path on the right side, and the jump connection path. The encoder path is also called a contraction network. Through multiple continuous downsampling operations, the image size is reduced and the number of channels is doubled. During the downsampling process, the network can obtain the shallow feature information of the image. The decoder path is also called an expansion network. Through multiple continuous upsampling operations, the image size is increased and the number of channels is halved. During this continuous upsampling process, the network can obtain the deep feature information of the image. U-net sets up a jump connection path between the encoder and decoder of each layer to connect the two, helping the model to splice image features of the same layer and the same size. In this architecture, the encoder is responsible for feature extraction, and the decoder restores the original resolution. The jump connection path helps the model successfully fuse deep feature information and shallow feature information, realize the effective fusion of coarse-grained features and fine-grained features, provide the network with more refined image features, and help the network achieve pixel-level semantic segmentation.

[0149] In an alternative embodiment, see Figure 5 , determining a cue point set according to the first mask, including:

[0150] S210, traversing the first mask according to a preset step size and a preset order;

[0151] S220: Determine a cue point set from the pixel points in the first mask that meet preset requirements.

[0152] It should be noted that the preset step size, preset order and preset requirement are determined according to actual applications, and this embodiment does not impose specific restrictions. For example, the preset step size is set to 0.5 or 0.3, the preset order is set to from left to right or from right to left, and the preset requirement is that the pixel is greater than or less than the set value. It can be understood by those skilled in the art that the first mask can be scaled if necessary.

[0153] In a specific embodiment, the first mask of the local area is scaled to obtain a second mask image of w*h. The second mask image is traversed from the starting point (0, 0) from left to right with a preset step size as step, and (1*step, 0), (2*step, 0), ...., (w / step, h / step) are traversed in sequence. If the current pixel value is greater than 0, the current coordinate is added to the set of prompt points.

[0154] In an alternative embodiment, see Figure 6 , determining a first image according to the cue point set, including:

[0155] S230, initializing an initial image with the same pixel value;

[0156] S240: Reset the pixel values ​​of the prompt point set at the corresponding positions of the initial image to determine the first image.

[0157] The initial image refers to an image with the same pixel values, such as an image with all pixel values ​​being 0 or all pixel values ​​being 1. Resetting means resetting the pixel points with the original element value of 0 to 1, or resetting the pixel points with the original element value of 1 to 0.

[0158] In a specific embodiment, a cue point mask image of size w*h is constructed, an image of all zeros is initialized, and then all pixels of corresponding coordinates in the cue point set are set to 1.

[0159] In an alternative embodiment, see Figure 7 , determining a second image according to a preset color and a cue point set, including:

[0160] S310, constructing a seventh image according to preset colors, and performing channel separation on the seventh image to obtain an eighth image containing chromaticity information;

[0161] S320: retain the color values ​​of the prompt point set of the first mask at the corresponding positions of the eighth image to determine the second image.

[0162] The seventh image refers to an image of a preset color in a Lab color space, and the eighth image refers to an image of the seventh image in an ab channel.

[0163] Specifically, if the preset color is a color in the RGB color space, the preset color is converted from the RGB color space to the Lab color space to obtain the seventh image, and the seventh image is subjected to channel separation to obtain the eighth image whose ab channel contains chromaticity information; if the preset color is a color in the Lab color space, the seventh image is subjected to channel separation directly to obtain the eighth image whose ab channel contains chromaticity information.

[0164] In an alternative embodiment, see Figure 8 , determining a fourth image containing chrominance information from the brightness image, the first image, and the second image of the original image, including:

[0165] S411, splicing the brightness image of the original image, the first image, and the second image to obtain a ninth image;

[0166] S412: Input the ninth image into the coloring model to determine a fourth image containing chromaticity information.

[0167] The brightness image of the original image refers to the image of the L channel of the original image in the Lab color space. If the original image is in the color of the RGB color space, the preset color is converted from the RGB color space to the Lab color space, and the channels are separated to obtain the brightness image of the original image. Stitching refers to connecting multi-channel images. The ninth image refers to the multi-channel image after the brightness image, the first image and the second image of the original image are spliced. It should be noted that the coloring model is determined according to the actual application, and this embodiment does not make specific restrictions, for example, the Transformer network model. It can be understood by those skilled in the art that before the brightness image, the first image and the second image of the original image are spliced, the brightness image, the first image and the second image need to be adjusted to the same size.

[0168] In a specific embodiment, the coloring model adopts a Transformer network model, scales the brightness image, the first image, and the second image of the original image to obtain images of the same size, and then splices them to obtain a ninth image; the ninth image is input into the Transformer network model to obtain a fourth image whose L channel in the Lab space contains chromaticity information.

[0169] In a specific embodiment, the specific structure of the Transformer network model is shown in Fig. 9 , the Transformer network is stacked with N layers of encoders (for example, 4 layers) and M layers of decoders (for example, 4 layers), and finally the prediction head of the model is composed of a feedforward neural network to output the final detection box category and bounding box; among them, the multi-head attention mechanism structure adopts an attention mechanism with L heads (for example, 8 heads), residual connection and normalization represent residual connection to prevent network degradation, normalization represents layer standardization, which is used to normalize the activation values ​​of each layer, and the feedforward neural network is a fully connected layer.

[0170] Among them, the input of the encoder includes the feature vector information of the image features and the spatial position encoding information generated by the spatial position encoding. The position of the feature vector information of the image features is limited by the spatial position encoding information. The attention mechanism integrates the artistic conception of the image into the feature vector information of the image features to help learn the dependencies between the various parts of the image. The calculation of the attention mechanism introduces three vectors: V (Value), K (Key) and Q (Query). These three vectors are the results of multiplying the feature vector of the input image features with the set Wq, Wk, and Wv matrices. The corresponding Wq, Wk, and Wv matrices are randomly initialized and continuously optimized through model learning. The Query vector represents the vector that will be queried and calculated later, the Key vector represents the vector that will be queried and calculated later, and the Value vector represents the current actual feature; the Query vector of each part of the image is calculated by the inner product with the Key vector of the entire sequence word. The larger the inner product, the closer the relationship between the two parts. The greater the correlation between the two parts; the constant obtained by the inner product is divided by the square root of the matrix dimension, and then normalized by the softmax function to obtain the corresponding score, which indicates the correlation between the current part and each part of the entire image; the scores calculated for each part and all parts in the image are used as weights and weighted cumulative calculations are performed with the Value vector corresponding to each part to obtain the output Z of the current part; in actual application, in order to improve the calculation rate of the model, it is often calculated in the form of a matrix. The model optimizes the weight parameters of the Wq, Wk, and Wv matrices through training, and then performs weighted calculations with the Wv matrix through the softmax function. The multi-head attention mechanism extracts different features through multiple sets of Query, Key, and Value matrices, and then concatenates the extracted feature vectors to obtain the final output feature vector. For example, 8 sets of Query, Key, and Value matrices are used for feature extraction to obtain 8 input feature matrices Z. The vector calculated by the encoder output maintains the same format as the starting input vector. Therefore, the output of the encoder can still be used as the input sequence of the next layer of encoder for calculation again. The encoding layers are stacked many times and feature vectors are continuously extracted to form the encoder end of the Transformer model. Due to the feature extraction of multiple layers of attention mechanism, in order to retain the original features and ensure that the vector output by the encoder contains the original features, a residual connection is designed in the model. The initial input vector is added to the output vector calculated by the attention mechanism without any transformation, and then normalized as its output vector.

[0171] The decoder input includes object queries, which represent the image features to be predicted. The encoder converts the feature vector output from the top layer into K and V matrices through the calculation of the multi-layer attention mechanism and passes them to the decoder part. Together with the output Q matrix of the multi-head attention production layer in the decoder, it is input and output to the next multi-attention mechanism for calculation. The image feature vector output by the last layer of the decoder is input into two feedforward neural networks to map the detection category and bounding box.

[0172] In an alternative embodiment, see Fig.10 , determining a transformation coefficient according to the original image and the preset color, adjusting the original image according to the transformation coefficient, and obtaining a fifth image containing brightness information, including:

[0173] S421, performing a first operation according to the brightness mean of the original image and the brightness mean of the preset color to determine a transformation coefficient;

[0174] S422. Adjust the brightness image of the original image according to the transformation coefficient to obtain a fifth image containing brightness information.

[0175] The brightness mean of the original image refers to the mean of all pixels of the original image in the L channel of the Lab color space, and the brightness mean of the preset color refers to the mean of all pixels of the preset color in the L channel of the Lab color space. The fifth image refers to the image after the brightness adjustment of the original image. It should be noted that the first operation is determined according to the actual application, and this embodiment does not make specific restrictions, such as determining according to experimental comparison or empirical values. The transformation coefficient is determined according to the first operation.

[0176] In a specific embodiment, first, the brightness mean a of the preset color in the L channel and the brightness mean b of the original image in the L channel are calculated respectively, then, their logarithmic averages are calculated to obtain a gamma coefficient, and finally, the L channel of the original image is gamma transformed according to the current gamma coefficient to obtain the L channel image M after brightness adjustment.

[0177] In a specific embodiment, the specific calculation formula is as follows:

[0178] gamma = log(b) / log(a)

[0179] L1=min(((L0 / 255)^gamma)*255,255)

[0180] Among them, gamma represents the gamma coefficient, b represents the average brightness of the original image, a represents the average brightness of the preset color, L0 represents the brightness value of the original image, and L1 represents the brightness value of the fifth image.

[0181] In an alternative embodiment, see Fig.11, the first mask, the original image and the third image are fused to obtain a colored sixth image, including:

[0182] S510, determining a tenth image within the target coloring area according to the first mask and the third image;

[0183] S520, determining an eleventh image outside the target coloring area according to the first mask and the original image;

[0184] S530: Fusing the tenth image and the eleventh image to obtain a colored sixth image.

[0185] The tenth image refers to the image after the target coloring area is colored, the eleventh image refers to the image of the original image outside the target coloring area, and the sixth image refers to the image of the original image after the target area is colored.

[0186] In a specific embodiment, a merging calculation is performed based on the first mask and the third image to determine the tenth image within the target coloring area; a negation operation is performed based on the first mask and the original image to determine the eleventh image outside the target coloring area; and the tenth image and the eleventh image are added and fused to obtain the colored sixth image.

[0187] In a specific embodiment, the calculation formula is as follows:

[0188] R0=Mask*G+(1-Mask)*Origin

[0189] R1=min(R0,255)

[0190] R0 represents the image obtained by fusing the tenth image and the eleventh image, Mask represents the first mask, G represents the third image, Origin represents the original image, and R1 represents the colored sixth image.

[0191] In a specific embodiment, see Fig.12 , the process of the local conditional coloring method of the image is as follows:

[0192] Step S1: Convert a preset color value whose color space is RGB into a Lab space, where the Lab space includes an L channel and an ab channel, where the L channel contains brightness information and the ab channel contains chromaticity information.

[0193] Step S2: construct a Lab three-channel of size w*h*c based on the preset color value of the Lab space, and separate the L channel and the ab channel, where w, h, and c represent the image length, width, and number of channels, respectively. The values ​​of w, h, and c are determined according to the actual application, and are not specifically limited in this embodiment. For example, the values ​​of w, h, and c are 224, 224, and 3, respectively. Wherein, w and h are determined according to the resolution of the coloring model.

[0194] Step S3: Obtain the first mask image corresponding to the local colored area through the image semantic segmentation model. Fig.13 , Fig.13 (a) represents the original image. An external rectangular frame is used to segment the original image to obtain an image of size w1*h1*c1, where w1, h1, and c1 represent the width, height, and number of channels of the image respectively. The image of size w1*h1*c1 contains the target colored area. Fig.13 (b) represents the image corresponding to the local area; the image of size w1*h1*c1 is scaled to a local image of size 256*256*3, and the local image is input into the semantic segmentation model. Fig.13 (c) represents the semantic segmentation model of the U-net network structure, and obtains a 256*256*1 probability map and a w1*h1*1 binary map. Fig.13 (d) represents the binary image corresponding to the target coloring area; the binary image is expanded to the original image size according to the coordinate information of the circumscribed rectangular frame to obtain the first mask corresponding to the local area. Fig.13 In the figure (e), it indicates the first mask.

[0195] Step S4: Scale the first mask image corresponding to the local area to obtain a mask image of w*h, where w and h represent the width and height of the image, representing the resolution, and w and h are 224 and 224 respectively. Fig.14 , Fig.14 (a) represents the first mask image before scaling. Fig.14 (b) shows the first mask image after scaling.

[0196] Step S5: traverse the mask image with a preset step length to obtain a large number of cue point coordinates to form a cue point set.

[0197] Step S6: Construct a cue point mask image of size w*h, initialize an all-zero image, and then set all pixels of corresponding coordinates in the cue point set to 1 to obtain the cue point set. Fig.15 , where the white dot corresponds to the prompt point position.

[0198] Step S7: modify the preset color ab channel obtained in step S2 according to the position of the prompt point. The original color value is retained at the prompt point position, otherwise it is 0.

[0199] Step S8: Convert the original image to a color space to obtain an image in the corresponding Lab color space. Fig.16 , Fig.16 (a) represents the original image. Fig.16 (b) represents the brightness image of the L channel of the original image in the Lab color space. Fig.16(c) represents the chromaticity image of the a channel. Fig.16 (d) in the middle shows the chromaticity image of the b channel.

[0200] Step S9: Separate the Lab space image channel and obtain the L channel image representing brightness information.

[0201] Step S10: scaling the L channel of the original image to obtain an image of a uniform size of w*h.

[0202] Step S11: stitching the images obtained in step S6, step S7 and step S10 to obtain an image G with a size of w*h*4.

[0203] Step S12: Use the image G as the input data of the coloring model. After the coloring model completes the inference prediction, an image of size w*h*2 can be obtained. Fig.17 , Fig.17 (a) represents the ninth image G after the brightness image of the original image, the first image and the second image are stitched together. Fig.17 (b) represents the coloring model. Fig.17 The image of w*h*2 is output in (c). The ninth image G is composed of the ab channel map of the preset color, the L channel map corresponding to the original image, and the mask map constructed based on the prompt point. The final size is w*h*4, where w and h are 224 and 224, respectively. The ab channel map of the preset color only retains the value of the prompt point position, and the others are set to 0. The main structure of the coloring model is based on the transformer. First, feature extraction is performed based on the convolutional network to form the input token of the transformer. Then, a multi-layer convolutional network is connected behind the transformer for decoding. Finally, the a and b channels predicted by the model are output, with sizes of w*h*1 respectively.

[0204] Step S13: scaling the image obtained in step 12 according to the width and height information of the original image so that its width and height are consistent with those of the original image. The fourth image H obtained in this step represents the predicted ab channel data.

[0205] Step S14: Calculate the L channel mean a of the preset color and the L channel mean b corresponding to the original image, then calculate their logarithmic mean to obtain a gamma coefficient, and finally perform gamma transformation on the L channel of the original image according to the current gamma coefficient, so as to obtain the adjusted L channel fifth image M. Fig.18 , Fig.18 (a) shows the image before adjustment. Fig.18 (b) shows the adjusted image, where the gamma coefficient is 0.7798.

[0206] Step S15: Merge the channels of the fifth image M and the fourth image H obtained in step S14 and step S13 in Lab order to obtain a coloring result image in Lab color space.

[0207] Step S16: Perform color space conversion on the coloring result image of step 15 to obtain a coloring result image (third image) G in RGB space.

[0208] Step S17: weighted fusion is performed on the result image (third image) G of step 16 and the original image Origin, and the fusion weight is the first mask image Mask obtained in step S3, and finally the result image (sixth image) R after local conditional coloring of the image is obtained. Fig.19 , Fig.19 (a) represents the original image. Fig.19 (b) represents the first mask corresponding to the local area. Fig.19 (c) shows the fourth image after being colored by the coloring model. Fig.19 Middle (d) shows the six images after the target coloring area is colored.

[0209] In some specific embodiments, see Fig. 20 , Fig. 20 (a) represents the original image. Fig. 20 (b) to (e) show images after the target coloring area is colored with different preset colors.

[0210] See also Fig.21 , an embodiment of the present invention provides a local image conditional colorization system, comprising:

[0211] The first module is used to process the original image according to the local area to obtain a first mask;

[0212] A second module is used to determine the second image according to the preset color and the cue point set of the first mask, and determine the first image according to the cue point set;

[0213] The third module is used to determine a third image according to the original image, the first image and the second image.

[0214] In an optional embodiment, the third module is used to:

[0215] Determine the transformation coefficient according to the original image and the preset color;

[0216] A third image after the target coloring area is colored is determined according to the original image, the first image, the second image and the transformation coefficient.

[0217] In an optional embodiment, the third module is used to:

[0218] Determine a fourth image containing chromaticity information according to the brightness image of the original image, the first image, and the second image;

[0219] Determining a transformation coefficient according to brightness information of the original image and brightness information of a preset color, and adjusting the original image according to the transformation coefficient to obtain a fifth image containing brightness information;

[0220] The fourth image and the fifth image are combined to determine a third image after the target coloring area is colored.

[0221] In an optional embodiment, the local image conditional coloring system further includes a fourth module, and the fourth module is used to:

[0222] The first mask, the original image and the third image are fused to obtain a colored sixth image.

[0223] In an optional embodiment, the first module is used to:

[0224] Segment the original image according to the local area to obtain a local image of a preset size;

[0225] The local image is input into the semantic segmentation model to obtain the first mask.

[0226] In an optional embodiment, the second module is used to:

[0227] Traversing the first mask according to a preset step size and a preset order;

[0228] The pixel points in the first mask that meet the preset requirements are used to determine the prompt point set.

[0229] In an optional embodiment, the second module is used to:

[0230] Initialize an initial image with the same pixel value;

[0231] The pixel values ​​of the cue point set at the corresponding positions of the initial image are reset to determine the first image.

[0232] In an optional embodiment, the second module is used to:

[0233] Constructing a seventh image according to preset colors, performing channel separation on the seventh image, and obtaining an eighth image containing chromaticity information;

[0234] The color values ​​of the prompt point set at the corresponding positions of the eighth image are retained to determine the second image.

[0235] In an optional embodiment, the third module is used to:

[0236] A first operation is performed based on the brightness mean of the original image and the brightness mean of the preset color to determine the transformation coefficient.

[0237] In an optional embodiment, the third module is used to:

[0238] The brightness image, the first image and the second image of the original image are concatenated to obtain a ninth image;

[0239] The ninth image is input into the coloring model to determine the fourth image including chromaticity information.

[0240] In an optional embodiment, the third module is used to:

[0241] The brightness image of the original image is adjusted according to the transformation coefficient to obtain a fifth image containing brightness information.

[0242] In an optional embodiment, the fourth module is used to:

[0243] determining a tenth image within the target shading area based on the first mask and the third image;

[0244] determining an eleventh image outside the target coloring area according to the first mask and the original image;

[0245] The tenth image and the eleventh image are fused to obtain the colored sixth image.

[0246] It can be seen that the contents of the above method embodiments are all applicable to the present system embodiments, the functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0247] See also Fig. 22 , an embodiment of the present invention provides an electronic device, including:

[0248] at least one processor;

[0249] at least one memory for storing at least one program;

[0250] When at least one program is executed by at least one processor, the at least one processor implements the above method.

[0251] Among them, the memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. The memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a remote memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0252] It can be seen that the contents of the above method embodiments are all applicable to the electronic device embodiment. The functions specifically implemented by the electronic device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0253] It is understood that all or some steps and systems in the disclosed method above can be implemented as software, firmware, hardware and appropriate combinations thereof. Some physical components or all physical components can be implemented as software executed by a processor, such as a central processing unit, a digital signal processor or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, and the computer-readable medium can include a computer storage medium (or a non-transitory medium) and a communication medium (or a temporary medium). As known to those of ordinary skill in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical disk storage, magnetic cassette, magnetic tape, disk storage or other magnetic storage device, or any other medium that can be used to store desired information and can be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication media typically embodies computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0254] The above is a specific description of the preferred implementation of the present invention, but the invention is not limited to the embodiments. Those skilled in the art may make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.

Claims

1. A local image conditional coloring method, characterized in that: include: Processing the original image according to the local area to obtain a first mask; Determine the second image according to the preset color and the cue point set of the first mask, and determine the first image according to the cue point set; A third image is determined according to the original image, the first image and the second image.

2. The method according to claim 1, characterized in that The determining of the third image according to the original image, the first image and the second image comprises: Determining a transformation coefficient according to the original image and the preset color; A third image after coloring a target coloring area is determined according to the original image, the first image, the second image and the transformation coefficient.

3. The method according to claim 2, characterized in that Determining a third image after coloring a target coloring area according to the original image, the first image, the second image, and the transformation coefficients includes: Determine a fourth image containing chromaticity information according to the brightness image of the original image, the first image, and the second image; Adjust the original image according to the transformation coefficient to obtain a fifth image containing brightness information; The fourth image and the fifth image are combined to determine a third image after the target coloring area is colored.

4. The method according to any one of claims 1 to 3, characterized in that: The method further comprises: The first mask, the original image and the third image are fused to obtain a colored sixth image.

5. The method according to claim 1, characterized in that The processing of the original image according to the local area to obtain a first mask includes: Segmenting the original image according to the local area to obtain a local image of a preset size; The local image is input into a semantic segmentation model to obtain a first mask.

6. The method according to claim 1, characterized in that The cue point set of the first mask is determined by the following method: Traversing the first mask according to a preset step size and a preset order; The cue point set is determined by using pixel points in the first mask that meet preset requirements.

7. The method according to claim 1, characterized in that The determining the first image according to the cue point set comprises: Initialize an initial image with the same pixel value; The pixel values ​​of the cue point set at corresponding positions of the initial image are reset to determine the first image.

8. The method according to claim 1, characterized in that The step of determining the second image according to the preset color and the cue point set of the first mask includes: Constructing a seventh image according to the preset color, and performing channel separation on the seventh image to obtain an eighth image containing chromaticity information; The color values ​​of the prompt point set of the first mask at the corresponding positions of the eighth image are retained to determine the second image.

9. The method according to claim 2, characterized in that: The determining of the transformation coefficient according to the original image and the preset color includes: A first operation is performed according to the brightness mean of the original image and the brightness mean of the preset color to determine a transformation coefficient.

10. The method according to claim 3, characterized in that The step of determining a fourth image containing chromaticity information by using the brightness image of the original image, the first image, and the second image comprises: splicing the brightness image of the original image, the first image, and the second image to obtain a ninth image; The ninth image is input into a coloring model to determine a fourth image containing chromaticity information.

11. The method according to claim 3, characterized in that The step of adjusting the original image according to the transformation coefficient to obtain a fifth image containing brightness information includes: The brightness image of the original image is adjusted according to the transformation coefficient to obtain a fifth image containing brightness information.

12. The method according to claim 4, characterized in that The step of fusing the first mask, the original image and the third image to obtain a colored sixth image includes: determining a tenth image within a target shading area based on the first mask and the third image; determining an eleventh image outside the target coloring area according to the first mask and the original image; The tenth image and the eleventh image are fused to obtain a colored sixth image.

13. A local image conditional coloring system, characterized in that: include: The first module is used to process the original image according to the local area to obtain a first mask; A second module is used to determine the second image according to the preset color and the cue point set of the first mask, and determine the first image according to the cue point set; The third module is used to determine a third image according to the original image, the first image and the second image.

14. An electronic device, characterized in that: include: at least one processor; at least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the method according to any one of claims 1 to 12.

Images