A method and device for coloring line draft images

Abstract

The present application discloses a method and device for coloring line draft images, which are used to accurately color line draft images and avoid color bleeding between adjacent areas. The present application method includes: obtaining all line draft images to be colored; dividing each line draft image into multiple areas according to a preset image segmentation algorithm; selecting at least one line draft image as a reference image, and using line draft images other than the reference image as target images; obtaining the label of each area of ​​the reference image; matching all areas in the line draft image in pairs; determining the label of each area in the target image according to the matching results and the label of the area of ​​the reference image; and coloring all line draft images according to the preset correspondence between the label and the area color.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a method and device for coloring a line drawing image. Background Art

[0002] With the vigorous development of the game industry and the comics industry, the update speed of game products and comics products is gradually accelerating. The update of game products and comics products mainly includes two processes: drawing line draft images and coloring line draft images.

[0003] A game product or comic product generally contains a large number of line drawings. Based on the need to color a large number of line drawings, a current method is to use a neural network to train the line drawings to be colored in order to complete the coloring of the line drawings.

[0004] However, since the method of using neural network training for colorization is not mature enough, problems such as color bleeding between adjacent areas often occur in the obtained color images, resulting in inaccurate colorization results. Summary of the invention

[0005] The embodiments of the present application provide a method and device for coloring a line draft image, which are used to accurately color the line draft image and avoid color bleeding between adjacent areas.

[0006] In view of this, the first aspect of the present application provides a method for coloring a line drawing image, comprising:

[0007] Get all the line drawings to be colored;

[0008] Each line drawing image is divided into multiple regions according to a preset image segmentation algorithm;

[0009] Selecting at least one line drawing image as a reference image, and using line drawing images other than the reference image as target images;

[0010] Obtaining a label for each region of the reference image;

[0011] Perform pairwise matching on all regions in all the line draft images;

[0012] Determine a label of each region in the target image according to the matching result and the label of the region of the reference image;

[0013] All the line draft images are colored according to the preset correspondence between the labels and the area colors.

[0014] In a possible design, in a first implementation of the first aspect of the embodiments of the present application,

[0015] Selecting one of the unlabeled regions for labeling according to the obtained label of the unlabeled region includes:

[0016] Calculate the risk of labeling each unlabeled region according to the obtained labels of the unlabeled regions according to a preset error estimation algorithm;

[0017] According to the obtained labels of the unlabeled regions, the unlabeled regions with the lowest risk are selected for labeling.

[0018] In a possible design, in a second implementation of the first aspect of the embodiments of the present application,

[0019] After each iterative calculation, before selecting one of the unlabeled regions for labeling according to the obtained labels of the unlabeled regions, the following steps are also included:

[0020] The obtained labels of the unlabeled regions are optimized according to a preset mixed integer quadratic programming algorithm and an adjacency relationship between labels, wherein the adjacency relationship between the labels corresponds to the adjacency relationship between the regions.

[0021] A second aspect of the present application provides a coloring device for a line drawing image, comprising:

[0022] A first acquisition unit, used to acquire all line drawing images to be colored;

[0023] A segmentation unit, used for segmenting each line drawing image into a plurality of regions according to a preset image segmentation algorithm;

[0024] A first selection unit, configured to select at least one line draft image as a reference image, and to use line draft images other than the reference image as target images;

[0025] A second acquisition unit, used to acquire a label of each area of ​​the reference image;

[0026] A matching unit, used for performing pairwise matching on all regions in all the line draft images;

[0027] a label determination unit, configured to determine a label of each region in the target image according to a matching result and a label of a region of the reference image;

[0028] The coloring unit is used to color all the line draft images according to the preset corresponding relationship between the label and the area color.

[0029] In a possible design, in a first implementation of the second aspect of the embodiments of the present application,

[0030] The matching unit is used for:

[0031] Extracting shape features of each area of ​​the line drawing image according to a preset shape context internal distance algorithm;

[0032] All regions in the line drawing image are matched pairwise according to the shape features.

[0033] In a possible design, in a second implementation of the second aspect of the embodiment of the present application,

[0034] The matching unit is used for:

[0035] For each region of the line drawing image, taking a plurality of points on the region boundary;

[0036] The point features of all points on the region boundary are obtained as the shape features of the region. The point features of a point include the shortest distance between the current point and other points on the region boundary within the region, and the angle between the shortest distance line between the current point and other points on the region boundary and the tangent line of the current point;

[0037] All regions in the line drawing image are matched pairwise according to the shape features.

[0038] In a possible design, in a third implementation of the second aspect of the embodiment of the present application,

[0039] The matching unit is used for:

[0040] For each region of the line drawing image, taking a plurality of points on the region boundary;

[0041] The point features of all points on the region boundary are obtained as the shape features of the region. The point features of a point include the shortest distance between the current point and other points on the region boundary within the region, and the angle between the shortest distance line between the current point and other points on the region boundary and the tangent line of the current point;

[0042] For each area of ​​the line drawing image, obtaining a plurality of shortest distance reference intervals and a plurality of angle reference intervals;

[0043] Establish a point vector corresponding to each point one by one, wherein the dimension of the point vector is equal to the product of the number of the shortest distance reference intervals and the number of the angle reference intervals, and the value of the point vector in each dimension is equal to the number of points located in one of the shortest distance reference intervals and one of the angle reference intervals among other points on the same region boundary as the current point;

[0044] Calculate the matching value of each point between two regions in the line drawing image according to the point vector;

[0045] Matching values ​​between regions in the line drawing image are calculated based on the matching values ​​of the points.

[0046] In a possible design, in a fourth implementation of the second aspect of the embodiments of the present application,

[0047] The matching unit is used for:

[0048] For each region of the line drawing image, taking a plurality of points on the region boundary;

[0049] The point features of all points on the region boundary are obtained as the shape features of the region. The point features of a point include the shortest distance between the current point and other points on the region boundary within the region, and the angle between the shortest distance line between the current point and other points on the region boundary and the tangent line of the current point;

[0050] For each area of ​​the line drawing image, obtaining a plurality of shortest distance reference intervals and a plurality of angle reference intervals;

[0051] Establish a point vector corresponding to each point one by one, wherein the dimension of the point vector is equal to the product of the number of the shortest distance reference intervals and the number of the angle reference intervals, and the value of the point vector in each dimension is equal to the number of points located in one of the shortest distance reference intervals and one of the angle reference intervals among other points on the same region boundary as the current point;

[0052] Calculate the matching value of each point between two regions in the line drawing image according to the point vector;

[0053] Setting the point matching value between any two areas in the line drawing image that is greater than a threshold as a preset penalty value;

[0054] For any two regions in the line drawing image, all matching point combinations that meet a preset condition are obtained, wherein the matching point combinations include a group of points selected from a plurality of points on a region boundary and a group of points selected from a plurality of points on another region boundary, the two groups of points correspond one to one on the respective region boundaries according to a preset direction and the matching values ​​of the two corresponding points are both less than a threshold value, and the preset direction is a clockwise direction or a counterclockwise direction;

[0055] For each matching point combination, obtaining the sum of each matching value in the matching point group and the corresponding total penalty value, wherein the total penalty value is the product of the preset penalty value and the penalty amount, and the penalty amount is the difference between the number of the multiple points and the number of points in a group of points;

[0056] The minimum value of the sum is taken as the matching value of the two regions.

[0057] In a possible design, in a fifth implementation of the second aspect of the embodiments of the present application,

[0058] The matching unit is used for:

[0059] For each region of the line drawing image, taking a plurality of points on the region boundary;

[0060] The point features of all points on the region boundary are obtained as the shape features of the region. The point features of a point include the shortest distance between the current point and other points on the region boundary within the region, and the angle between the shortest distance line between the current point and other points on the region boundary and the tangent line of the current point;

[0061] For each area of ​​the line drawing image, obtain the maximum value of the shortest distance between points on the boundary of the area and the maximum value of the angle between the shortest distance line and the tangent line;

[0062] The maximum value of the shortest distance is evenly divided according to the number of preset shortest distance reference intervals to obtain multiple shortest distance reference intervals;

[0063] The maximum value of the angle is evenly divided according to the number of preset angle reference intervals to obtain a plurality of angle reference intervals;

[0064] Establish a point vector corresponding to each point one by one, wherein the dimension of the point vector is equal to the product of the number of the shortest distance reference intervals and the number of the angle reference intervals, and the value of the point vector in each dimension is equal to the number of points located in one of the shortest distance reference intervals and one of the angle reference intervals among other points on the same region boundary as the current point;

[0065] Calculate the matching value of each point between two regions in the line drawing image according to the point vector;

[0066] Setting the point matching value between any two areas in the line drawing image that is greater than a threshold as a preset penalty value;

[0067] For any two regions in the line drawing image, all matching point combinations that meet a preset condition are obtained, wherein the matching point combinations include a group of points selected from a plurality of points on a region boundary and a group of points selected from a plurality of points on another region boundary, the two groups of points correspond one to one on the respective region boundaries according to a preset direction and the matching values ​​of the two corresponding points are both less than a threshold value, and the preset direction is a clockwise direction or a counterclockwise direction;

[0068] For each matching point combination, obtaining the sum of each matching value in the matching point group and the corresponding total penalty value, wherein the total penalty value is the product of the preset penalty value and the penalty amount, and the penalty amount is the difference between the number of the multiple points and the number of points in a group of points;

[0069] The minimum value of the sum is taken as the matching value of the two regions.

[0070] In a possible design, in a sixth implementation of the second aspect of the embodiments of the present application,

[0071] The label determination unit is used for:

[0072] The label of the region of the reference image having the smallest matching value with the region of the target image is used as the label of the region of the target image.

[0073] In a possible design, in a seventh implementation of the second aspect of the embodiments of the present application,

[0074] The label is a number;

[0075] The label determination unit is used for:

[0076] For any two areas in the line drawing image, obtaining a ratio of a matching value to the number of points in a group of points in the matching point combination corresponding to the matching value as a weight;

[0077] The first energy function The minimum is the objective function that performs at least one iterative calculation. After each iterative calculation, one of the unlabeled regions is selected for labeling according to the obtained label of the unlabeled region, and the next iterative calculation is performed after labeling. After the number of iterations reaches the preset number, all the remaining unlabeled regions are labeled according to the label of the unlabeled region obtained by the last iterative calculation. The unlabeled region is the region of the target image whose label is not determined. f(j) represents the label of the jth region, f(i) represents the label of the ith region, and w i,j Represents the weights corresponding to the i-th region and the j-th region, where i and j are positive integers.

[0078] In a possible design, in an eighth implementation of the second aspect of the embodiments of the present application,

[0079] The label is a number;

[0080] The label determination unit is used for:

[0081] For any two areas in the line drawing image, obtaining a ratio of a matching value to the number of points in a group of points in the matching point combination corresponding to the matching value as a weight;

[0082] The first energy function The minimum is the objective function that performs at least one iterative calculation. After each iterative calculation, the risk of selecting each unlabeled area for labeling is calculated according to the obtained label of the unlabeled area according to the preset error estimation algorithm; the unlabeled area with the smallest risk is selected for labeling according to the obtained label of the unlabeled area, and the next iterative calculation is performed after labeling, until the number of iterations reaches the preset number, and all remaining unlabeled areas are labeled according to the label of the unlabeled area obtained by the last iterative calculation, wherein the unlabeled area is an area of ​​the target image whose label is not determined, f(j) represents the label of the jth area, f(i) represents the label of the i-th area, and w i,j Represents the weights corresponding to the i-th region and the j-th region, where i and j are positive integers.

[0083] In a possible design, in a ninth implementation of the second aspect of the embodiments of the present application,

[0084] The coloring device for the line drawing image also includes:

[0085] The optimization unit is used to optimize the obtained labels of the unlabeled regions according to a preset mixed integer quadratic programming algorithm and the adjacency relationship between the labels, wherein the adjacency relationship between the labels corresponds to the adjacency relationship between the regions.

[0086] The third aspect of an embodiment of the present application provides a terminal device, including: a memory, a transceiver, a processor and a bus system; wherein the memory is used to store programs; the processor is used to execute the instructions in the memory, so that the communication device realizes the function of the coloring device of the line drawing image in the aforementioned second aspect.

[0087] A fourth aspect of an embodiment of the present application provides a computer-readable storage medium, in which instructions are stored. When the computer-readable storage medium is run on a computer, the computer executes the function of the coloring device for the line drawing image in the second aspect described above.

[0088] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages:

[0089] First, all line draft images are segmented, and then the regions of the reference image are labeled. All regions in the line draft images are matched pairwise to determine the labels of the target image regions. Finally, the line draft images are colored according to the correspondence between the labels and the region colors. This allows accurate coloring of the line draft images, and the smallest coloring unit is a segmented region. As long as the line draft images can be segmented accurately, color bleeding between adjacent regions can be avoided. BRIEF DESCRIPTION OF THE DRAWINGS

[0090] Figure 1This is a schematic diagram of an embodiment of a line drawing image to be colored in an embodiment of the present application;

[0091] Figure 2 This is another schematic diagram of an embodiment of a line drawing image to be colored in an embodiment of the present application;

[0092] Figure 3 A schematic diagram of an embodiment of a method for coloring a line drawing image provided in an embodiment of the present application;

[0093] Figure 4 This is a schematic diagram of an embodiment of a line drawing image with discontinuous line drawing boundaries in an embodiment of the present application;

[0094] Figure 5 In the embodiment of this application, the SEEDS algorithm is used to Figure 4 Schematic diagram of the coloring result after segmenting the line drawing image;

[0095] Figure 6 In the embodiment of this application, the ERS algorithm is used to Figure 4 Schematic diagram of the coloring result after segmenting the line drawing image;

[0096] Figure 7 In the embodiment of this application, a small ball segmentation algorithm is used to Figure 4 Schematic diagram of the coloring result after segmenting the line drawing image;

[0097] Figure 8 Connected component analysis is used to Figure 4 Schematic diagram of the coloring result of the line drawing image;

[0098] Fig. 9 This is a schematic diagram of an embodiment of a reference image in an embodiment of the present application;

[0099] Fig.10 To use the Patch-Match algorithm to match the target image with Fig. 9 Schematic diagram of the result after region matching of the reference image shown;

[0100] Fig.11 To use SIFT feature algorithm to compare the target image with Fig. 9 Schematic diagram of the result after region matching of the reference image shown;

[0101] Fig.12 To use the shape context algorithm to compare the target image with Fig. 9 Schematic diagram of the result after region matching of the reference image shown;

[0102] Fig.13 To use the shape context internal distance algorithm to compare the target image with Fig. 9 Schematic diagram of the result after region matching of the reference image shown;

[0103] Fig.14 A schematic diagram of an embodiment of matching all regions in a line drawing image in pairs provided in an embodiment of the present application;

[0104] Fig.15 A schematic diagram of an embodiment of the area of ​​a line drawing image in an embodiment of the present application;

[0105] Fig.16 This is a schematic diagram of an embodiment of calculating the matching values ​​between two regions in a line drawing image according to the matching values ​​of each point in an embodiment of the present application;

[0106] FIG. 17( a ) is a schematic diagram showing a comparison of label accuracy rates for three situations: not using active learning and mixed integer quadratic programming algorithm, using active learning, and using active learning and mixed integer quadratic programming algorithm;

[0107] FIG17( b ) is a schematic diagram of an embodiment of a reference image in an embodiment of the present application;

[0108] FIG. 17( c ) is a schematic diagram of another embodiment of a line drawing image to be colored in an embodiment of the present application;

[0109] FIG17( d ) is a schematic diagram showing the effect of coloring the line drawing image in FIG17( c ) according to the reference image in FIG17( b ) without using the mixed integer quadratic programming algorithm;

[0110] FIG17( e ) is a schematic diagram showing the effect of coloring the line drawing image in FIG17( c ) according to the reference image in FIG17( b ) using a mixed integer quadratic programming algorithm;

[0111] Fig.18 A schematic diagram showing the comparison of label accuracy of the target image area when λ takes different values;

[0112] Fig.19 A schematic structural diagram of an embodiment of a device for coloring a line drawing image provided in an embodiment of the present application;

[0113] Fig. 20 A schematic diagram of the structure of an embodiment of a terminal device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0114] The embodiments of the present application provide a method and device for coloring a line draft image, which are used to accurately color the line draft image and avoid color bleeding between adjacent areas.

[0115] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein, for example. In addition, the terms "including" and "corresponding to" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0116] It should be understood that this application is applicable to scenes such as animation production, animation production and game production. For details, please refer to Figure 1 , a schematic diagram of an embodiment of a line drawing image to be colored in an embodiment of the present application. Figure 1 The figure shows a line drawing of a person. In the line drawing, the line drawing boundary divides the person into multiple areas. Coloring the line drawing is to fill each area with a corresponding color. Figure 2 As shown, another embodiment of the line drawing image to be colored in the embodiment of the present application is shown. It can be seen that Figure 2 and Figure 1 Shown are images of the same person in different states.

[0117] The method for coloring a line drawing image provided in the embodiment of the present application is applicable to a scene in which multiple line drawing images are to be colored, and the multiple line drawings may include any objects such as people, animals, and objects.

[0118] For ease of understanding, the following will be introduced with specific embodiments. Figure 3 , a schematic diagram of an embodiment of a method for coloring a line draft image provided in an embodiment of the present application. An embodiment of the present application provides an embodiment of a method for coloring a line draft image, including:

[0119] 301, obtain all line drawing images to be colored.

[0120] It is understandable that all line drawing images may include images of the same object in different states, or may include images of different objects, which is not limited in the embodiments of the present application.

[0121] 302 , dividing each line drawing image into multiple regions according to a preset image segmentation algorithm.

[0122] It is understandable that if each area of ​​the line draft image needs to be colored separately, each area needs to be extracted, so the embodiment of the present application uses an image segmentation algorithm to segment the line draft image to obtain multiple areas.

[0123] However, some line drawings have discontinuous boundaries, such as Figure 4 As shown, a schematic diagram of an embodiment of a line drawing image with discontinuous line drawing boundaries in an embodiment of the present application. Figure 4 The one shown is Doraemon, from Figure 4 It can be seen that the boundary between the area of ​​one of the Doraemon's feet and the area of ​​the body is discontinuous. Generally, the color of the foot and the body are different, so the image segmentation algorithm in the embodiment of the present application needs to accurately segment adjacent areas with discontinuous boundaries to prevent the two areas on both sides of the discontinuous boundary from being mistakenly segmented into one area.

[0124] There are many image segmentation algorithms, including SEEDS algorithm, ERS algorithm and ball segmentation algorithm. In order to test the segmentation effect of these three algorithms, the present application embodiment uses these three algorithms to Figure 4 The line drawing image shown in the figure is segmented, and then the segmented areas are colored respectively. The coloring results are as follows Figure 5 , Figure 6 and Figure 7 As shown, Figure 5 In the embodiment of this application, the SEEDS algorithm is used to Figure 4 Schematic diagram of the coloring result after segmentation of the line drawing image. Figure 6 In the embodiment of this application, the ERS algorithm is used to Figure 4 Schematic diagram of the coloring result after segmentation of the line drawing image. Figure 7 In the embodiment of this application, a small ball segmentation algorithm is used to Figure 4 Schematic diagram of the coloring result after segmentation of the line drawing image.

[0125] from Figure 5 It can be seen that the SEEDS algorithm is used to Figure 4 The line drawing image is segmented, and the original area shape of "Doraemon" cannot be guaranteed; Figure 6 It can be seen that the ERS algorithm is used to Figure 4 The line drawing image of “Doraemon” is segmented. A line in the boundary of the line drawing of “Doraemon” is divided into multiple small areas. The size constraint is weak, and two adjacent areas with discontinuous boundaries cannot be segmented. Figure 7 It can be seen that the small ball segmentation algorithm is used to Figure 4 The line drawing image is segmented with good effect, and two adjacent areas with discontinuous boundaries can be accurately segmented.

[0126] In order to better reflect the use of small ball segmentation algorithm Figure 4 The embodiment of the present application adopts the method of connected component analysis to segment the line drawing image. Figure 4 Each connected region in the line drawing image is marked with a different color. The result is as follows Figure 8 As shown in the figure, the connected component analysis method is used to Figure 4 Schematic diagram of the coloring result of the line drawing image.

[0127] contrast Figure 7 and Figure 8 It can be seen that the small ball segmentation algorithm can accurately separate the area of ​​the "Doraemon"'s feet from the area of ​​its body.

[0128] Therefore, in order to ensure accurate segmentation of the line draft image, the embodiment of the present application may adopt a small ball segmentation algorithm as an image segmentation algorithm. In addition, the embodiment of the present application may also adopt other algorithms that can accurately segment the line draft image.

[0129] 303 , selecting at least one line draft image as a reference image, and taking the line draft images other than the reference image as target images.

[0130] It should be noted that the number of reference images can be adjusted according to the number of line draft images to be colored. For example, if the number of line draft images to be colored is large, dozens of line draft images can be selected as reference images; if the number of line draft images to be colored is small, several line draft images can be selected as reference images; in addition, the number of reference images can be adjusted according to the complexity of the line draft images to be colored. For example, if the complexity of the line draft images to be colored is high, dozens of line draft images can be selected as reference images; if the complexity of the line draft images to be colored is low, several line draft images can be selected as reference images.

[0131] It should also be noted that the embodiment of the present application does not limit the execution order between step 302 and step 303.

[0132] 304, obtain a label of each region of the reference image.

[0133] It should be noted that the method for coloring the line drawing image in the embodiment of the present application is to mark the label of each area of ​​the reference image after selecting the reference image. The marking process can be manual marking, and this step is to obtain the marked label result.

[0134] Labels are used to mark areas and can include various forms, such as numbers, letters, or Chinese characters.

[0135] 305, pairwise matching is performed on all regions in all line draft images.

[0136] It should be noted that there are many matching methods, which are not limited in this application. The matching result can be the matching degree of two arbitrary regions, or another region with the highest matching degree with a region.

[0137] The embodiment of the present application does not limit the execution order between step 304 and step 305.

[0138] 306 , determining a label of each region in the target image according to the matching result and the label of the region of the reference image.

[0139] It should be noted that there may be multiple methods for determining region labels based on the same matching result, and the methods for determining region labels corresponding to different matching results may also be different. Therefore, the embodiments of the present application do not specifically limit the method for determining region labels.

[0140] 307 , coloring all line drawing images according to the preset correspondence between the labels and the region colors.

[0141] It is understandable that if the labels of each area of ​​all line drawing images are determined, then the color of the label can be specified, for example, label 1 is green, label 2 is red, and finally all line drawing images can be colored according to the correspondence between the label and the area color.

[0142] The embodiment of the present application can realize accurate coloring of line drawing images, and the smallest unit for coloring is a segmented area. As long as the line drawing image can be accurately segmented, the color bleeding of adjacent areas can be avoided. In addition, in the embodiment of the present application, it is only necessary to mark the label of the reference image to complete the coloring of all line drawing images. Especially when there are many line drawing images to be colored, the method of the embodiment of the present application can reduce the workload of label marking.

[0143] In addition, in another embodiment of a method for coloring a line drawing image provided by an embodiment of the present application, matching an area of ​​the target image with an area of ​​the reference image may include:

[0144] Extract shape features of each area of ​​the line drawing image according to a preset shape context internal distance algorithm;

[0145] All regions in the line drawing image are matched pairwise based on shape features.

[0146] It should be noted that the shape features may include multiple types, and the embodiments of the present application do not limit this.

[0147] The algorithms for matching two areas include Patch-Match algorithm, SIFT feature algorithm, shape context algorithm, shape context internal distance algorithm and other algorithms. The embodiment of the present application selects the shape context internal distance algorithm to match the two areas in the line draft image. In order to verify that the shape context internal distance algorithm has a good matching effect, the embodiment of the present application respectively uses the Patch-Match algorithm, SIFT feature algorithm, shape context algorithm and shape context internal distance algorithm to match the same reference image and the same target image, where the reference image is as follows Fig. 9 As shown, the matching results are as follows Fig.10 , Fig.11 , Fig.12 as well as Fig.13 shown.

[0148] exist Fig. 9 In , different grayscales represent different labels; Fig.10 , Fig.11 , Fig.12 as well as Fig.13 In the figure, different grayscales also represent different labels, and these labels are matched by the Patch-Match algorithm, SIFT feature algorithm, shape context algorithm, and shape context internal distance algorithm. Fig. 9 , Fig.10 , Fig.11 , Fig.12 as well as Fig.13 It can be seen that the shape context internal distance algorithm is the best for matching two regions.

[0149] Furthermore, in order to facilitate understanding of the matching process between the target image region and the reference image region, a more detailed description will be given below. Fig.14 , a schematic diagram of an embodiment of the present application for performing pairwise matching on all regions in a line draft image. The present application provides an embodiment of performing pairwise matching on all regions in a line draft image, including:

[0150] 401 , for each region of the line drawing image, multiple points are taken on the region boundary.

[0151] It should be noted that the embodiment of the present application does not specifically limit the method of selecting points, as long as the method of selecting points in all areas is the same. For example, multiple points can be randomly selected or multiple points can be equidistantly selected.

[0152] 402, obtaining point features of all points on the region boundary as shape features of the region, wherein the point features of a point include the shortest distance between the current point and other points on the region boundary within the region, and the angle between the shortest distance line between the current point and other points on the region boundary and the tangent line of the current point.

[0153] See also Fig.15 , a schematic diagram of an embodiment of the area of ​​the line drawing image in the embodiment of the present application, Fig.15 What is shown is an area, and there are two points p and q on the boundary of the area. Assuming that the point feature of point p needs to be obtained, it is necessary to obtain the shortest distance between point p and other points in the area, as well as the angle between the shortest distance line between point p and other points and the tangent line of the current point.

[0154] Here, we take point p and point q as an example. The shortest distance between point p and point q refers to the shortest distance between points p and q when the line connecting them only passes through the area. For example, Fig.15 As shown, the dotted line between point p and point q represents the shortest distance line, and θ represents the angle between the shortest distance line between point p and point q and the tangent line of the current point.

[0155] It should be noted that the shortest distance line is generally composed of multiple broken line segments. When calculating the angle, the broken line passing through point p and the tangent line can be selected to calculate the angle.

[0156] 403 , for each area of ​​the line drawing image, obtain a plurality of shortest distance reference intervals and a plurality of angle reference intervals.

[0157] It should be noted that in the embodiment of the present application, the interval lengths of multiple shortest distance reference intervals can be equal or different. For example, if they are all 0.5 mm, then the multiple shortest distance reference intervals can include [0, 0.5 mm] and [0.5 mm, 1 mm]; similarly, the interval lengths of the angle reference intervals can be equal or different, which will not be described in detail.

[0158] 404, establish a point vector corresponding to each point one by one, the dimension of the point vector is equal to the product of the number of the shortest distance reference intervals and the number of the angle reference intervals, and the value of the point vector in each dimension is equal to the number of points in the same area boundary as the current point that are located in a shortest distance reference interval and in an angle reference interval.

[0159] It can be understood that, assuming that 100 points are selected on the boundary of the area, the number of shortest distance reference intervals is 8, and the number of angle reference intervals is 12; taking one point as an example, the dimension of the point vector corresponding to the point is 8*12=96, and each dimension corresponds to a shortest distance reference interval and an angle reference interval, and each dimension value is the number of other 99 points except the point that are simultaneously located in a shortest distance reference interval and an angle reference interval.

[0160] 405 , calculating the matching value of each point between two regions in the line drawing image according to the point vector.

[0161] It should be noted that there are many methods for calculating the matching value between two points. For example, the similarity of the two corresponding point vectors can be used as the matching value of the two points, and the similarity of the two point vectors can be cosine similarity. It can also be calculated by the formula Calculate the matching value of two points, where n represents the dimension, m represents the total number of dimensions, A represents one area, B represents another area, and h represents the dimension value.

[0162] 406 , calculating the matching values ​​between two regions in the line drawing image according to the matching values ​​of the points.

[0163] It should be noted that there are multiple methods for calculating the matching value between regions, which is not limited in the embodiments of the present application. The calculation process of the matching value between regions will be described in detail using one calculation method later.

[0164] See also Fig.16 , a schematic diagram of an embodiment of calculating the matching values ​​between two regions in a line drawing image according to the matching values ​​of each point in an embodiment of the present application.

[0165] In this embodiment, calculating the matching values ​​between two regions in the line drawing image according to the matching values ​​of each point includes:

[0166] 601, setting the point matching value between any two regions in the line drawing image that is greater than a threshold as a preset penalty value.

[0167] It should be noted that when the formula When calculating the matching value of two points, the higher the matching degree of the two points, the smaller the matching value. Therefore, when the matching value of two points is greater than the threshold, it can be considered that the two points are not matched, and the matching value of the two points is set to the preset penalty value.

[0168] 602. For any two areas in the line drawing image, all matching point combinations that meet preset conditions are obtained. The matching point combinations include a group of points selected from multiple points on the boundary of one area and a group of points selected from multiple points on the boundary of another area. The two groups of points correspond to each other in a preset direction on the boundaries of their respective areas, and the matching values ​​of the corresponding two points are both less than a threshold value. The preset direction is clockwise or counterclockwise.

[0169] It should be noted that, in the matching point combination, the two groups of points have to meet the preset condition of one-to-one correspondence, so the number of points in the two groups is equal; assuming that 8 points in one area are included and 8 points in another area are included, then these 8 pairs of points need to correspond one-to-one according to the preset direction on the boundary and the corresponding matching values ​​are less than the threshold. Specifically, assuming that the 8 points in one area are numbered as abcdefgh in a clockwise direction, and the 8 points in the other area are numbered as ABCDEFGH in a clockwise direction, if a and B are taken as the starting points respectively, the matching values ​​of a and B, b and C, c and D, d and E, e and F, f and G, g and H, h and A must be less than the threshold.

[0170] The number of points between a and b may be the same as or different from the number of points between A and B, and this embodiment of the present application does not limit this.

[0171] 603, for each matching point combination, obtain the sum of each matching value in the matching point group and the corresponding total penalty value, the total penalty value is the product of the preset penalty value and the penalty number, and the penalty number is the difference between the number of the multiple points and the number of points in a group of points.

[0172] It should be noted that, if the number of points selected on a region boundary is 10, and a group of points in the matching point combination includes 8 points, then the number of penalties is 10-8=2.

[0173] In the embodiment of the present application, a matching point combination represents a matching scheme of two regions, so a sum represents the matching degree of a matching scheme.

[0174] 604, taking the minimum value of the sum as the matching value of the two regions.

[0175] It can be understood that the sum represents the degree of matching between two regions. The embodiment of the present application selects the sum when the matching degree is the largest as the matching value of the two regions, that is, the minimum value of the sum is used as the final matching value of the two regions.

[0176] After the above calculation, there is a matching value between any two areas.

[0177] Further, for each area of ​​the line drawing image, obtaining a plurality of shortest distance reference intervals and a plurality of angle reference intervals may include:

[0178] For each area of ​​the line drawing image, obtain the maximum value of the shortest distance between points on the boundary of the area and the maximum value of the angle between the shortest distance line and the tangent line;

[0179] The maximum value of the shortest distance is evenly divided according to the number of preset shortest distance reference intervals to obtain multiple shortest distance reference intervals;

[0180] The maximum value of the angle is evenly divided according to the number of preset angle reference intervals to obtain a plurality of angle reference intervals.

[0181] For example, suppose an area of ​​a line drawing image has three points on the boundary of the area, then the shortest distance between any two points includes three results, assuming they are 10mm, 15mm and 20mm respectively. If four shortest distance reference intervals are required, 20mm is selected for average division, and the shortest distance reference intervals obtained are [0, 0.5mm], [0.5mm, 1mm], [1mm, 1.5mm] and [1.5mm, 2mm] respectively; similarly, multiple angle reference intervals can also be obtained using the above method, which will not be elaborated here.

[0182] In addition to the above method, a preset method can also be used to obtain multiple shortest distance reference intervals and multiple angle reference intervals. Taking the angle reference interval as an example, three angle reference intervals of 0-15 degrees, 15-20 degrees, and 20-30 degrees can be preset in sequence.

[0183] Further, determining the label of each region in the target image according to the matching result and the label of the region of the reference image may include:

[0184] The label of the region of the reference image that has the smallest matching value with the region of the target image is used as the label of the region of the target image.

[0185] It can be understood that when the matching value between the region of the target image and the region of the reference image is the smallest, it proves that the similarity between the two regions is the greatest. Therefore, the embodiment of the present application uses the region label of the reference image with the smallest matching value as the label of the target image region. In this way, the labels of each region in the target image can be determined.

[0186] In another embodiment of a method for coloring a line drawing image provided in the present application, in order to further improve the label accuracy of the target image area, the embodiment of the present application adopts an active learning method. The following will specifically introduce the process of determining the label of each area in the target image based on the matching results and the labels of the areas of the reference image.

[0187] In the embodiment of the present application, since the label needs to be calculated, the label is a number, and the label of each area in the target image is determined according to the matching result and the label of the area of ​​the reference image, including:

[0188] For any two regions in the line drawing image, the ratio of the matching value to the number of points in a set of matching point combinations corresponding to the matching value is obtained as a weight;

[0189] The first energy function The minimum is the objective function that performs at least one iterative calculation. After each iterative calculation, one of the unlabeled regions is selected for labeling according to the obtained label of the unlabeled region, and the next iterative calculation is performed after labeling. After the number of iterations reaches the preset number, all the remaining unlabeled regions are labeled according to the label of the unlabeled region obtained by the last iterative calculation. The unlabeled region is the region of the target image whose label is not determined. f(j) represents the label of the jth region, f(i) represents the label of the i-th region, and w i,j Represents the weights corresponding to the i-th region and the j-th region, where i and j are positive integers.

[0190] It should be noted that in the embodiment of the present application, an iterative calculation is performed with the minimum of the first energy function E(f) as the objective function, that is, an optimization process, where f(j) represents the label of the j-th labeled area, which means that f(j) is a known label. After each iterative calculation, a set of labels f(i) for unlabeled areas will be obtained.

[0191] For example, assuming that the reference image includes 5 regions, the labels of these 5 regions are known, and the target image also includes 5 regions, the labels of these 5 regions are unknown. After one iterative calculation, the labels of the 5 regions of the target image will be obtained when the first energy function E(f) is minimized.

[0192] However, there may be errors in the labels of the five areas of the target image. Therefore, when marking the areas of the target image according to the five labels obtained by iterative calculation, the embodiment of the present application first selects an area of ​​the target image for marking. After the marking is completed, the number of marked areas becomes 6. The embodiment of the present application performs a second iterative calculation to recalculate the remaining 4 unlabeled areas.

[0193] The above process is repeated until the number of iterations reaches the preset number. It can be understood that, assuming that after two iterative calculations, the number of marked areas becomes 7 and the number of unmarked areas becomes 3, the second iterative calculation will obtain the labels of the unmarked areas. The embodiment of the present application marks each unmarked area in turn according to the label obtained by the second iterative calculation, and finally determines the label of each area of ​​the target image.

[0194] In the above process, after one iterative calculation, only one of the unlabeled areas is selected for labeling, and then the next iterative calculation is performed. This is to ensure the accuracy of the label of each area of ​​the target image. Therefore, the unlabeled area with the lowest risk must be selected for labeling.

[0195] Therefore, in another embodiment of a method for coloring a line drawing image provided by an embodiment of the present application, selecting one of the unmarked areas for marking according to the obtained label of the unmarked area may include:

[0196] Calculate the risk of labeling each unlabeled region according to the obtained labels of the unlabeled regions according to a preset error estimation algorithm;

[0197] According to the obtained labels of the unlabeled regions, the unlabeled regions with the lowest risk are selected for labeling.

[0198] Further, in order to optimize the label result of the unlabeled area, after each iterative calculation, before selecting one of the unlabeled areas for labeling according to the obtained label of the unlabeled area, the method for coloring the line drawing image in the embodiment of the present application may further include:

[0199] The obtained labels of the unlabeled regions are optimized according to a preset mixed integer quadratic programming algorithm and the adjacency relations between the labels, and the adjacency relations between the labels correspond to the adjacency relations between the regions.

[0200] It should be noted that, in the embodiment of the present application, the above-mentioned optimization step is performed after each iterative calculation. In addition, it can also be performed after the iterative calculation is completely completed.

[0201] In the embodiment of the present application, there is an adjacency relationship between the regions of the target image, and there should also be an adjacency relationship between the corresponding labels, so the labels of the obtained unlabeled regions are optimized according to the adjacency relationship between the labels and the mixed integer quadratic programming algorithm.

[0202] In the mixed integer quadratic programming algorithm, the second energy function E = E match +λE local The maximum is the objective function, E match represents the cost energy of labeling the unlabeled area, i identifies the i-th area, N represents the total number of line drawing image areas, c i represents the probability that the i-th region label is correct, y i represents the label corresponding to the i-th region, and λ is used to adjust the local constraint E local The strength of E local represents a local constraint, Near(i) represents all neighbors of the ith region, M represents the adjacency relationship between the ith region and its neighbors, and y j Denotes the label of the jth neighborhood of the ith region.

[0203] To verify the effectiveness of the mixed integer quadratic programming algorithm, see Figure 17(b) to Figure 17(e), FIG17(b) is a schematic diagram of an embodiment of a reference image in an embodiment of the present application, FIG17(c) is a schematic diagram of another embodiment of a line drawing image to be colored in an embodiment of the present application, FIG17(d) is a schematic diagram of the effect of coloring the line drawing image in FIG17(c) according to the reference image in FIG17(b) without using the mixed integer quadratic programming algorithm, and FIG17(e) is a schematic diagram of the effect of coloring the line drawing image in FIG17(c) according to the reference image in FIG17(b) when the mixed integer quadratic programming algorithm is used. By comparing the above figures, it can be seen that after optimization using the mixed integer quadratic programming algorithm, the coloring accuracy is higher.

[0204] In the embodiment of the present application, in order to verify the effectiveness of active learning and mixed integer quadratic programming algorithms, the label accuracy rates were statistically analyzed for three cases: not using active learning and mixed integer quadratic programming algorithms, using active learning, and using active learning and mixed integer quadratic programming algorithms, as shown in FIG17( a). In FIG17( a), the horizontal axis represents the number of line draft images, and the vertical axis represents the label accuracy rate. Curve 1 represents the label accuracy rate when active learning and mixed integer quadratic programming algorithms are not used, curve 2 represents the label accuracy rate when active learning is used, and curve 3 represents the label accuracy rate when active learning and mixed integer quadratic programming algorithms are used. By comparison, it can be seen that active learning can improve the label accuracy rate of the target image area, and the mixed integer quadratic programming algorithm can further improve the label accuracy rate of the target image area on the basis of active learning.

[0205] It should be noted that the size of λ can be adjusted according to actual conditions. Fig.18 , when λ takes different values, the label accuracy of the target image area is Fig.18 In the figure, the horizontal axis is the number of line drawings, and the vertical axis is the label accuracy. Fig.18 It can be seen that when λ is 0.1, the label accuracy is the highest, so in this embodiment of the application, λ is taken as 0.1.

[0206] The following is a detailed description of the device for coloring the line drawing image in the embodiment of the present application. Fig.19 , an embodiment of the present application provides a structural schematic diagram of an embodiment of a device for coloring a line drawing image.

[0207] The present application embodiment provides a coloring device for a line drawing image, comprising:

[0208] A first acquisition unit 901 is used to acquire all line drawing images to be colored;

[0209] The segmentation unit 902 is used to segment each line drawing image into multiple regions according to a preset image segmentation algorithm;

[0210] A first selection unit 903 is used to select at least one line draft image as a reference image, and to use line draft images other than the reference image as target images;

[0211] A second acquisition unit 904 is used to acquire a label of each region of the reference image;

[0212] A matching unit 905 is used to perform pairwise matching on all regions in the line drawing image;

[0213] A label determination unit 906, configured to determine a label of each region in the target image according to the matching result and the label of the region of the reference image;

[0214] The coloring unit 907 is used to color all line drawing images according to the preset correspondence between the label and the area color.

[0215] Furthermore, in another embodiment of the apparatus for coloring a line drawing image, the matching unit 905 is used to:

[0216] Extract shape features of each area of ​​the line drawing image according to a preset shape context internal distance algorithm;

[0217] All regions in the line drawing image are matched pairwise based on shape features.

[0218] Furthermore, in another embodiment of the apparatus for coloring a line drawing image, the matching unit 905 is used to:

[0219] For each region of the line drawing image, take multiple points on the region boundary;

[0220] Get the point features of all points on the region boundary as the shape features of the region. The point features of a point include the shortest distance between the current point and other points on the same region boundary within the region, and the angle between the shortest distance line between the current point and other points on the same region boundary and the tangent line of the current point.

[0221] All regions in the line drawing image are matched pairwise based on shape features.

[0222] Furthermore, in another embodiment of the apparatus for coloring a line drawing image, the matching unit 905 is used to:

[0223] For each region of the line drawing image, take multiple points on the region boundary;

[0224] Get the point features of all points on the region boundary as the shape features of the region. The point features of a point include the shortest distance between the current point and other points on the same region boundary within the region, and the angle between the shortest distance line between the current point and other points on the same region boundary and the tangent line of the current point.

[0225] For each area of ​​the line drawing image, a plurality of shortest distance reference intervals and a plurality of angle reference intervals are obtained;

[0226] Establish a point vector corresponding to each point one by one. The dimension of the point vector is equal to the product of the number of the shortest distance reference intervals and the number of the angle reference intervals. The value of the point vector in each dimension is equal to the number of points in the same area boundary as the current point that are located in a shortest distance reference interval and an angle reference interval.

[0227] Calculate the matching value of each point between two regions in the line draft image according to the point vector;

[0228] The matching values ​​between two regions in the line draft image are calculated based on the matching values ​​of each point.

[0229] Furthermore, in another embodiment of the apparatus for coloring a line drawing image, the matching unit 905 is used to:

[0230] For each region of the line drawing image, take multiple points on the region boundary;

[0231] Get the point features of all points on the region boundary as the shape features of the region. The point features of a point include the shortest distance between the current point and other points on the same region boundary within the region, and the angle between the shortest distance line between the current point and other points on the same region boundary and the tangent line of the current point.

[0232] For each area of ​​the line drawing image, a plurality of shortest distance reference intervals and a plurality of angle reference intervals are obtained;

[0233] Establish a point vector corresponding to each point one by one. The dimension of the point vector is equal to the product of the number of the shortest distance reference intervals and the number of the angle reference intervals. The value of the point vector in each dimension is equal to the number of points in the same area boundary as the current point that are located in a shortest distance reference interval and an angle reference interval.

[0234] Calculate the matching value of each point between two regions in the line draft image according to the point vector;

[0235] The point matching value between any two areas in the line drawing image that is greater than the threshold is set as a preset penalty value;

[0236] For any two regions in the line drawing image, all matching point combinations that meet the preset conditions are obtained, the matching point combinations include a group of points selected from multiple points on the boundary of one region and a group of points selected from multiple points on the boundary of another region, the two groups of points correspond one to one on the boundaries of their respective regions according to a preset direction and the matching values ​​of the two corresponding points are both less than a threshold value, and the preset direction is a clockwise direction or a counterclockwise direction;

[0237] For each matching point combination, obtain the sum of each matching value in the matching point group and the corresponding total penalty value, the total penalty value is the product of the preset penalty value and the penalty number, and the penalty number is the difference between the number of multiple points and the number of points in a group of points;

[0238] The minimum value of the sum is taken as the matching value of the two areas.

[0239] Furthermore, in another embodiment of the apparatus for coloring a line drawing image, the matching unit 905 is used to:

[0240] For each region of the line drawing image, take multiple points on the region boundary;

[0241] Get the point features of all points on the region boundary as the shape features of the region. The point features of a point include the shortest distance between the current point and other points on the same region boundary within the region, and the angle between the shortest distance line between the current point and other points on the same region boundary and the tangent line of the current point.

[0242] For each area of ​​the line drawing image, obtain the maximum value of the shortest distance between points on the boundary of the area and the maximum value of the angle between the shortest distance line and the tangent line;

[0243] The maximum value of the shortest distance is evenly divided according to the number of preset shortest distance reference intervals to obtain multiple shortest distance reference intervals;

[0244] The maximum value of the angle is evenly divided according to the number of preset angle reference intervals to obtain multiple angle reference intervals;

[0245] Establish a point vector corresponding to each point one by one. The dimension of the point vector is equal to the product of the number of the shortest distance reference intervals and the number of the angle reference intervals. The value of the point vector in each dimension is equal to the number of points in the same area boundary as the current point that are located in a shortest distance reference interval and an angle reference interval.

[0246] Calculate the matching value of each point between two regions in the line draft image according to the point vector;

[0247] The point matching value between any two areas in the line drawing image that is greater than the threshold is set as a preset penalty value;

[0248] For any two regions in the line drawing image, all matching point combinations that meet the preset conditions are obtained, the matching point combinations include a group of points selected from multiple points on the boundary of one region and a group of points selected from multiple points on the boundary of another region, the two groups of points correspond one to one on the boundaries of their respective regions according to a preset direction and the matching values ​​of the two corresponding points are both less than a threshold value, and the preset direction is a clockwise direction or a counterclockwise direction;

[0249] For each matching point combination, obtain the sum of each matching value in the matching point group and the corresponding total penalty value, the total penalty value is the product of the preset penalty value and the penalty number, and the penalty number is the difference between the number of multiple points and the number of points in a group of points;

[0250] The minimum value of the sum is taken as the matching value of the two areas.

[0251] Furthermore, in another embodiment of the apparatus for coloring a line drawing image, the label determination unit 906 is used to:

[0252] The label of the region of the reference image that has the smallest matching value with the region of the target image is used as the label of the region of the target image.

[0253] Further, in another embodiment of the device for coloring a line drawing image, the label is a number;

[0254] The label determination unit 906 is used to:

[0255] For any two regions in the line drawing image, the ratio of the matching value to the number of points in a set of matching point combinations corresponding to the matching value is obtained as a weight;

[0256] The first energy function The minimum is the objective function that performs at least one iterative calculation. After each iterative calculation, one of the unlabeled regions is selected for labeling according to the obtained label of the unlabeled region, and the next iterative calculation is performed after labeling. After the number of iterations reaches the preset number, all the remaining unlabeled regions are labeled according to the label of the unlabeled region obtained by the last iterative calculation. The unlabeled region is the region of the target image whose label is not determined. f(j) represents the label of the jth region, f(i) represents the label of the i-th region, and w i,j Represents the weights corresponding to the i-th region and the j-th region, where i and j are positive integers.

[0257] Further, in another embodiment of the device for coloring a line drawing image, the label is a number;

[0258] The label determination unit 906 is used to:

[0259] For any two regions in the line drawing image, the ratio of the matching value to the number of points in a set of matching point combinations corresponding to the matching value is obtained as a weight;

[0260] The first energy function The minimum is the objective function that performs at least one iterative calculation. After each iterative calculation, the risk of selecting each unlabeled area for labeling is calculated according to the labels of the unlabeled areas obtained according to the preset error estimation algorithm. The unlabeled area with the smallest risk is selected for labeling according to the labels of the unlabeled areas obtained, and the next iterative calculation is performed after labeling until the number of iterations reaches the preset number. All remaining unlabeled areas are labeled according to the labels of the unlabeled areas obtained by the last iterative calculation. The unlabeled area is the area of ​​the target image whose label is not determined. f(j) represents the label of the jth area, f(i) represents the label of the ith area, and w i,j Represents the weights corresponding to the i-th region and the j-th region, where i and j are positive integers.

[0261] Furthermore, in another embodiment of the device for coloring a line draft image, the device for coloring a line draft image further comprises:

[0262] The optimization unit 908 is used to optimize the obtained labels of the unlabeled regions according to a preset mixed integer quadratic programming algorithm and the adjacency relationship between the labels, where the adjacency relationship between the labels corresponds to the adjacency relationship between the regions.

[0263] Next, the present application embodiment also provides a terminal device, such as Fig. 20 For the sake of convenience, only the parts related to the embodiments of the present invention are shown. For specific technical details not disclosed, please refer to the method part of the embodiments of the present invention. The attribute information display device can be any terminal device including a mobile phone, a tablet computer, a personal digital assistant (PDA), a point of sales (POS), a car computer, etc., taking the attribute information display device as a mobile phone as an example:

[0264] Fig. 20 FIG. 1 is a block diagram showing a portion of the structure of a mobile phone related to the attribute information display device provided by an embodiment of the present invention. Fig. 20 The mobile phone includes: a radio frequency (RF) circuit 2010, a memory 2020, an input unit 2030, a display unit 2040, a sensor 2050, an audio circuit 2060, a wireless fidelity (WiFi) module 2070, a processor 2080, and a power supply 2090. Those skilled in the art will understand that Fig. 20 The mobile phone structure shown in the figure does not constitute a limitation on the mobile phone, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0265] Combine the following Fig. 20The following is a detailed introduction to the various components of the mobile phone:

[0266] RF circuit 2010 can be used for receiving and sending signals during information transmission or call, in particular, after receiving the downlink information of the base station, it is sent to processor 2080 for processing; in addition, the designed uplink data is sent to the base station. Usually, RF circuit 2010 includes but is not limited to antenna, at least one amplifier, transceiver, coupler, low noise amplifier (Low Noise Amplifier, LNA), duplexer, etc. In addition, RF circuit 2010 can also communicate with the network and other devices through wireless communication. The above wireless communication can use any communication standard or protocol, including but not limited to Global System of Mobile communication (Global System of Mobile communication, GSM), General Packet Radio Service (General Packet Radio Service, GPRS), Code Division Multiple Access (Code Division Multiple Access, CDMA), Wideband Code Division Multiple Access (Wideband Code Division Multiple Access, WCDMA), Long Term Evolution (Long Term Evolution, LTE), email, Short Messaging Service (SMS), etc.

[0267] The memory 2020 can be used to store software programs and modules. The processor 2080 executes various functional applications and data processing of the mobile phone by running the software programs and modules stored in the memory 2020. The memory 2020 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory 2020 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0268] The input unit 2030 can be used to receive input digital or character information, and to generate key signal input related to the user settings and function control of the mobile phone. Specifically, the input unit 2030 may include a touch panel 2031 and other input devices 2032. The touch panel 2031, also known as a touch screen, can collect the user's touch operation on or near it (such as the user using any suitable object or accessory such as a finger, stylus, etc. on the touch panel 2031 or near the touch panel 2031), and drive the corresponding connection device according to a pre-set program. Optionally, the touch panel 2031 may include two parts: a touch detection device and a touch controller. Among them, the touch detection device detects the user's touch orientation, detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into contact coordinates, and then sends it to the processor 2080, and can receive and execute commands sent by the processor 2080. In addition, the touch panel 2031 can be implemented in various types such as resistive, capacitive, infrared, and surface acoustic waves. In addition to the touch panel 2031, the input unit 2030 may also include other input devices 2032. Specifically, the other input devices 2032 may include but are not limited to one or more of a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, a joystick, etc.

[0269] The display unit 2040 can be used to display information input by the user or information provided to the user and various menus of the mobile phone. The display unit 2040 may include a display panel 2041. Optionally, the display panel 2041 may be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), etc. Further, the touch panel 2031 may cover the display panel 2041. When the touch panel 2031 detects a touch operation on or near it, it is transmitted to the processor 2080 to determine the type of touch event. Subsequently, the processor 2080 provides a corresponding visual output on the display panel 2041 according to the type of touch event. Although in Fig. 20 In the embodiment, the touch panel 2031 and the display panel 2041 are used as two independent components to realize the input and output functions of the mobile phone, but in some embodiments, the touch panel 2031 and the display panel 2041 can be integrated to realize the input and output functions of the mobile phone.

[0270] The mobile phone may also include at least one sensor 2050, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor, wherein the ambient light sensor may adjust the brightness of the display panel 2041 according to the brightness of the ambient light, and the proximity sensor may turn off the display panel 2041 and / or the backlight when the mobile phone is moved to the ear. As a type of motion sensor, the accelerometer sensor can detect the magnitude of acceleration in all directions (generally three axes), and can detect the magnitude and direction of gravity when stationary. It can be used for applications that identify the posture of the mobile phone (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), etc.; as for other sensors that can also be configured in the mobile phone, such as gyroscopes, barometers, hygrometers, thermometers, infrared sensors, etc., they will not be repeated here.

[0271] The audio circuit 2060, the speaker 2061, and the microphone 2062 can provide an audio interface between the user and the mobile phone. The audio circuit 2060 can transmit the received audio data to the speaker 2061 after converting the received audio data into an electrical signal, which is converted into a sound signal for output; on the other hand, the microphone 2062 converts the collected sound signal into an electrical signal, which is received by the audio circuit 2060 and converted into audio data, and then the audio data is output to the processor 2080 for processing, and then sent to another mobile phone through the RF circuit 2010, or the audio data is output to the memory 2020 for further processing.

[0272] WiFi is a short-range wireless transmission technology. The mobile phone can help users send and receive emails, browse web pages and access streaming media through the WiFi module 2070. It provides users with wireless broadband Internet access. Fig. 20 A WiFi module 2070 is shown, but it is understandable that it is not an essential component of the mobile phone and can be omitted as needed without changing the essence of the invention.

[0273] The processor 2080 is the control center of the mobile phone. It uses various interfaces and lines to connect various parts of the entire mobile phone. It executes various functions of the mobile phone and processes data by running or executing software programs and / or modules stored in the memory 2020 and calling data stored in the memory 2020. Optionally, the processor 2080 may include one or more processing units; optionally, the processor 2080 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 2080.

[0274] The mobile phone also includes a power supply 2090 (such as a battery) for supplying power to various components. Optionally, the power supply can be logically connected to the processor 2080 through a power management system, thereby implementing functions such as charging, discharging, and power consumption management through the power management system.

[0275] Although not shown, the mobile phone may also include a camera module, a Bluetooth module, etc., which will not be described in detail here.

[0276] In the embodiment of the present invention, the processor 2080 included in the terminal device also has the function of the aforementioned device for coloring the line drawing image.

[0277] An embodiment of the present application also provides a computer storage medium for storing computer software instructions used by a device or server for coloring the above-mentioned line drawing image, which includes a program designed for executing a construction device or terminal device for requesting synonyms.

[0278] An embodiment of the present application also provides a computer program product, which includes computer software instructions. The computer software instructions can be loaded by a processor to implement the process in the above-mentioned method of coloring a line drawing image.

[0279] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0280] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0281] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0282] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0283] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.

[0284] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for coloring a line drawing image, characterized in that: include: Get all the line drawings to be colored; Divide each line drawing image into multiple regions according to a preset image segmentation algorithm, wherein the image segmentation algorithm is a small ball segmentation algorithm; Selecting at least one line drawing image as a reference image, and using line drawing images other than the reference image as target images; Obtaining a label for each region of the reference image; Performing pairwise matching on all regions in all line draft images, including: for each region of the line draft image, taking multiple points on the region boundary; obtaining point features of all points on the region boundary as shape features of the region, the point features of a point including the shortest distance between the current point and other points on the same region boundary within the region, and the angle between the line connecting the shortest distance between the current point and other points on the same region boundary and the tangent of the current point; for each region of the line draft image, obtaining multiple shortest distance reference intervals and multiple angle reference intervals; establishing a point vector corresponding to each point one by one, the dimension of the point vector being equal to the product of the number of the shortest distance reference intervals and the number of the angle reference intervals, the value of the point vector in each dimension being equal to the number of points located in one of the shortest distance reference intervals and one of the angle reference intervals among other points on the same region boundary as the current point; and obtaining a point vector according to the point vector. Calculate the matching value of each point between two regions in the line draft image; set the matching value of the point between any two regions in the line draft image that is greater than a threshold as a preset penalty value; for any two regions in the line draft image, obtain all matching point combinations that meet the preset conditions, the matching point combination includes a group of points selected from a plurality of points on the boundary of one region and a group of points selected from a plurality of points on the boundary of another region, the two groups of points correspond one to one on the boundaries of their respective regions in a preset direction and the matching values ​​of the corresponding two points are both less than the threshold, the preset direction is a clockwise direction or a counterclockwise direction; for each matching point combination, obtain the sum of each matching value in the matching point group and the corresponding total penalty value, the total penalty value is the product of the preset penalty value and the number of penalties, the penalty number is the difference between the number of the plurality of points and the number of points in a group of points; take the minimum value of the sum as the matching value of the two regions; Determine a label of each region in the target image according to the matching result and the label of the region of the reference image; All the line draft images are colored according to the preset correspondence between the labels and the area colors.

2. The method for coloring a line drawing image according to claim 1, characterized in that: For each area of ​​the line drawing image, obtaining a plurality of shortest distance reference intervals and a plurality of angle reference intervals comprises: For each area of ​​the line drawing image, obtain the maximum value of the shortest distance between points on the boundary of the area and the maximum value of the angle between the shortest distance line and the tangent line; The maximum value of the shortest distance is evenly divided according to the number of preset shortest distance reference intervals to obtain multiple shortest distance reference intervals; The maximum value of the angle is evenly divided according to the number of preset angle reference intervals to obtain a plurality of angle reference intervals.

3. The method for coloring a line drawing image according to claim 1, characterized in that: Determining the label of each area in the target image according to the matching result and the label of the area of ​​the reference image comprises: The label of the region of the reference image having the smallest matching value with the region of the target image is used as the label of the region of the target image.

4. The method for coloring a line drawing image according to claim 1, characterized in that: The label is a number; Determining the label of each area in the target image according to the matching result and the label of the area of ​​the reference image comprises: For any two areas in the line drawing image, obtaining a ratio of a matching value to the number of points in a group of points in the matching point combination corresponding to the matching value as a weight; The first energy function The minimum is the objective function that performs at least one iterative calculation. After each iterative calculation, one of the unlabeled regions is selected for labeling according to the obtained label of the unlabeled region, and the next iterative calculation is performed after labeling. After the number of iterations reaches the preset number, all the remaining unlabeled regions are labeled according to the label of the unlabeled region obtained by the last iterative calculation. The unlabeled region is the region of the target image whose label is not determined. f(j) represents the label of the jth region, f(i) represents the label of the ith region, and w i,j Represents the weights corresponding to the i-th region and the j-th region, where i and j are positive integers.

5. A coloring device for a line drawing image, characterized in that: include: A first acquisition unit, used to acquire all line drawing images to be colored; A segmentation unit, used to segment each line draft image into a plurality of regions according to a preset image segmentation algorithm, wherein the image segmentation algorithm is a small ball segmentation algorithm; A first selection unit, configured to select at least one line draft image as a reference image, and to use line draft images other than the reference image as target images; A second acquisition unit, used to acquire a label of each area of ​​the reference image; A matching unit is used to perform pairwise matching on all areas in all line draft images, which includes: for each area of ​​the line draft image, multiple points are taken on the area boundary; point features of all points on the area boundary are obtained as shape features of the area, and the point features of a point include the shortest distance between the current point and other points on the same area boundary within the area, and the angle between the shortest distance line between the current point and other points on the same area boundary and the tangent of the current point; for each area of ​​the line draft image, multiple shortest distance reference intervals and multiple angle reference intervals are obtained; a point vector corresponding to each point is established, the dimension of the point vector is equal to the product of the number of the shortest distance reference intervals and the number of the angle reference intervals, and the value of the point vector in each dimension is equal to the number of points located in one of the shortest distance reference intervals and one of the angle reference intervals among other points on the same area boundary as the current point; according to the The point vector is used to calculate the matching value of each point between two areas in the line draft image; the matching value of the point between any two areas in the line draft image that is greater than a threshold is set as a preset penalty value; for any two areas in the line draft image, all matching point combinations that meet the preset conditions are obtained, the matching point combination includes a group of points selected from multiple points on the boundary of one area and a group of points selected from multiple points on the boundary of another area, the two groups of points are respectively on the boundaries of their respective areas in a one-to-one correspondence according to a preset direction and the matching values ​​of the corresponding two points are both less than a threshold, the preset direction is a clockwise direction or a counterclockwise direction; for each matching point combination, the sum of each matching value in the matching point group and the corresponding total penalty value is obtained, the total penalty value is the product of the preset penalty value and the penalty number, the penalty number is the difference between the number of the multiple points and the number of points in a group of points; the minimum value of the sum is taken as the matching value of the two areas; a label determination unit, configured to determine a label of each region in the target image according to a matching result and a label of a region of the reference image; The coloring unit is used to color all line drawing images according to the preset correspondence between the label and the area color.

6. The device according to claim 5, characterized in that The matching unit acquires a plurality of shortest distance reference intervals and a plurality of angle reference intervals for each area of ​​the line drawing image, including: For each area of ​​the line drawing image, obtain the maximum value of the shortest distance between points on the boundary of the area and the maximum value of the angle between the shortest distance line and the tangent line; The maximum value of the shortest distance is evenly divided according to the number of preset shortest distance reference intervals to obtain multiple shortest distance reference intervals; The maximum value of the angle is evenly divided according to the number of preset angle reference intervals to obtain a plurality of angle reference intervals.

7. The device according to claim 5, characterized in that The label determination unit is used to: The label of the region of the reference image having the smallest matching value with the region of the target image is used as the label of the region of the target image.

8. The device according to claim 5, characterized in that The label is a number; The label determination unit is used to obtain, for any two areas in the line draft image, a ratio of a matching value to the number of points in a group of points in the matching point combination corresponding to the matching value as a weight; The first energy function The minimum is the objective function that performs at least one iterative calculation. After each iterative calculation, one of the unlabeled regions is selected for labeling according to the obtained label of the unlabeled region, and the next iterative calculation is performed after labeling. After the number of iterations reaches the preset number, all the remaining unlabeled regions are labeled according to the label of the unlabeled region obtained by the last iterative calculation. The unlabeled region is the region of the target image whose label is not determined. f(j) represents the label of the jth region, f(i) represents the label of the ith region, and w i,j Represents the weights corresponding to the i-th region and the j-th region, where i and j are positive integers.

9. A computer-readable storage medium, wherein instructions are stored in the computer-readable storage medium, and when the computer-readable storage medium is executed on a computer, the computer is enabled to execute the method according to any one of claims 1 to 4.

10. A computer program product, characterized in that The computer program product comprises computer software instructions, and the computer software instructions are loaded by a processor to implement the method according to any one of the preceding claims 1-4.

Images