Sketch generation method, apparatus and device, and readable storage medium

Abstract

The invention provides a sketch graph generation method, device and equipment and a readable storage medium, relates to the technical field of image processing, and aims to solve the problems of relatively large data volume and relatively low image coding efficiency of a semantic representation method in the prior art. The method comprises the following steps: carrying out edge detection on a to-be-processed image to obtain an edge image of the to-be-processed image; determining initial sketch points in the edge graph, wherein the initial sketch points comprise sketch graph points belonging to edge points in the edge graph; based on the image features of the edge image and the position information of the initial sketch points, the connection probability of an initial sketch point pair is determined, and the initial sketch point pair comprises any two initial sketch points in the initial sketch points in the edge image; and generating a target sketch based on the connection probability of the initial sketch point pairs and the position information of the initial sketch points. According to the invention, the image coding efficiency can be improved.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a sketch generation method, device, equipment and a readable storage medium. Background Art

[0002] With the development of multimedia capture devices and the increasing demand for multimedia services, efficient multimedia communication with large data volumes has attracted widespread attention from researchers. In order to adapt to such scenarios, the concept of multimedia computing communication is proposed. Multimedia computing communication significantly improves transmission efficiency by introducing artificial intelligence into the communication process. Among them, by extracting the high-level semantics of multimedia data, the amount of data transmitted during the communication process can be reduced. Therefore, how to perform effective semantic representation has become a key factor affecting the performance of multimedia computing communication. However, in the prior art, semantic representation generally adopts image representation methods, such as gradient directional map representation methods, which extract pixel values ​​to form a total feature vector, but this method has a large amount of data and low image coding efficiency. Summary of the invention

[0003] The embodiments of the present invention provide a sketch image generation method, device, equipment and readable storage medium. In current practical applications, the amount of data represented by semantics is large, resulting in low image compression efficiency.

[0004] In order to solve the above problems, the embodiments of the present invention adopt the following technical solutions:

[0005] In a first aspect, an embodiment of the present invention provides a method for generating a sketch, comprising:

[0006] Performing edge detection on the image to be processed to obtain an edge map of the image to be processed;

[0007] Determine initial sketch points in the edge map, the initial sketch points comprising sketch points in the edge map that are edge points;

[0008] Determine, based on the image features of the edge map and the position information of the initial sketch points, a connection probability of an initial sketch point pair, wherein the initial sketch point pair includes any two initial sketch points among the initial sketch points in the edge map;

[0009] A target sketch map is generated based on the connection probability of the initial sketch point pairs and the position information of the initial sketch points.

[0010] Optionally, determining the initial sketch point in the edge map includes:

[0011] Performing feature extraction on the edge map to obtain image features of the edge map;

[0012] Performing a dot multiplication process on the image feature of the edge map and the edge map to obtain at least two first heat maps, wherein the at least two first heat maps are respectively used to characterize the first category probability of each edge pixel in the edge map, and the first category probability is the probability that each edge pixel belongs to the category corresponding to the first heat map;

[0013] Based on the at least two first heat maps, initial sketch points in the edge map are determined.

[0014] Optionally, determining the initial sketch point in the edge map based on the at least two first heat maps includes:

[0015] Performing non-maximum suppression processing on the at least two first heat maps to obtain at least two second heat maps, wherein the at least two second heat maps are respectively used to characterize the second category probability of each edge pixel point in the edge map;

[0016] Determine the probability of the category to which each edge pixel belongs according to the values ​​of each edge pixel in the at least two second heat maps;

[0017] Determine the category to which each edge pixel belongs according to the probability of the category to which each edge pixel belongs, the category to which each edge pixel belongs being the category to which each edge pixel belongs with the greatest probability;

[0018] Based on the category to which each edge pixel point belongs, an initial sketch point in the edge map is determined.

[0019] Optionally, determining the connection probability of the initial sketch point pair based on the image features of the edge map and the position information of the initial sketch point includes:

[0020] Based on the position information of the initial sketch point, sampling is performed on the connecting line of the initial sketch point pair to obtain a sampling point sequence of the initial sketch point pair;

[0021] Determining features of the initial sketch point pair based on the sampling point sequence and image features of the edge map;

[0022] Based on the features of the initial sketch point pairs, a connection probability of the initial sketch point pairs is determined.

[0023] Optionally, determining the connection probability of the initial sketch point pair based on the features of the initial sketch point pair includes:

[0024] Using a convolutional neural network and a fully connected neural network to perform feature extraction processing on the features of the initial sketch point pair to obtain processed features of the initial sketch point pair;

[0025] An activation function is used to classify the processed initial sketch point pair features to obtain the connection probability of the initial sketch point pair.

[0026] Optionally, generating a target sketch map based on the connection probability and the position information of the initial sketch point includes:

[0027] Determine a first adjacency matrix corresponding to the connection probability of the initial sketch point pair, wherein the first adjacency matrix is ​​used to characterize the connection relationship of the initial sketch points;

[0028] A target sketch map is generated based on the first adjacency matrix and the position information of the initial sketch points.

[0029] Optionally, generating a target sketch map based on the first adjacency matrix and the position information of the initial sketch points includes:

[0030] Generate an initial sketch map based on the first adjacency matrix and the position information of the initial sketch points;

[0031] According to the initial sketch, abnormal lines in the initial sketch are determined, wherein the abnormal lines include redundant lines in the initial sketch;

[0032] Assigning 0 to the elements in the first adjacency matrix corresponding to the redundant lines, to obtain a second adjacency matrix;

[0033] A target sketch map is generated based on the second adjacency matrix and the position information of the initial sketch points.

[0034] Optionally, the abnormal lines also include disconnected lines in the initial sketch map, and before generating the target sketch map based on the second adjacency matrix and the position information of the initial sketch points, the method further includes:

[0035] The elements in the first adjacency matrix corresponding to the disconnected lines are assigned a value of 1 to obtain a second adjacency matrix.

[0036] In a second aspect, an embodiment of the present invention provides a sketch generating device, comprising:

[0037] A detection module, used for performing edge detection on the image to be processed to obtain an edge map of the image to be processed;

[0038] A first determining module is used to determine initial sketch points in the edge map, wherein the initial sketch points include sketch points in the edge map that are edge points;

[0039] A second determination module is used to determine the connection probability of an initial sketch point pair based on the image features of the edge map and the position information of the initial sketch points, wherein the initial sketch point pair includes any two initial sketch points in the initial sketch points in the edge map;

[0040] A generation module is used to generate a target sketch map based on the connection probability of the initial sketch point pairs and the position information of the initial sketch points.

[0041] Optionally, the first determining module includes:

[0042] An extraction unit, used for performing feature extraction on the edge map to obtain image features of the edge map;

[0043] a dot multiplication unit, configured to perform dot multiplication processing on the image features of the edge map and the edge map to obtain at least two first heat maps, wherein the at least two first heat maps are respectively used to characterize the first category probability of each edge pixel in the edge map, and the first category probability is the probability that each edge pixel belongs to the category corresponding to the first heat map;

[0044] The first determining unit is used to determine the initial sketch point in the edge map based on the at least two first thermal maps.

[0045] Optionally, the first determining unit is used to:

[0046] Performing non-maximum suppression processing on the at least two first heat maps to obtain at least two second heat maps, wherein the at least two second heat maps are respectively used to characterize the second category probability of each edge pixel point in the edge map;

[0047] Determine the probability of the category to which each edge pixel belongs according to the values ​​of each edge pixel in the at least two second heat maps;

[0048] Determine the category to which each edge pixel belongs according to the probability of the category to which each edge pixel belongs, the category to which each edge pixel belongs being the category to which each edge pixel belongs with the greatest probability;

[0049] Based on the category to which each edge pixel point belongs, an initial sketch point in the edge map is determined.

[0050] Optionally, the second determining module includes:

[0051] A sampling unit, configured to sample the lines of the initial sketch point pairs based on the position information of the initial sketch point pairs to obtain a sampling point sequence of the initial sketch point pairs;

[0052] A second determining unit, configured to determine features of the initial sketch point pair based on the sampling point sequence and image features of the edge map;

[0053] The third determining unit is used to determine the connection probability of the initial sketch point pair based on the characteristics of the initial sketch point pair.

[0054] Optionally, the third determining unit is used to:

[0055] Using a convolutional neural network and a fully connected neural network to perform feature extraction processing on the features of the initial sketch point pair to obtain processed features of the initial sketch point pair;

[0056] An activation function is used to classify the processed initial sketch point pair features to obtain the connection probability of the initial sketch point pair.

[0057] Optionally, the generating module includes:

[0058] a fourth determining unit, configured to determine a first adjacency matrix corresponding to the connection probability of the initial sketch point pairs, wherein the first adjacency matrix is ​​used to characterize the connection relationship of the initial sketch points;

[0059] A generating unit is used to generate a target sketch map based on the first adjacency matrix and the position information of the initial sketch points.

[0060] Optionally, the generating unit is used to:

[0061] Generate an initial sketch map based on the first adjacency matrix and the position information of the initial sketch points;

[0062] According to the initial sketch, abnormal lines in the initial sketch are determined, wherein the abnormal lines include redundant lines in the initial sketch;

[0063] Assigning 0 to the elements in the first adjacency matrix corresponding to the redundant lines, to obtain a second adjacency matrix;

[0064] A target sketch map is generated based on the second adjacency matrix and the position information of the initial sketch points.

[0065] Optionally, the abnormal lines also include disconnected lines in the initial sketch image, and the generating unit is further used for:

[0066] Used to assign 1 to the elements in the first adjacency matrix corresponding to the disconnected lines, so as to obtain a second adjacency matrix.

[0067] In a third aspect, an embodiment of the present invention provides a sketch generation device, wherein the sketch extraction device comprises: a transceiver, a memory, a processor, and a program stored in the memory and executable on the processor; the processor is used to read the program in the memory to implement the steps in the method described in the first aspect.

[0068] In a fourth aspect, an embodiment of the present invention provides a readable storage medium, on which a program or instruction is stored, wherein the program or instruction, when executed by a processor, implements the steps in the method described in the first aspect.

[0069] In an embodiment of the present invention, the sketch image generation method can obtain an edge map of the image to be processed by performing edge detection on the image to be processed, and select edge pixel points in the edge map to obtain initial sketch points, and then determine the connection probability of the initial sketch point pairs according to the image features of the edge map, thereby generating a target sketch image. The target sketch image can be represented as a set of sketch points and sketch lines. The set of sketch points and sketch lines can describe the pixel changes, geometric feature distribution and structural information of the image, so that the target sketch image better retains the semantic features of the image, greatly reduces the amount of data, and has higher accuracy and stronger representation ability. When used for image encoding, the encoding efficiency is higher. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0071] Figure 1 This is one of the flow charts of a method for generating a sketch provided by an embodiment of the present invention;

[0072] Figure 2 This is a second flowchart of a method for generating a sketch provided by an embodiment of the present invention;

[0073] Figure 3 It is a structural schematic diagram of a sketch image generating device provided by an embodiment of the present invention;

[0074] Figure 4 It is a structural schematic diagram of a sketch image generating device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0075] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0076] Unless otherwise defined, the technical terms or scientific terms used in the present invention shall have the usual meanings understood by persons with ordinary skills in the field to which the present invention belongs. The words "first", "second" and similar words used in the present invention do not indicate any order, quantity or importance, but are only used to distinguish different components. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship also changes accordingly.

[0077] See also Figure 1 , Figure 1 is one of the flowcharts of a sketch image generation method provided by an embodiment of the present invention, the sketch image generation method comprising:

[0078] Step 101: perform edge detection on an image to be processed to obtain an edge map of the image to be processed.

[0079] Specifically, the image to be processed can be any image, such as a photo, a video frame, etc.; the edge detection can be performed by a feature extraction module of an hourglass structure, wherein the number of upsampling and downsampling in the hourglass structure is the same, and the feature extraction module of the hourglass structure includes multiple convolution layers of different scales, the last convolution layer has a single convolution kernel, and the convolution layer is followed by a sigmoid activation function to obtain the probability that each pixel of the image belongs to the edge; after extracting the edge probability map, in order to facilitate subsequent processing, a single-pixel edge can be obtained by non-maximum suppression, and non-maximum suppression searches for the edge strength value of the pixel in the gradient direction to determine whether the pixel has obtained a local maximum, thereby suppressing non-maximum points and obtaining a single-pixel edge strength map, and then according to the probability that each pixel of the image belongs to the edge, an edge map of the image can be obtained. For example, a pixel whose probability of belonging to the edge is greater than a threshold can be determined as an edge pixel in the edge map.

[0080] Step 102, determining initial sketch points in the edge map, wherein the initial sketch points include sketch points in the edge map that are edge points.

[0081] Specifically, the initial sketch points include sketch points in the edge map that are edge points, and may include intersection points, inflection points and endpoints in the sketch points, wherein an intersection point indicates that the point connects more than or equal to three sketch points, an inflection point indicates that the point connects two sketch points, and an endpoint indicates that the point connects one sketch point; the determination of the initial sketch points in the edge map may be to first determine the category of each edge pixel point in the edge map, and after determining the category of each edge pixel point, the initial sketch points may be determined, wherein the categories of edge pixel points may include intersection points, inflection points, endpoints, non-sketch points and non-edge points, and edge pixel points whose categories are not non-sketch points may be determined as initial sketch points.

[0082] Step 103, determining the connection probability of an initial sketch point pair based on the image features of the edge map and the position information of the initial sketch points, wherein the initial sketch point pair includes any two initial sketch points in the edge map.

[0083] Specifically, the image features of the edge map may be edge features of the edge map extracted by a feature extraction module of an hourglass structure, or a feature extraction module of other structures may be used. The position information of the initial sketch point may be the two-dimensional coordinates of the initial sketch point, and the connection probability of the initial sketch point pair may be the probability that two initial sketch points in the initial sketch point pair are connected.

[0084] Step 104: Generate a target sketch map based on the connection probability of the initial sketch point pairs and the position information of the initial sketch points.

[0085] Specifically, the target sketch map can be an image generated by visualizing the sketch map based on the connection probability of the initial sketch point pairs and the position information of the initial sketch points, or it can be represented as the coordinates of the initial sketch points and the adjacency matrix corresponding to the initial sketch points. The adjacency matrix can be obtained based on the connection probability of the initial sketch point pairs; the target sketch map can be directly used for compression encoding and decoding, and the sketch points in the target sketch map can be rearranged so that the sketch points on the same edge line are connected in sequence. According to the rearrangement rule of each sketch point, the optimized adjacency matrix is ​​adjusted and XORed with the ideal adjacency matrix to obtain an output matrix. The ideal adjacency matrix can be an adjacency matrix in which all elements on the secondary diagonal are 1 and all other elements are 0. The rearranged coordinates of the optimized sketch points are quantized and encoded to obtain a sketch point code stream, and the positions of the elements that are 1 in the output matrix are encoded to obtain an adjacency matrix code stream. According to the sketch point code stream and the adjacency matrix code stream, a sketch map code stream is obtained.

[0086] For example, a specific compression coding and decoding method may include adaptive sketch compression, sketch rearrangement and sketch encoding. Adaptive sketch compression may be that when performing sketch compression, the distance from each inflection point to the straight line formed by the sketch point pair connected to it is first calculated, and the sketch points are removed in order of distance from low to high. After removing a sketch point, the rows and columns of its corresponding adjacency matrix are also removed, and the rows and columns of its two adjacent sketch points in the adjacency matrix are updated, that is, their corresponding positions are set to 1. After removing a sketch point, it is necessary to update the corresponding distance of the two sketch points connected to it. The above process is repeated until the amount of sketch data obtained is lower than the set value, that is, the adaptive sketch compression process is completed. Adaptive sketch compression is a method for simplifying the sketch after obtaining the target sketch, thereby reducing the number of sketch points and the dimension of the adjacency matrix, thereby achieving a sketch with less data without affecting its representation performance. The algorithm performs adaptive compression according to the set data volume limit to meet the set data volume requirement.

[0087] Sketch rearrangement may be to rearrange the pixels in the sketch after generating the target sketch and compressing the sketch, so that the pixels on an edge line are arranged in order, and the adjacency matrix is ​​also rearranged accordingly.

[0088] Sketch image encoding can be performed by performing an exclusive OR (XOR) operation on the rearranged adjacency matrix and the ideal adjacency matrix to obtain an output matrix, wherein the elements on the sub-diagonal of the ideal adjacency matrix are all 1 and the remaining elements are all 0, encoding the positions of the elements that are 1 in the output matrix to obtain an adjacency matrix code stream, and encoding the rearranged sketch point coordinates into a sketch point code stream, and the two together constitute the code stream after the sketch image is encoded. Since the sketch points in the sketch image are usually connected to form long and continuous lines, most of the 1s in the rearranged adjacency matrix are concentrated on the sub-diagonal, while there are fewer sub-diagonal elements with non-0 values ​​and elements in other positions that are 1. Therefore, in the result of the exclusive OR operation, encoding the position of 1 can greatly reduce the amount of data.

[0089] In an embodiment of the present invention, the sketch image generation method can obtain an edge map of the image to be processed by performing edge detection on the image to be processed, and select edge pixel points in the edge map to obtain initial sketch points, and then determine the connection probability of the initial sketch point pairs according to the image features of the edge map, thereby generating a target sketch image. The target sketch image can be represented as a set of sketch points and sketch lines. The set of sketch points and sketch lines can describe the pixel changes, geometric feature distribution and structural information of the image, so that the target sketch image better retains the semantic features of the image, greatly reduces the amount of data, and has higher accuracy and stronger representation ability. When used for image encoding, the encoding efficiency is higher.

[0090] Optionally, determining the initial sketch point in the edge map includes:

[0091] Performing feature extraction on the edge map to obtain image features of the edge map;

[0092] Performing a dot multiplication process on the image feature of the edge map and the edge map to obtain at least two first heat maps, wherein the at least two first heat maps are respectively used to characterize the first category probability of each edge pixel in the edge map, and the first category probability is the probability that each edge pixel belongs to the category corresponding to the first heat map;

[0093] Based on the at least two first heat maps, initial sketch points in the edge map are determined.

[0094] Specifically, the feature extraction of the edge map may be performed by using a feature extraction module with an hourglass structure, the image feature of the edge map may be the edge feature of the edge map, the coordinates of each pixel point included in the first heat map correspond one-to-one with the coordinates of each edge pixel point included in the edge map, and the first category probability may be the value of each pixel point on the first heat map, and the value of each pixel point on the first heat map represents the probability that the edge pixel point corresponding to the pixel point belongs to the category corresponding to the heat map.

[0095] In an embodiment of the present invention, the sketch image generation method can generate at least two first heat maps, and the at least two first heat maps are respectively used to characterize the first category probability of each edge pixel point in the edge image, determine the category of each edge pixel point in the edge image, and then determine the initial sketch point, so that the selection of the initial sketch point is more accurate, thereby improving the accuracy of the generated target sketch image.

[0096] Optionally, determining the initial sketch point in the edge map based on the at least two first heat maps includes:

[0097] Performing non-maximum suppression processing on the at least two first heat maps to obtain at least two second heat maps, wherein the at least two second heat maps are respectively used to characterize the second category probability of each edge pixel point in the edge map;

[0098] Determine the probability of the category to which each edge pixel belongs according to the values ​​of each edge pixel in the at least two second heat maps;

[0099] Determine the category to which each edge pixel belongs according to the probability of the category to which each edge pixel belongs, the category to which each edge pixel belongs being the category to which each edge pixel belongs with the greatest probability;

[0100] Based on the category to which each edge pixel point belongs, an initial sketch point in the edge map is determined.

[0101] Specifically, the non-maximum suppression processing may be performing non-maximum suppression on pixel points in the at least two first heat maps, wherein all points suppressed by the non-maximum suppression operation are treated as non-sketch points and do not participate in the generation of subsequent sketch maps; the second category probability may be the probability of the category to which the pixel points in the at least two second heat maps belong; the category to which each edge pixel point belongs may include intersection points, inflection points, endpoints, non-sketch points and non-edge points, and edge pixel points whose categories are not non-sketch points are determined as initial sketch points, and the two-dimensional coordinates of each initial sketch point may be obtained, and the initial sketch point may include intersection points, inflection points and endpoints.

[0102] In an embodiment of the present invention, the sketch image generation method can perform non-maximum suppression on the pixel points in the at least two first thermograms so that the respective pixel points in the at least two second thermograms obtained constitute a single-pixel thermogram, thereby making the subsequently obtained target sketch image more accurate.

[0103] Optionally, determining the connection probability of the initial sketch point pair based on the image features of the edge map and the position information of the initial sketch point includes:

[0104] Based on the position information of the initial sketch point, sampling is performed on the connecting line of the initial sketch point pair to obtain a sampling point sequence of the initial sketch point pair;

[0105] Determining features of the initial sketch point pair based on the sampling point sequence and image features of the edge map;

[0106] Based on the features of the initial sketch point pairs, a connection probability of the initial sketch point pairs is determined.

[0107] Specifically, the sampling of the line connecting the initial sketch point pairs may be performed by uniformly sampling a certain number of times on the line segment connecting the two initial sketch points of the initial sketch point pairs to obtain a sampling point sequence of the initial sketch point pairs, wherein the sampling point sequence includes the initial sketch point pairs and points sampled at equal intervals on the line connecting the initial sketch point pairs, wherein the number of sampling points of each initial sketch point pair is the same; the image features of the edge map may be edge features of the edge map, and the features of the initial sketch point pairs may be determined by sampling at each sampling point position in the sampling point sequence of each initial sketch point pair based on the edge features to obtain the features of the initial sketch point pairs.

[0108] For the determination of connection probability, the dimension sampled from the feature of the initial sketch point pair is proportional to the square of the number of initial sketch points. Since spatially adjacent initial sketch points are more likely to have a connection relationship, the divide-and-conquer method can be used to reduce the computational pressure. Simple spatial grouping is performed on all initial sketch points, and the connection relationship of each group of initial sketch points is detected. In order to make up for the connection relationship of the initial sketch point pair that is lost due to the division of two initial sketch points into different groups, all unsaturated initial sketch points can be collected again for connection relationship detection, and a rough sketch map can be obtained. Unsaturated initial sketch points can be all initial sketch points whose number of detected connection relationships is less than their ideal degree, where the ideal degree of the intersection point is 4, the ideal degree of the inflection point is 2, and the ideal degree of the endpoint is 1.

[0109] The connection relationship detection based on the divide-and-conquer method is implemented by setting the detection quantity threshold. Each time, the current initial sketch point is divided into two groups of equal number according to the proximity criterion of the spatial coordinates. If the number of initial sketch points in one group after grouping is lower than the detection quantity threshold, the connection relationship detection is performed directly; if it is higher than the detection quantity threshold, it is divided again until the number of initial sketch points in all groups is lower than the detection quantity threshold.

[0110] After each group of initial sketch points is detected, the connection relationship can be corrected to make the target sketch more robust and accurate. For example, when three initial sketch points are connected to each other and almost arranged in a line, the connection of the pair with the longest distance will be cancelled; if there are initial sketch points whose current degrees are greater than the theoretical values, the weakest connection can be cancelled according to the positions of the connected points; if there is a weak connection between two unsaturated initial sketch points, their connection will be strengthened to compensate for the possibility of lost connection.

[0111] In an embodiment of the present invention, the sketch image generation method can obtain the connection probability of the initial sketch point pairs by performing a certain number of uniform sampling on the line segments connected by the initial sketch point pairs, so that the accuracy of the connection probability is relatively high, and then determine the connection relationship of the initial sketch point pairs, thereby improving the accuracy of target sketch image extraction.

[0112] Optionally, determining the connection probability of the initial sketch point pair based on the features of the initial sketch point pair includes:

[0113] Using a convolutional neural network and a fully connected neural network to perform feature extraction processing on the features of the initial sketch point pair to obtain processed features of the initial sketch point pair;

[0114] An activation function is used to classify the processed initial sketch point pair features to obtain the connection probability of the initial sketch point pair.

[0115] Specifically, the processed initial sketch point pair features may be depth features that can reflect whether the initial sketch point pairs are connected, and the activation function may be a sigmoid activation function.

[0116] In an embodiment of the present invention, the sketch image generation method can extract deep features reflecting whether the initial sketch point pairs are connected by using a convolutional neural network and a fully connected neural network, and use an activation function to obtain the connection probability of the initial sketch point pairs, thereby improving the accuracy of determining the connection probability of the initial sketch point pairs.

[0117] Optionally, generating a target sketch map based on the connection probability and the position information of the initial sketch point includes:

[0118] Determine a first adjacency matrix corresponding to the connection probability of the initial sketch point pair, wherein the first adjacency matrix is ​​used to characterize the connection relationship of the initial sketch points;

[0119] A target sketch map is generated based on the first adjacency matrix and the position information of the initial sketch points.

[0120] Specifically, based on the connection probability of the initial sketch point pairs, it can be determined whether there is a connection relationship between each initial sketch point pair, and then the first adjacency matrix can be obtained. The first adjacency matrix can be obtained by adjusting the initial adjacency matrix accordingly according to the connection probability of the initial sketch point pairs.

[0121] For example, the initial adjacency matrix can be an adjacency matrix in which all elements except the main diagonal are 0, indicating that there may be connections between all initial sketch points. When the probability of the initial sketch point pairs being connected is greater than the probability threshold, it is determined that the initial sketch point pairs can be connected. Then, when the initial sketch point pairs need to be connected, the initial sketch point pairs are connected, that is, the corresponding positions in the initial adjacency matrix are assigned a value of 1; when the initial sketch point pairs are not connected, the connection between the initial sketch point pairs is disconnected, that is, the corresponding positions in the initial adjacency matrix are assigned a value of 0. According to whether the initial sketch point pairs are connected, the first adjacency matrix can be obtained.

[0122] The position information of the initial sketch point may be the two-dimensional coordinates of the initial sketch point, and the target sketch map may be the initial sketch map obtained by visualizing the sketch map according to the first adjacency matrix and the coordinates of the initial sketch point. The target sketch map may also be directly represented as a dictionary consisting of N×2 point positions (two-dimensional coordinates of the initial sketch point) and an N×N first adjacency matrix, where N is the number of the initial sketch points.

[0123] In an embodiment of the present invention, the sketch image generation method can determine the first adjacency matrix through the connection probability of the initial sketch point pairs, and generate a target sketch image according to the first adjacency matrix and the position information of the initial sketch points, so that the target sketch image can be represented as a set of sketch points and sketch lines, and the set of sketch points and sketch lines can describe the pixel changes, geometric feature distribution and structural information of the image, so that the sketch image retains the semantic features of the image.

[0124] Optionally, generating a target sketch map based on the first adjacency matrix and the position information of the initial sketch points includes:

[0125] Generate an initial sketch map based on the first adjacency matrix and the position information of the initial sketch points;

[0126] According to the initial sketch, abnormal lines in the initial sketch are determined, wherein the abnormal lines include redundant lines in the initial sketch;

[0127] Assigning 0 to the elements in the first adjacency matrix corresponding to the redundant lines, to obtain a second adjacency matrix;

[0128] A target sketch map is generated based on the second adjacency matrix and the position information of the initial sketch points.

[0129] Specifically, the redundant lines may include incorrectly drawn connecting lines, repeatedly drawn connecting lines and non-maximum connecting lines. Incorrectly drawn connecting lines represent connections that lead to unexpected lines, which are completely inconsistent with the original single-pixel edge map; repeatedly drawn connecting lines represent connections that have been covered by other lines, which are redundant connection relationships and have no effect on the accuracy of the extracted sketch image after removal; non-maximum connecting lines are connecting lines whose intensity is not the largest among the thicker adjacent parallel lines in the restored sketch image.

[0130] The elements in the first adjacency matrix are 1, indicating that the corresponding initial sketch points are connected, and the elements in the first adjacency matrix are 0, indicating that the corresponding initial sketch points are not connected. The redundant lines in the initial sketch can be identified, and the elements in the first adjacency matrix corresponding to the redundant lines are assigned 0, so that the redundant lines can be eliminated to obtain the second adjacency matrix. The target sketch generated based on the second adjacency matrix removes the redundant lines.

[0131] In an embodiment of the present invention, the sketch image generation method can generate an initial sketch image based on the first adjacency matrix, identify redundant lines in the initial sketch image, and obtain a target sketch image with the redundant lines removed by reassigning elements in the first adjacency matrix corresponding to the redundant lines, thereby making the target sketch image more accurate, reducing the amount of image data, and improving image coding efficiency.

[0132] Optionally, the abnormal lines also include disconnected lines in the initial sketch map, and before generating the target sketch map based on the second adjacency matrix and the position information of the initial sketch points, the method further includes:

[0133] The elements in the first adjacency matrix corresponding to the disconnected lines are assigned a value of 1 to obtain a second adjacency matrix.

[0134] Specifically, the broken lines may be accidental breaks in the lines, which are completely inconsistent with the original single-pixel edge map. The broken lines in the initial sketch map may be identified, and corresponding elements in the first adjacency matrix corresponding to the connected broken lines may be assigned a value of 1 to obtain a second adjacency matrix.

[0135] In an embodiment of the present invention, the sketch generation method can generate an initial sketch based on the first adjacency matrix, identify broken lines in the initial sketch, and obtain a target sketch connecting the broken lines by reassigning elements in the first adjacency matrix corresponding to the broken lines, thereby making the target sketch more accurate.

[0136] Figure 2 This is a second flowchart of a method for generating a sketch provided by an embodiment of the present invention. Figure 2As shown, firstly, an image to be processed is input, and an hourglass module is used to perform edge detection on the image to be processed to obtain an edge map of the image to be processed; then, the hourglass module is used to perform feature extraction on the edge map to obtain image features of the edge map, and the image features of the edge map and the edge map are subjected to point multiplication processing to obtain at least two first heat maps; non-maximum suppression is performed on the at least two first heat maps to obtain at least two second heat maps, and the maximum value position in the at least two second heat maps is detected to determine the category to which each edge pixel point belongs, thereby determining the initial sketch point, that is, according to the position of each edge pixel point in the at least two second heat maps, the initial sketch point is determined. The value of the heat map is determined to determine the probability of the category to which each edge pixel belongs, and the category to which each edge pixel belongs is the category to which each edge pixel belongs with the greatest probability; the initial sketch points are paired to obtain initial sketch point pairs, and the initial sketch point pairs are interpolated to obtain a sampling point sequence of the initial sketch point pairs; sampling is performed based on the sampling point sequence and the edge features of the edge map to determine the features of the initial sketch point pairs; a convolutional neural network and an activation function are used to perform feature extraction and shape processing on the features of the initial sketch point pairs to obtain an adjacency matrix, and a target sketch map can be generated according to the adjacency matrix; wherein, Figure 2 As shown, the hourglass module includes an upsampling module and a downsampling module, and the number of upsampling is the same as the number of downsampling.

[0137] See also Figure 3 , Figure 3 is a schematic diagram of the structure of a sketch image generating device provided by an embodiment of the present invention. Figure 3 As shown, the sketch image generating device 300 includes:

[0138] A detection module 301 is used to perform edge detection on the image to be processed to obtain an edge map of the image to be processed;

[0139] A first determining module 302 is used to determine initial sketch points in the edge map, wherein the initial sketch points include sketch points in the edge map that are edge points;

[0140] A second determination module 303 is used to determine the connection probability of an initial sketch point pair based on the image features of the edge map and the position information of the initial sketch points, wherein the initial sketch point pair includes any two initial sketch points in the initial sketch points in the edge map;

[0141] The generation module 304 is used to generate a target sketch map based on the connection probability of the initial sketch point pairs and the position information of the initial sketch points.

[0142] Specifically, the sketch image generating device may be configured with an image processing model, and the training method of the image processing model may be: pre-acquiring a plurality of image samples and sketch sample images of the image samples, wherein the image samples may be images in the BSDS500 data set, and the sketch image samples of the image samples may be generated by using a sketch image generating method of the related technology;

[0143] The image sample is input into the initial model, and the initial model can output a predicted sketch of the image sample, wherein the model architecture of the initial model and the image processing model is the same, except that the model parameters in the initial model have not been trained yet;

[0144] According to the difference between the predicted sketch and the sketch sample, a cross entropy loss function can be established. The initial model is trained according to the cross entropy loss function. During the training process, the model parameters of the initial model are continuously updated to finally obtain an image processing model.

[0145] Among them, the cross entropy loss function can include binary cross entropy loss function, multivariate cross entropy loss function and weighted cross entropy loss function. The binary cross entropy loss function can be used for edge detection and connection relationship detection, the multivariate cross entropy loss function can be used for sketch point detection, and the weighted cross entropy loss function can solve the problem of sample imbalance during edge detection and sketch point detection.

[0146] Optionally, the first determining module 302 includes:

[0147] An extraction unit, used for performing feature extraction on the edge map to obtain image features of the edge map;

[0148] a dot multiplication unit, configured to perform dot multiplication processing on the image features of the edge map and the edge map to obtain at least two first heat maps, wherein the at least two first heat maps are respectively used to characterize the first category probability of each edge pixel in the edge map, and the first category probability is the probability that each edge pixel belongs to the category corresponding to the first heat map;

[0149] The first determining unit is used to determine the initial sketch point in the edge map based on the at least two first thermal maps.

[0150] Optionally, the first determining unit is used to:

[0151] Performing non-maximum suppression processing on the at least two first heat maps to obtain at least two second heat maps, wherein the at least two second heat maps are respectively used to characterize the second category probability of each edge pixel point in the edge map;

[0152] Determine the probability of the category to which each edge pixel belongs according to the values ​​of each edge pixel in the at least two second heat maps;

[0153] Determine the category to which each edge pixel belongs according to the probability of the category to which each edge pixel belongs, the category to which each edge pixel belongs being the category to which each edge pixel belongs with the greatest probability;

[0154] Based on the category to which each edge pixel point belongs, an initial sketch point in the edge map is determined.

[0155] Optionally, the second determining module 303 includes:

[0156] A sampling unit, configured to sample the lines of the initial sketch point pairs based on the position information of the initial sketch point pairs to obtain a sampling point sequence of the initial sketch point pairs;

[0157] A second determining unit, configured to determine features of the initial sketch point pair based on the sampling point sequence and image features of the edge map;

[0158] The third determining unit is used to determine the connection probability of the initial sketch point pair based on the characteristics of the initial sketch point pair.

[0159] Optionally, the third determining unit is used to:

[0160] Using a convolutional neural network and a fully connected neural network to perform feature extraction processing on the features of the initial sketch point pair to obtain processed features of the initial sketch point pair;

[0161] An activation function is used to classify the processed initial sketch point pair features to obtain the connection probability of the initial sketch point pair.

[0162] Optionally, the generating module 304 includes:

[0163] a fourth determining unit, configured to determine a first adjacency matrix corresponding to the connection probability of the initial sketch point pairs, wherein the first adjacency matrix is ​​used to characterize the connection relationship of the initial sketch points;

[0164] A generating unit is used to generate a target sketch map based on the first adjacency matrix and the position information of the initial sketch points.

[0165] Optionally, the generating unit is used to:

[0166] Generate an initial sketch map based on the first adjacency matrix and the position information of the initial sketch points;

[0167] According to the initial sketch, abnormal lines in the initial sketch are determined, wherein the abnormal lines include redundant lines in the initial sketch;

[0168] Assigning 0 to the elements in the first adjacency matrix corresponding to the redundant lines, to obtain a second adjacency matrix;

[0169] A target sketch map is generated based on the second adjacency matrix and the position information of the initial sketch points.

[0170] Optionally, the abnormal lines also include disconnected lines in the initial sketch image, and the generating unit is further used for:

[0171] Used to assign 1 to the elements in the first adjacency matrix corresponding to the disconnected lines, so as to obtain a second adjacency matrix.

[0172] See also Figure 4 , Figure 4 is a schematic diagram of the structure of a sketch image generating device provided by an embodiment of the present invention, such as Figure 4 As shown, the sketch image generating device includes a bus 401, a transceiver 402, an antenna 403, a bus interface 404, a processor 405 and a memory 406:

[0173] The processor 405 is used to: perform edge detection on the image to be processed to obtain an edge map of the image to be processed;

[0174] Determine initial sketch points in the edge map, the initial sketch points comprising sketch points in the edge map that are edge points;

[0175] Determine, based on the image features of the edge map and the position information of the initial sketch points, a connection probability of an initial sketch point pair, wherein the initial sketch point pair includes any two initial sketch points among the initial sketch points in the edge map;

[0176] A target sketch map is generated based on the connection probability of the initial sketch point pairs and the position information of the initial sketch points.

[0177] Optionally, the processor 405 is specifically configured to:

[0178] Performing feature extraction on the edge map to obtain image features of the edge map;

[0179] Performing a dot multiplication process on the image feature of the edge map and the edge map to obtain at least two first heat maps, wherein the at least two first heat maps are respectively used to characterize the first category probability of each edge pixel in the edge map, and the first category probability is the probability that each edge pixel belongs to the category corresponding to the first heat map;

[0180] Based on the at least two first heat maps, initial sketch points in the edge map are determined.

[0181] Optionally, the processor 405 is specifically configured to:

[0182] Performing non-maximum suppression processing on the at least two first heat maps to obtain at least two second heat maps, wherein the at least two second heat maps are respectively used to characterize the second category probability of each edge pixel point in the edge map;

[0183] Determine the probability of the category to which each edge pixel belongs according to the values ​​of each edge pixel in the at least two second heat maps;

[0184] Determine the category to which each edge pixel belongs according to the probability of the category to which each edge pixel belongs, the category to which each edge pixel belongs being the category to which each edge pixel belongs with the greatest probability;

[0185] Based on the category to which each edge pixel point belongs, an initial sketch point in the edge map is determined.

[0186] Optionally, the processor 405 is specifically configured to:

[0187] Based on the position information of the initial sketch point, sampling is performed on the connecting line of the initial sketch point pair to obtain a sampling point sequence of the initial sketch point pair;

[0188] Determining features of the initial sketch point pair based on the sampling point sequence and image features of the edge map;

[0189] Based on the features of the initial sketch point pairs, a connection probability of the initial sketch point pairs is determined.

[0190] Optionally, the processor 405 is specifically configured to:

[0191] Using a convolutional neural network and a fully connected neural network to perform feature extraction processing on the features of the initial sketch point pair to obtain processed features of the initial sketch point pair;

[0192] An activation function is used to classify the processed initial sketch point pair features to obtain the connection probability of the initial sketch point pair.

[0193] Optionally, the processor 405 is specifically configured to:

[0194] Determine a first adjacency matrix corresponding to the connection probability of the initial sketch point pair, wherein the first adjacency matrix is ​​used to characterize the connection relationship of the initial sketch points;

[0195] A target sketch map is generated based on the first adjacency matrix and the position information of the initial sketch points.

[0196] Optionally, the processor 405 is specifically configured to:

[0197] Generate an initial sketch map based on the first adjacency matrix and the position information of the initial sketch points;

[0198] According to the initial sketch, abnormal lines in the initial sketch are determined, wherein the abnormal lines include redundant lines in the initial sketch;

[0199] Assigning 0 to the elements in the first adjacency matrix corresponding to the redundant lines, to obtain a second adjacency matrix;

[0200] A target sketch map is generated based on the second adjacency matrix and the position information of the initial sketch points.

[0201] Optionally, the processor 405 is further configured to:

[0202] The elements in the first adjacency matrix corresponding to the disconnected lines are assigned a value of 1 to obtain a second adjacency matrix.

[0203] exist Figure 4 In the embodiment, a bus architecture (represented by bus 401) is shown, bus 401 may include any number of interconnected buses and bridges, bus 401 links various circuits including one or more processors represented by processor 405 and memory represented by memory 406. Bus 401 may also link various other circuits such as peripherals, voltage regulators, and power management circuits, which are well known in the art and are therefore not further described herein. Bus interface 404 provides an interface between bus 401 and transceiver 402. Transceiver 402 may be one element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices on a transmission medium. Data processed by processor 405 is transmitted on a wireless medium via antenna 403, and further, antenna 403 also receives data and transmits the data to processor 405.

[0204] Processor 405 is responsible for managing bus 401 and general processing, and may also provide various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. Memory 406 may be used to store data used by processor 405 when performing operations.

[0205] Optionally, the processor 405 may be a CPU, an ASIC, an FPGA or a CPLD.

[0206] The embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, each process of the auxiliary device state control method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it is not repeated here. The computer-readable storage medium is, for example, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0207] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0208] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0209] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation modes, which are merely illustrative rather than restrictive. Under the guidance of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the present invention and the claims, all of which are within the protection of the present invention.

Claims

1. A method for generating a sketch, characterized in that: include: Performing edge detection on the image to be processed to obtain an edge map of the image to be processed; Determine initial sketch points in the edge map, the initial sketch points comprising sketch points in the edge map that are edge points; Determine, based on the image features of the edge map and the position information of the initial sketch points, a connection probability of an initial sketch point pair, wherein the initial sketch point pair includes any two initial sketch points among the initial sketch points in the edge map; A target sketch map is generated based on the connection probability of the initial sketch point pairs and the position information of the initial sketch points.

2. The method according to claim 1, characterized in that The determining of the initial sketch point in the edge map comprises: Performing feature extraction on the edge map to obtain image features of the edge map; Performing a dot multiplication process on the image feature of the edge map and the edge map to obtain at least two first heat maps, wherein the at least two first heat maps are respectively used to characterize the first category probability of each edge pixel in the edge map, and the first category probability is the probability that each edge pixel belongs to the category corresponding to the first heat map; Based on the at least two first heat maps, initial sketch points in the edge map are determined.

3. The method according to claim 2, characterized in that The step of determining the initial sketch point in the edge map based on the at least two first thermal maps comprises: Performing non-maximum suppression processing on the at least two first heat maps to obtain at least two second heat maps, wherein the at least two second heat maps are respectively used to characterize the second category probability of each edge pixel point in the edge map; Determine the probability of the category to which each edge pixel belongs according to the values ​​of each edge pixel in the at least two second heat maps; Determine the category to which each edge pixel belongs according to the probability of the category to which each edge pixel belongs, the category to which each edge pixel belongs being the category to which each edge pixel belongs with the greatest probability; Based on the category to which each edge pixel point belongs, an initial sketch point in the edge map is determined.

4. The method according to claim 1, characterized in that: The determining the connection probability of the initial sketch point pair based on the image features of the edge map and the position information of the initial sketch point includes: Based on the position information of the initial sketch point, sampling is performed on the connecting line of the initial sketch point pair to obtain a sampling point sequence of the initial sketch point pair; Determining features of the initial sketch point pair based on the sampling point sequence and image features of the edge map; Based on the features of the initial sketch point pairs, a connection probability of the initial sketch point pairs is determined.

5. The method according to claim 4, characterized in that The determining, based on the features of the initial sketch point pairs, the connection probability of the initial sketch point pairs comprises: Using a convolutional neural network and a fully connected neural network to perform feature extraction processing on the features of the initial sketch point pair to obtain processed features of the initial sketch point pair; An activation function is used to classify the processed initial sketch point pair features to obtain the connection probability of the initial sketch point pair.

6. The method according to claim 1, characterized in that The step of generating a target sketch map based on the connection probability and the position information of the initial sketch point includes: Determine a first adjacency matrix corresponding to the connection probability of the initial sketch point pair, wherein the first adjacency matrix is ​​used to characterize the connection relationship of the initial sketch points; A target sketch map is generated based on the first adjacency matrix and the position information of the initial sketch points.

7. The method according to claim 6, characterized in that The step of generating a target sketch map based on the first adjacency matrix and the position information of the initial sketch points includes: Generate an initial sketch map based on the first adjacency matrix and the position information of the initial sketch points; According to the initial sketch, abnormal lines in the initial sketch are determined, wherein the abnormal lines include redundant lines in the initial sketch; Assigning 0 to the elements in the first adjacency matrix corresponding to the redundant lines, to obtain a second adjacency matrix; A target sketch map is generated based on the second adjacency matrix and the position information of the initial sketch points.

8. The method according to claim 7, characterized in that The abnormal lines also include disconnected lines in the initial sketch map. Before generating the target sketch map based on the second adjacency matrix and the position information of the initial sketch points, the method further includes: The elements in the first adjacency matrix corresponding to the disconnected lines are assigned a value of 1 to obtain a second adjacency matrix.

9. A sketch generating device, characterized in that: include: A detection module, used for performing edge detection on the image to be processed to obtain an edge map of the image to be processed; A first determining module is used to determine initial sketch points in the edge map, wherein the initial sketch points include sketch points in the edge map that are edge points; A second determination module is used to determine the connection probability of an initial sketch point pair based on the image features of the edge map and the position information of the initial sketch points, wherein the initial sketch point pair includes any two initial sketch points in the initial sketch points in the edge map; A generation module is used to generate a target sketch map based on the connection probability of the initial sketch point pairs and the position information of the initial sketch points.

10. A sketch generating device, the sketch extracting device comprising: A transceiver, a memory, a processor, and a program stored in the memory and executable on the processor; characterized in that: The processor is used to read the program in the memory to implement the steps in any one of the methods according to claims 1 to 8.

11. A readable storage medium, characterized in that: The readable storage medium stores a program or instruction, wherein the program or instruction, when executed by a processor, implements the steps of the method according to any one of claims 1 to 8.

Images