Sketch drawing generation method, device, equipment and readable storage medium

Abstract

The present invention provides a sketch image generation method, apparatus, device, and readable storage medium, relating to the field of image processing technology, to address the problems of large data volumes and low image encoding efficiency in existing semantic representation methods. The method comprises: performing edge detection on an image to be processed to obtain an edge map of the image to be processed; determining initial sketch points in the edge map, wherein the initial sketch points include sketch points in the edge map that are edge points; determining the connection probability of an initial sketch point pair based on image features of the edge map and position information of the initial sketch points, wherein the initial sketch point pair includes any two initial sketch points in the edge map; and generating a target sketch image based on the connection probability of the initial sketch point pair and the position information of the initial sketch points. The present invention can improve image encoding efficiency.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a sketch image generation method, device, equipment and readable storage medium. Background Art

[0002] With the development of multimedia capture devices and the increasing demand for multimedia services, efficient multimedia communication with large data volumes has attracted widespread attention from researchers. To adapt to such scenarios, the concept of multimedia computing communication has been proposed. Multimedia computing communication significantly improves transmission efficiency by introducing artificial intelligence into the communication process. Extracting the high-level semantics of multimedia data can reduce the amount of data transmitted during communication. Therefore, effective semantic representation has become a key factor affecting the performance of multimedia computing communication. However, existing semantic representation methods generally use image representation methods, such as gradient directional map representation, which extracts pixel values ​​to form a total feature vector. However, this method requires a large amount of data and has low image coding efficiency. Summary of the Invention

[0003] The embodiments of the present invention provide a method, apparatus, device and readable storage medium for generating a sketch image. In current practical applications, the amount of data represented by semantics is large, resulting in low image compression efficiency.

[0004] In order to solve the above problems, the embodiments of the present invention adopt the following technical solutions:

[0005] In a first aspect, an embodiment of the present invention provides a method for generating a sketch, comprising:

[0006] Performing edge detection on the image to be processed to obtain an edge map of the image to be processed;

[0007] Determining initial sketch points in the edge map, the initial sketch points including sketch points in the edge map that are edge points;

[0008] determining, based on image features of the edge map and position information of the initial sketch points, a connection probability of an initial sketch point pair, the initial sketch point pair comprising any two initial sketch points in the edge map;

[0009] A target sketch map is generated based on the connection probability of the initial sketch point pairs and the position information of the initial sketch points.

[0010] Optionally, determining the initial sketch point in the edge map includes:

[0011] Performing feature extraction on the edge map to obtain image features of the edge map;

[0012] Performing a dot multiplication process on the image features of the edge map and the edge map to obtain at least two first heat maps, wherein the at least two first heat maps are respectively used to represent a first category probability of each edge pixel in the edge map, where the first category probability is a probability that each edge pixel belongs to a category corresponding to the first heat map;

[0013] Based on the at least two first heat maps, initial sketch points in the edge map are determined.

[0014] Optionally, determining the initial sketch point in the edge map based on the at least two first heat maps includes:

[0015] Performing non-maximum suppression processing on the at least two first heat maps to obtain at least two second heat maps, wherein the at least two second heat maps are respectively used to represent the second category probability of each edge pixel point in the edge map;

[0016] Determining the probability of the category to which each edge pixel belongs based on the values ​​of each edge pixel in the at least two second heat maps;

[0017] Determining the category to which each edge pixel belongs according to the probability of the category to which each edge pixel belongs, the category to which each edge pixel belongs being the category to which each edge pixel belongs having the greatest probability;

[0018] Based on the category to which each edge pixel point belongs, an initial sketch point in the edge map is determined.

[0019] Optionally, determining the connection probability of the initial sketch point pairs based on the image features of the edge map and the position information of the initial sketch points includes:

[0020] Sampling the lines connecting the initial sketch point pairs based on the position information of the initial sketch point pairs to obtain a sampling point sequence of the initial sketch point pairs;

[0021] Determining features of the initial sketch point pair based on the sampling point sequence and image features of the edge map;

[0022] Based on the features of the initial sketch point pairs, a connection probability of the initial sketch point pairs is determined.

[0023] Optionally, determining the connection probability of the initial sketch point pairs based on the features of the initial sketch point pairs includes:

[0024] Using a convolutional neural network and a fully connected neural network to perform feature extraction processing on the features of the initial sketch point pairs to obtain processed features of the initial sketch point pairs;

[0025] An activation function is used to classify the processed initial sketch point pair features to obtain the connection probability of the initial sketch point pair.

[0026] Optionally, generating a target sketch map based on the connection probability and the position information of the initial sketch points includes:

[0027] Determine a first adjacency matrix corresponding to the connection probability of the initial sketch point pairs, where the first adjacency matrix is ​​used to represent the connection relationship of the initial sketch points;

[0028] A target sketch graph is generated based on the first adjacency matrix and the position information of the initial sketch points.

[0029] Optionally, generating a target sketch map based on the first adjacency matrix and the position information of the initial sketch points includes:

[0030] generating an initial sketch map based on the first adjacency matrix and the position information of the initial sketch points;

[0031] Determining abnormal lines in the initial sketch according to the initial sketch, wherein the abnormal lines include redundant lines in the initial sketch;

[0032] Assigning 0 to the elements in the first adjacency matrix corresponding to the redundant lines to obtain a second adjacency matrix;

[0033] A target sketch map is generated based on the second adjacency matrix and the position information of the initial sketch points.

[0034] Optionally, the abnormal lines further include disconnected lines in the initial sketch image, and before generating the target sketch image based on the second adjacency matrix and the position information of the initial sketch points, the method further includes:

[0035] The elements in the first adjacency matrix corresponding to the broken lines are assigned a value of 1 to obtain a second adjacency matrix.

[0036] In a second aspect, an embodiment of the present invention provides a sketch generating device, comprising:

[0037] A detection module, configured to perform edge detection on the image to be processed to obtain an edge map of the image to be processed;

[0038] A first determining module is configured to determine initial sketch points in the edge map, wherein the initial sketch points include sketch points in the edge map that are edge points;

[0039] a second determining module, configured to determine a connection probability of an initial sketch point pair based on image features of the edge map and position information of the initial sketch points, the initial sketch point pair comprising any two initial sketch points in the edge map;

[0040] A generation module is used to generate a target sketch map based on the connection probability of the initial sketch point pairs and the position information of the initial sketch points.

[0041] Optionally, the first determining module includes:

[0042] an extraction unit, configured to perform feature extraction on the edge map to obtain image features of the edge map;

[0043] a dot multiplication unit, configured to perform dot multiplication of the image features of the edge map and the edge map to obtain at least two first heat maps, wherein the at least two first heat maps are respectively used to represent a first category probability of each edge pixel in the edge map, where the first category probability is a probability that each edge pixel belongs to a category corresponding to the first heat map;

[0044] The first determining unit is configured to determine an initial sketch point in the edge map based on the at least two first heat maps.

[0045] Optionally, the first determining unit is configured to:

[0046] Performing non-maximum suppression processing on the at least two first heat maps to obtain at least two second heat maps, wherein the at least two second heat maps are respectively used to represent the second category probability of each edge pixel point in the edge map;

[0047] Determining the probability of the category to which each edge pixel belongs based on the values ​​of each edge pixel in the at least two second heat maps;

[0048] Determining the category to which each edge pixel belongs according to the probability of the category to which each edge pixel belongs, the category to which each edge pixel belongs being the category to which each edge pixel belongs having the greatest probability;

[0049] Based on the category to which each edge pixel point belongs, an initial sketch point in the edge map is determined.

[0050] Optionally, the second determining module includes:

[0051] a sampling unit, configured to sample the lines connecting the initial sketch point pairs based on the position information of the initial sketch point pairs to obtain a sampling point sequence of the initial sketch point pairs;

[0052] a second determining unit, configured to determine features of the initial sketch point pair based on the sampling point sequence and image features of the edge map;

[0053] The third determining unit is configured to determine a connection probability of the initial sketch point pair based on features of the initial sketch point pair.

[0054] Optionally, the third determining unit is configured to:

[0055] Using a convolutional neural network and a fully connected neural network to perform feature extraction processing on the features of the initial sketch point pairs to obtain processed features of the initial sketch point pairs;

[0056] An activation function is used to classify the processed initial sketch point pair features to obtain the connection probability of the initial sketch point pair.

[0057] Optionally, the generating module includes:

[0058] a fourth determining unit, configured to determine a first adjacency matrix corresponding to the connection probabilities of the initial sketch point pairs, wherein the first adjacency matrix is ​​used to represent the connection relationships of the initial sketch points;

[0059] A generating unit is used to generate a target sketch map based on the first adjacency matrix and the position information of the initial sketch points.

[0060] Optionally, the generating unit is configured to:

[0061] generating an initial sketch map based on the first adjacency matrix and the position information of the initial sketch points;

[0062] Determining abnormal lines in the initial sketch according to the initial sketch, wherein the abnormal lines include redundant lines in the initial sketch;

[0063] Assigning 0 to the elements in the first adjacency matrix corresponding to the redundant lines to obtain a second adjacency matrix;

[0064] A target sketch map is generated based on the second adjacency matrix and the position information of the initial sketch points.

[0065] Optionally, the abnormal lines further include disconnected lines in the initial sketch image, and the generating unit is further configured to:

[0066] and assigning 1 to the elements in the first adjacency matrix corresponding to the disconnected lines to obtain a second adjacency matrix.

[0067] In a third aspect, an embodiment of the present invention provides a sketch generating device, which includes: a transceiver, a memory, a processor, and a program stored in the memory and executable on the processor; the processor is configured to read the program in the memory to implement the steps of the method described in the first aspect.

[0068] In a fourth aspect, an embodiment of the present invention provides a readable storage medium, on which a program or instruction is stored, wherein the program or instruction, when executed by a processor, implements the steps in the method described in the first aspect.

[0069] In an embodiment of the present invention, the sketch image generation method can obtain an edge map of the image to be processed by performing edge detection on the image to be processed, and select edge pixel points in the edge map to obtain initial sketch points, and then determine the connection probability of the initial sketch point pairs according to the image features of the edge map, thereby generating a target sketch image. The target sketch image can be represented as a set of sketch points and sketch lines. The set of sketch points and sketch lines can describe the pixel changes, geometric feature distribution and structural information of the image, so that the target sketch image better retains the semantic features of the image, greatly reduces the amount of data, and has higher accuracy and stronger representation ability. When used for image encoding, the encoding efficiency is higher. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in describing the embodiments of the present invention. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0071] Figure 1 This is one of the flow charts of a method for generating a sketch provided by an embodiment of the present invention;

[0072] Figure 2 This is the second flowchart of a method for generating a sketch provided by an embodiment of the present invention;

[0073] Figure 3 This is a schematic structural diagram of a sketch generating device provided by an embodiment of the present invention;

[0074] Figure 4 It is a structural schematic diagram of a sketch image generating device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0075] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0076] Unless otherwise defined, technical or scientific terms used in this disclosure shall have the ordinary meanings understood by persons of ordinary skill in the art to which this disclosure pertains. The terms "first," "second," and similar expressions used in this disclosure do not denote any order, quantity, or importance; they are used solely to distinguish between components. Terms such as "upper," "lower," "left," and "right" are used solely to indicate relative positions; when the absolute position of the object being described changes, the relative position changes accordingly.

[0077] See Figure 1 , Figure 1 This is one of the flowcharts of a sketch generation method provided by an embodiment of the present invention, the sketch generation method comprising:

[0078] Step 101: perform edge detection on an image to be processed to obtain an edge map of the image to be processed.

[0079] Specifically, the image to be processed can be any image, such as a photo, a video frame, etc.; the edge detection can be performed by a feature extraction module of an hourglass structure, wherein the number of upsampling and downsampling in the hourglass structure is the same, and the feature extraction module of the hourglass structure includes multiple convolution layers of different scales, the last convolution layer has a single convolution kernel, and the convolution layer is followed by a sigmoid activation function to obtain the probability that each pixel of the image belongs to the edge; after extracting the edge probability map, in order to facilitate subsequent processing, a single-pixel edge can be obtained by non-maximum suppression. Non-maximum suppression searches for the edge intensity value of the pixel point in the gradient direction to determine whether the pixel point has obtained a local maximum, thereby suppressing the non-maximum point and obtaining a single-pixel edge intensity map, and then according to the probability that each pixel point of the image belongs to the edge, the edge map of the image can be obtained. For example, a pixel point whose probability of belonging to the edge is greater than a threshold can be determined as an edge pixel point in the edge map.

[0080] Step 102: Determine initial sketch points in the edge map, where the initial sketch points include sketch points in the edge map that are edge points.

[0081] Specifically, the initial sketch points include sketch points in the edge map that are edge points, and may include intersection points, inflection points and endpoints in the sketch points, wherein an intersection point indicates that the point connects more than or equal to three sketch points, an inflection point indicates that the point connects two sketch points, and an endpoint indicates that the point connects one sketch point; determining the initial sketch points in the edge map may be to first determine the category of each edge pixel point in the edge map, and after determining the category of each edge pixel point, the initial sketch points may be determined, wherein the categories of edge pixel points may include intersection points, inflection points, endpoints, non-sketch points and non-edge points, and edge pixel points whose categories are not non-sketch points may be determined as initial sketch points.

[0082] Step 103: determining a connection probability of an initial sketch point pair based on the image features of the edge map and the position information of the initial sketch points, wherein the initial sketch point pair includes any two initial sketch points in the edge map.

[0083] Specifically, the image features of the edge map can be the edge features of the edge map extracted by a feature extraction module of an hourglass structure, or a feature extraction module of other structures can be used. The position information of the initial sketch point can be the two-dimensional coordinates of the initial sketch point, and the connection probability of the initial sketch point pair can be the probability that the two initial sketch points in the initial sketch point pair are connected.

[0084] Step 104: Generate a target sketch map based on the connection probability of the initial sketch point pairs and the position information of the initial sketch points.

[0085] Specifically, the target sketch map can be an image generated by visualizing the sketch map based on the connection probability of the initial sketch point pairs and the position information of the initial sketch points, or it can be represented as the coordinates of the initial sketch points and the adjacency matrix corresponding to the initial sketch points. The adjacency matrix can be obtained based on the connection probability of the initial sketch point pairs; the target sketch map can be directly used for compression encoding and decoding, and the sketch points in the target sketch map can be rearranged so that the sketch points on the same edge line are connected in sequence. According to the rearrangement rule of each sketch point, the optimized adjacency matrix is ​​adjusted and XORed with the ideal adjacency matrix to obtain an output matrix. The ideal adjacency matrix can be an adjacency matrix in which all elements on the secondary diagonal are 1 and all other elements are 0. The rearranged coordinates of the optimized sketch points are quantized and encoded to obtain a sketch point code stream. The positions of the elements that are 1 in the output matrix are encoded to obtain an adjacency matrix code stream. According to the sketch point code stream and the adjacency matrix code stream, a sketch map code stream is obtained.

[0086] For example, specific compression coding and decoding methods may include adaptive sketch compression, sketch rearrangement, and sketch encoding. Adaptive sketch compression may be performed by first calculating the distance from each inflection point to the straight line formed by the sketch point pair connected to it when compressing the sketch. Sketch points are removed in order of distance from low to high. After removing a sketch point, the rows and columns of its corresponding adjacency matrix are also removed, and the rows and columns of its two adjacent sketch points in the adjacency matrix are updated, that is, their corresponding positions are set to 1. After removing a sketch point, the corresponding distance of the two connected sketch points needs to be updated. The above process is repeated until the amount of sketch data obtained is lower than the set value, that is, the adaptive sketch compression process is completed. Adaptive sketch compression is a method of streamlining the sketch after obtaining the target sketch, thereby reducing the number of sketch points and the dimension of the adjacency matrix, thereby achieving a sketch with less data without affecting its representation performance. The algorithm performs adaptive compression according to the set data limit to meet the set data requirement.

[0087] Sketch rearrangement can be to rearrange the pixels in the sketch after generating the target sketch and compressing the sketch so that the pixels on an edge line are arranged in order, and also to rearrange the adjacency matrix accordingly.

[0088] Sketch image encoding can be performed by performing an exclusive OR (XOR) operation on the rearranged adjacency matrix and the ideal adjacency matrix to obtain an output matrix, wherein the elements on the sub-diagonal of the ideal adjacency matrix are all 1 and the remaining elements are all 0, encoding the positions of the elements with a value of 1 in the output matrix to obtain an adjacency matrix code stream, and encoding the rearranged sketch point coordinates into a sketch point code stream, and the two together constitute the code stream after the sketch image is encoded. Since the sketch points in the sketch image are usually connected to form long and coherent lines, most of the 1s in the rearranged adjacency matrix are concentrated on the sub-diagonal, while there are fewer sub-diagonal elements with non-0 values ​​and elements in other positions with a value of 1. Therefore, in the result of the exclusive OR operation, encoding the position of 1 can greatly reduce the amount of data.

[0089] In an embodiment of the present invention, the sketch image generation method can obtain an edge map of the image to be processed by performing edge detection on the image to be processed, and select edge pixel points in the edge map to obtain initial sketch points, and then determine the connection probability of the initial sketch point pairs according to the image features of the edge map, thereby generating a target sketch image. The target sketch image can be represented as a set of sketch points and sketch lines. The set of sketch points and sketch lines can describe the pixel changes, geometric feature distribution and structural information of the image, so that the target sketch image better retains the semantic features of the image, greatly reduces the amount of data, and has higher accuracy and stronger representation ability. When used for image encoding, the encoding efficiency is higher.

[0090] Optionally, determining the initial sketch point in the edge map includes:

[0091] Performing feature extraction on the edge map to obtain image features of the edge map;

[0092] Performing a dot multiplication process on the image features of the edge map and the edge map to obtain at least two first heat maps, wherein the at least two first heat maps are respectively used to represent a first category probability of each edge pixel in the edge map, where the first category probability is a probability that each edge pixel belongs to a category corresponding to the first heat map;

[0093] Based on the at least two first heat maps, initial sketch points in the edge map are determined.

[0094] Specifically, the feature extraction of the edge map can be performed by using a feature extraction module with an hourglass structure, the image feature of the edge map can be the edge feature of the edge map, the coordinates of each pixel point contained in the first heat map correspond one-to-one with the coordinates of each edge pixel point contained in the edge map, and the first category probability can be the value of each pixel point on the first heat map, and the value of each pixel point on the first heat map represents the probability that the edge pixel point corresponding to the pixel point belongs to the category corresponding to the heat map.

[0095] In an embodiment of the present invention, the sketch image generation method can generate at least two first heat maps, and the at least two first heat maps are respectively used to characterize the first category probability of each edge pixel point in the edge image, determine the category of each edge pixel point in the edge image, and then determine the initial sketch point, so that the selection of the initial sketch point is more accurate, thereby improving the accuracy of the generated target sketch image.

[0096] Optionally, determining the initial sketch point in the edge map based on the at least two first heat maps includes:

[0097] Performing non-maximum suppression processing on the at least two first heat maps to obtain at least two second heat maps, wherein the at least two second heat maps are respectively used to represent the second category probability of each edge pixel point in the edge map;

[0098] Determining the probability of the category to which each edge pixel belongs based on the values ​​of each edge pixel in the at least two second heat maps;

[0099] Determining the category to which each edge pixel belongs according to the probability of the category to which each edge pixel belongs, the category to which each edge pixel belongs being the category to which each edge pixel belongs having the greatest probability;

[0100] Based on the category to which each edge pixel point belongs, an initial sketch point in the edge map is determined.

[0101] Specifically, the non-maximum suppression processing can be non-maximum suppression of pixel points in the at least two first heat maps, wherein all points suppressed by the non-maximum suppression operation are treated as non-sketch points and do not participate in the generation of subsequent sketch maps; the second category probability can be the probability of the category to which the pixel points in the at least two second heat maps belong; the category to which each edge pixel point belongs can include intersection points, inflection points, endpoints, non-sketch points and non-edge points, and the edge pixel points whose category is not non-sketch points are determined as initial sketch points, and the two-dimensional coordinates of each initial sketch point can be obtained, and the initial sketch points can include intersection points, inflection points and endpoints.

[0102] In an embodiment of the present invention, the sketch image generation method can perform non-maximum suppression on the pixel points in the at least two first heat maps so that the individual pixel points in the at least two second heat maps obtained constitute a single-pixel heat map, thereby making the subsequently obtained target sketch image more accurate.

[0103] Optionally, determining the connection probability of the initial sketch point pairs based on the image features of the edge map and the position information of the initial sketch points includes:

[0104] Sampling the lines connecting the initial sketch point pairs based on the position information of the initial sketch point pairs to obtain a sampling point sequence of the initial sketch point pairs;

[0105] Determining features of the initial sketch point pair based on the sampling point sequence and image features of the edge map;

[0106] Based on the features of the initial sketch point pairs, a connection probability of the initial sketch point pairs is determined.

[0107] Specifically, the sampling of the line connecting the initial sketch point pairs may be performed by uniformly sampling a certain number of times on the line segment connecting the two initial sketch points of the initial sketch point pairs to obtain a sampling point sequence of the initial sketch point pairs, wherein the sampling point sequence includes the initial sketch point pairs and points sampled at equal intervals on the line connecting the initial sketch point pairs, wherein the number of sampling points of each initial sketch point pair is the same; the image features of the edge map may be the edge features of the edge map, and the features of the initial sketch point pairs may be determined by sampling at each sampling point position in the sampling point sequence of each initial sketch point pair based on the edge features to obtain the features of the initial sketch point pairs.

[0108] To determine the connection probability, the dimension of the sampled features of the initial sketch point pair is proportional to the square of the number of initial sketch points. Since spatially adjacent initial sketch points are more likely to be connected, a divide-and-conquer approach can be adopted to reduce computational overhead. Simple spatial grouping is performed on all initial sketch points, and the connection relationships of each group of initial sketch points are tested. To compensate for the connection loss of initial sketch point pairs due to two initial sketch points being divided into different groups, all unsaturated initial sketch points can be collected again for connection detection to obtain a rough sketch map. Unsaturated initial sketch points are those for which the number of detected connections is less than their ideal degree. The ideal degree of intersection points is 4, the ideal degree of inflection points is 2, and the ideal degree of endpoints is 1.

[0109] The connection relationship detection based on the divide-and-conquer method is implemented by setting the detection quantity threshold. Each time, the current initial sketch points are divided into two groups of equal number according to the proximity criterion of spatial coordinates. If the number of initial sketch points in one group after grouping is lower than the detection quantity threshold, the connection relationship detection is performed directly; if it is higher than the detection quantity threshold, the grouping is performed again until the number of initial sketch points in all groups is lower than the detection quantity threshold.

[0110] After each set of initial sketch points is detected, the connection relationship can be corrected to make the target sketch more robust and accurate. For example, when three initial sketch points are connected to each other and almost arranged in a line, the connection of the pair with the longest distance will be cancelled; if there is an initial sketch point with a current degree greater than the theoretical value, the weakest connection can be cancelled based on the position of the connected point; if there is a weak connection between two unsaturated initial sketch points, its connection will be strengthened to compensate for the possibility of lost connection.

[0111] In an embodiment of the present invention, the sketch image generation method can obtain the connection probability of the initial sketch point pairs by performing a certain number of uniform sampling on the line segments connected by the initial sketch point pairs, so that the accuracy of the connection probability is higher, and then determine the connection relationship of the initial sketch point pairs, thereby improving the accuracy of target sketch image extraction.

[0112] Optionally, determining the connection probability of the initial sketch point pairs based on the features of the initial sketch point pairs includes:

[0113] Using a convolutional neural network and a fully connected neural network to perform feature extraction processing on the features of the initial sketch point pairs to obtain processed features of the initial sketch point pairs;

[0114] An activation function is used to classify the processed initial sketch point pair features to obtain the connection probability of the initial sketch point pair.

[0115] Specifically, the processed initial sketch point pair features may be depth features that can reflect whether the initial sketch point pairs are connected, and the activation function may be a sigmoid activation function.

[0116] In an embodiment of the present invention, the sketch image generation method can extract deep features reflecting whether the initial sketch point pairs are connected by using a convolutional neural network and a fully connected neural network, and use an activation function to obtain the connection probability of the initial sketch point pairs, thereby improving the accuracy of determining the connection probability of the initial sketch point pairs.

[0117] Optionally, generating a target sketch map based on the connection probability and the position information of the initial sketch points includes:

[0118] Determine a first adjacency matrix corresponding to the connection probability of the initial sketch point pairs, where the first adjacency matrix is ​​used to represent the connection relationship of the initial sketch points;

[0119] A target sketch graph is generated based on the first adjacency matrix and the position information of the initial sketch points.

[0120] Specifically, based on the connection probability of the initial sketch point pairs, it can be determined whether there is a connection relationship between each initial sketch point pair, and then the first adjacency matrix can be obtained. The first adjacency matrix can be obtained by adjusting the initial adjacency matrix accordingly according to the connection probability of the initial sketch point pairs.

[0121] For example, the initial adjacency matrix can be an adjacency matrix in which all elements except the main diagonal are 0, indicating that there may be connections between all initial sketch points. When the probability of the initial sketch point pairs being connected is greater than the probability threshold, it is determined that the initial sketch point pairs can be connected. Then, when the initial sketch point pairs need to be connected, the initial sketch point pairs are connected, that is, the corresponding positions in the initial adjacency matrix are assigned a value of 1; when the initial sketch point pairs are not connected, the connection between the initial sketch point pairs is disconnected, that is, the corresponding positions in the initial adjacency matrix are assigned a value of 0. According to whether the initial sketch point pairs are connected, the first adjacency matrix can be obtained.

[0122] The position information of the initial sketch point may be the two-dimensional coordinates of the initial sketch point, and the target sketch map may be the initial sketch map obtained by visualizing the sketch map based on the first adjacency matrix and the coordinates of the initial sketch point. Alternatively, the target sketch map may be directly represented as a dictionary consisting of N×2 point positions (the two-dimensional coordinates of the initial sketch point) and an N×N first adjacency matrix, where N is the number of the initial sketch points.

[0123] In an embodiment of the present invention, the sketch image generation method can determine a first adjacency matrix through the connection probability of the initial sketch point pairs, and generate a target sketch image based on the first adjacency matrix and the position information of the initial sketch points, so that the target sketch image can be represented as a set of sketch points and sketch lines, and the set of sketch points and sketch lines can describe the pixel changes, geometric feature distribution and structural information of the image, so that the sketch image retains the semantic features of the image.

[0124] Optionally, generating a target sketch map based on the first adjacency matrix and the position information of the initial sketch points includes:

[0125] generating an initial sketch map based on the first adjacency matrix and the position information of the initial sketch points;

[0126] Determining abnormal lines in the initial sketch according to the initial sketch, wherein the abnormal lines include redundant lines in the initial sketch;

[0127] Assigning 0 to the elements in the first adjacency matrix corresponding to the redundant lines to obtain a second adjacency matrix;

[0128] A target sketch map is generated based on the second adjacency matrix and the position information of the initial sketch points.

[0129] Specifically, the redundant lines may include incorrectly drawn connecting lines, repeatedly drawn connecting lines and non-maximum connecting lines. Incorrectly drawn connecting lines represent connections that lead to unexpected lines, which are completely inconsistent with the original single-pixel edge map; repeatedly drawn connecting lines represent connections that have been covered by other lines, which are redundant connection relationships and have no effect on the accuracy of the extracted sketch image after removal; non-maximum connecting lines are connecting lines whose intensity is not the largest among the thicker adjacent parallel lines in the restored sketch image.

[0130] The elements in the first adjacency matrix are 1, indicating that the corresponding initial sketch points are connected, and the elements in the first adjacency matrix are 0, indicating that the corresponding initial sketch points are not connected. The redundant lines in the initial sketch image can be identified, and the elements in the first adjacency matrix corresponding to the redundant lines are assigned 0, which can achieve the elimination of the redundant lines and obtain the second adjacency matrix. The target sketch image generated based on the second adjacency matrix removes the redundant lines.

[0131] In an embodiment of the present invention, the sketch image generation method can generate an initial sketch image based on the first adjacency matrix, identify redundant lines in the initial sketch image, and obtain a target sketch image with the redundant lines removed by reassigning elements in the first adjacency matrix corresponding to the redundant lines, thereby making the target sketch image more accurate, reducing the amount of image data, and improving image encoding efficiency.

[0132] Optionally, the abnormal lines further include disconnected lines in the initial sketch image, and before generating the target sketch image based on the second adjacency matrix and the position information of the initial sketch points, the method further includes:

[0133] The elements in the first adjacency matrix corresponding to the broken lines are assigned a value of 1 to obtain a second adjacency matrix.

[0134] Specifically, the broken lines may be accidental disconnections of lines, which are completely inconsistent with the original single-pixel edge map. The broken lines in the initial sketch map can be identified, and the corresponding elements in the first adjacency matrix corresponding to the connected broken lines can be assigned 1 accordingly to obtain a second adjacency matrix.

[0135] In an embodiment of the present invention, the sketch image generation method can generate an initial sketch image based on the first adjacency matrix, identify broken lines in the initial sketch image, and obtain a target sketch image connecting the broken lines by reassigning elements in the first adjacency matrix corresponding to the broken lines, thereby making the target sketch image more accurate.

[0136] Figure 2 This is a second flow chart of a method for generating a sketch provided by an embodiment of the present invention. Figure 2As shown, first input the image to be processed, use the hourglass module to perform edge detection on the image to be processed, and obtain the edge map of the image to be processed; then use the hourglass module to extract features from the edge map to obtain image features of the edge map, perform point multiplication processing on the image features of the edge map and the edge map to obtain at least two first heat maps; perform non-maximum suppression on the at least two first heat maps to obtain at least two second heat maps, detect the maximum value position in the at least two second heat maps, determine the category to which each edge pixel point belongs, and thus determine the initial sketch point, that is, according to the position of each edge pixel point in the at least two second heat maps, the initial sketch point is determined. The value of the heat map is taken to determine the probability of the category to which each edge pixel point belongs, and the category to which each edge pixel point belongs is the category to which each edge pixel point belongs with the greatest probability; the initial sketch points are paired to obtain initial sketch point pairs, and the initial sketch point pairs are interpolated to obtain a sampling point sequence of the initial sketch point pairs; sampling is performed based on the sampling point sequence and the edge features of the edge map to determine the features of the initial sketch point pairs; a convolutional neural network and an activation function are used to perform feature extraction and shape processing on the features of the initial sketch point pairs to obtain an adjacency matrix, and a target sketch map can be generated according to the adjacency matrix; wherein, Figure 2 As shown, the hourglass module includes an upsampling module and a downsampling module, and the number of upsampling is the same as the number of downsampling.

[0137] See also Figure 3 , Figure 3 Schematic diagram of a sketch generating device provided by an embodiment of the present invention. Figure 3 As shown, the sketch generating device 300 includes:

[0138] A detection module 301 is configured to perform edge detection on an image to be processed to obtain an edge map of the image to be processed;

[0139] A first determining module 302 is configured to determine initial sketch points in the edge map, wherein the initial sketch points include sketch points in the edge map that are edge points;

[0140] A second determining module 303 is configured to determine a connection probability of an initial sketch point pair based on the image features of the edge map and the position information of the initial sketch points, where the initial sketch point pair includes any two initial sketch points in the edge map;

[0141] The generation module 304 is configured to generate a target sketch map based on the connection probability of the initial sketch point pairs and the position information of the initial sketch points.

[0142] Specifically, the sketch generating device may be configured with an image processing model, and the training method of the image processing model may be: pre-acquiring a plurality of image samples and sketch sample images of the image samples, wherein the image samples may be images in the BSDS500 dataset, and the sketch sample images of the image samples may be generated using a sketch generating method of related technology;

[0143] The image sample is input into the initial model, and the initial model can output a predicted sketch of the image sample. The model architecture of the initial model and the image processing model is the same, except that the model parameters in the initial model have not been trained yet;

[0144] According to the difference between the predicted sketch and the sketch sample, a cross entropy loss function can be established. The initial model is trained according to the cross entropy loss function. During the training process, the model parameters of the initial model are continuously updated to finally obtain the image processing model.

[0145] Among them, the cross entropy loss function can include binary cross entropy loss function, multivariate cross entropy loss function and weighted cross entropy loss function. The binary cross entropy loss function can be used for edge detection and connection relationship detection, the multivariate cross entropy loss function can be used for sketch point detection, and the weighted cross entropy loss function can solve the problem of sample imbalance in edge detection and sketch point detection.

[0146] Optionally, the first determining module 302 includes:

[0147] an extraction unit, configured to perform feature extraction on the edge map to obtain image features of the edge map;

[0148] a dot multiplication unit, configured to perform dot multiplication of the image features of the edge map and the edge map to obtain at least two first heat maps, wherein the at least two first heat maps are respectively used to represent a first category probability of each edge pixel in the edge map, where the first category probability is a probability that each edge pixel belongs to a category corresponding to the first heat map;

[0149] The first determining unit is configured to determine an initial sketch point in the edge map based on the at least two first heat maps.

[0150] Optionally, the first determining unit is configured to:

[0151] Performing non-maximum suppression processing on the at least two first heat maps to obtain at least two second heat maps, wherein the at least two second heat maps are respectively used to represent the second category probability of each edge pixel point in the edge map;

[0152] Determining the probability of the category to which each edge pixel belongs based on the values ​​of each edge pixel in the at least two second heat maps;

[0153] Determining the category to which each edge pixel belongs according to the probability of the category to which each edge pixel belongs, the category to which each edge pixel belongs being the category to which each edge pixel belongs having the greatest probability;

[0154] Based on the category to which each edge pixel point belongs, an initial sketch point in the edge map is determined.

[0155] Optionally, the second determining module 303 includes:

[0156] a sampling unit, configured to sample the lines connecting the initial sketch point pairs based on the position information of the initial sketch point pairs to obtain a sampling point sequence of the initial sketch point pairs;

[0157] a second determining unit, configured to determine features of the initial sketch point pair based on the sampling point sequence and image features of the edge map;

[0158] The third determining unit is configured to determine a connection probability of the initial sketch point pair based on features of the initial sketch point pair.

[0159] Optionally, the third determining unit is configured to:

[0160] Using a convolutional neural network and a fully connected neural network to perform feature extraction processing on the features of the initial sketch point pairs to obtain processed features of the initial sketch point pairs;

[0161] An activation function is used to classify the processed initial sketch point pair features to obtain the connection probability of the initial sketch point pair.

[0162] Optionally, the generating module 304 includes:

[0163] a fourth determining unit, configured to determine a first adjacency matrix corresponding to the connection probabilities of the initial sketch point pairs, wherein the first adjacency matrix is ​​used to represent the connection relationships of the initial sketch points;

[0164] A generating unit is used to generate a target sketch map based on the first adjacency matrix and the position information of the initial sketch points.

[0165] Optionally, the generating unit is configured to:

[0166] generating an initial sketch map based on the first adjacency matrix and the position information of the initial sketch points;

[0167] Determining abnormal lines in the initial sketch according to the initial sketch, wherein the abnormal lines include redundant lines in the initial sketch;

[0168] Assigning 0 to the elements in the first adjacency matrix corresponding to the redundant lines to obtain a second adjacency matrix;

[0169] A target sketch map is generated based on the second adjacency matrix and the position information of the initial sketch points.

[0170] Optionally, the abnormal lines further include disconnected lines in the initial sketch image, and the generating unit is further configured to:

[0171] and assigning 1 to the elements in the first adjacency matrix corresponding to the disconnected lines to obtain a second adjacency matrix.

[0172] See also Figure 4 , Figure 4 This is a schematic diagram of the structure of a sketch generating device provided by an embodiment of the present invention. Figure 4 As shown, the sketch image generating device includes a bus 401, a transceiver 402, an antenna 403, a bus interface 404, a processor 405 and a memory 406:

[0173] The processor 405 is configured to: perform edge detection on the image to be processed to obtain an edge map of the image to be processed;

[0174] Determining initial sketch points in the edge map, the initial sketch points including sketch points in the edge map that are edge points;

[0175] determining, based on image features of the edge map and position information of the initial sketch points, a connection probability of an initial sketch point pair, the initial sketch point pair comprising any two initial sketch points in the edge map;

[0176] A target sketch map is generated based on the connection probability of the initial sketch point pairs and the position information of the initial sketch points.

[0177] Optionally, the processor 405 is specifically configured to:

[0178] Performing feature extraction on the edge map to obtain image features of the edge map;

[0179] Performing a dot multiplication process on the image features of the edge map and the edge map to obtain at least two first heat maps, wherein the at least two first heat maps are respectively used to represent a first category probability of each edge pixel in the edge map, where the first category probability is a probability that each edge pixel belongs to a category corresponding to the first heat map;

[0180] Based on the at least two first heat maps, initial sketch points in the edge map are determined.

[0181] Optionally, the processor 405 is specifically configured to:

[0182] Performing non-maximum suppression processing on the at least two first heat maps to obtain at least two second heat maps, wherein the at least two second heat maps are respectively used to represent the second category probability of each edge pixel point in the edge map;

[0183] Determining the probability of the category to which each edge pixel belongs based on the values ​​of each edge pixel in the at least two second heat maps;

[0184] Determining the category to which each edge pixel belongs according to the probability of the category to which each edge pixel belongs, the category to which each edge pixel belongs being the category to which each edge pixel belongs having the greatest probability;

[0185] Based on the category to which each edge pixel point belongs, an initial sketch point in the edge map is determined.

[0186] Optionally, the processor 405 is specifically configured to:

[0187] Sampling the lines connecting the initial sketch point pairs based on the position information of the initial sketch point pairs to obtain a sampling point sequence of the initial sketch point pairs;

[0188] Determining features of the initial sketch point pair based on the sampling point sequence and image features of the edge map;

[0189] Based on the features of the initial sketch point pairs, a connection probability of the initial sketch point pairs is determined.

[0190] Optionally, the processor 405 is specifically configured to:

[0191] Using a convolutional neural network and a fully connected neural network to perform feature extraction processing on the features of the initial sketch point pairs to obtain processed features of the initial sketch point pairs;

[0192] An activation function is used to classify the processed initial sketch point pair features to obtain the connection probability of the initial sketch point pair.

[0193] Optionally, the processor 405 is specifically configured to:

[0194] Determine a first adjacency matrix corresponding to the connection probability of the initial sketch point pairs, where the first adjacency matrix is ​​used to represent the connection relationship of the initial sketch points;

[0195] A target sketch graph is generated based on the first adjacency matrix and the position information of the initial sketch points.

[0196] Optionally, the processor 405 is specifically configured to:

[0197] generating an initial sketch map based on the first adjacency matrix and the position information of the initial sketch points;

[0198] Determining abnormal lines in the initial sketch according to the initial sketch, wherein the abnormal lines include redundant lines in the initial sketch;

[0199] Assigning 0 to the elements in the first adjacency matrix corresponding to the redundant lines to obtain a second adjacency matrix;

[0200] A target sketch map is generated based on the second adjacency matrix and the position information of the initial sketch points.

[0201] Optionally, the processor 405 is further configured to:

[0202] The elements in the first adjacency matrix corresponding to the broken lines are assigned a value of 1 to obtain a second adjacency matrix.

[0203] exist Figure 4 In the embodiment, a bus architecture (represented by bus 401) is shown. Bus 401 may include any number of interconnected buses and bridges. Bus 401 links various circuits, including one or more processors represented by processor 405 and memory represented by memory 406. Bus 401 may also link various other circuits, such as peripheral devices, voltage regulators, and power management circuits. These are well known in the art and are therefore not described further herein. Bus interface 404 provides an interface between bus 401 and transceiver 402. Transceiver 402 may be a single component or multiple components, such as multiple receivers and transmitters, providing a means for communicating with various other devices over a transmission medium. Data processed by processor 405 is transmitted over a wireless medium via antenna 403. Antenna 403 also receives data and transmits it to processor 405.

[0204] Processor 405 is responsible for managing bus 401 and general processing, and may also provide various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. Memory 406 may be used to store data used by processor 405 when performing operations.

[0205] Optionally, the processor 405 may be a CPU, an ASIC, an FPGA, or a CPLD.

[0206] An embodiment of the present invention further provides a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the various processes of the auxiliary device status control method embodiment described above and achieves the same technical effects. To avoid repetition, the details are not described here. The computer-readable storage medium may be, for example, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0207] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or apparatus comprising the element.

[0208] Through the above description of the embodiments, those skilled in the art will clearly understand that the methods of the above embodiments can be implemented using software plus the necessary general-purpose hardware platform. Of course, hardware can also be used, but in many cases the former is the more preferred implementation method. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, or optical disk) and includes a number of instructions for enabling a terminal (such as a mobile phone, computer, server, air conditioner, or network device) to execute the methods described in the various embodiments of the present invention.

[0209] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the present invention and the claims, all of which are protected by the present invention.

Claims

1. A sketch generating method, characterized in that: include: Performing edge detection on the image to be processed to obtain an edge map of the image to be processed; Determining initial sketch points in the edge map, the initial sketch points including sketch points in the edge map that are edge points; determining, based on image features of the edge map and position information of the initial sketch points, a connection probability of an initial sketch point pair, the initial sketch point pair comprising any two initial sketch points in the edge map; A target sketch map is generated based on the connection probability of the initial sketch point pairs and the position information of the initial sketch points.

2. The method according to claim 1, characterized in that The determining of the initial sketch point in the edge map includes: Performing feature extraction on the edge map to obtain image features of the edge map; Performing a dot multiplication process on the image features of the edge map and the edge map to obtain at least two first heat maps, wherein the at least two first heat maps are respectively used to represent a first category probability of each edge pixel in the edge map, where the first category probability is a probability that each edge pixel belongs to a category corresponding to the first heat map; Based on the at least two first heat maps, initial sketch points in the edge map are determined.

3. The method according to claim 2, characterized in that The determining of the initial sketch point in the edge map based on the at least two first heat maps includes: Performing non-maximum suppression processing on the at least two first heat maps to obtain at least two second heat maps, wherein the at least two second heat maps are respectively used to represent the second category probability of each edge pixel point in the edge map; Determining the probability of the category to which each edge pixel belongs based on the values ​​of each edge pixel in the at least two second heat maps; Determining the category to which each edge pixel belongs according to the probability of the category to which each edge pixel belongs, the category to which each edge pixel belongs being the category to which each edge pixel belongs having the greatest probability; Based on the category to which each edge pixel point belongs, an initial sketch point in the edge map is determined.

4. The method according to claim 1, wherein The determining, based on the image features of the edge map and the position information of the initial sketch points, the connection probability of the initial sketch point pairs includes: Sampling the lines connecting the initial sketch point pairs based on the position information of the initial sketch point pairs to obtain a sampling point sequence of the initial sketch point pairs; Determining features of the initial sketch point pair based on the sampling point sequence and image features of the edge map; Based on the features of the initial sketch point pairs, a connection probability of the initial sketch point pairs is determined.

5. The method according to claim 4, characterized in that The determining, based on the features of the initial sketch point pairs, the connection probability of the initial sketch point pairs includes: Using a convolutional neural network and a fully connected neural network to perform feature extraction processing on the features of the initial sketch point pairs to obtain processed features of the initial sketch point pairs; An activation function is used to classify the processed initial sketch point pair features to obtain the connection probability of the initial sketch point pair.

6. The method according to claim 1, characterized in that The generating of a target sketch map based on the connection probability and the position information of the initial sketch point includes: Determine a first adjacency matrix corresponding to the connection probability of the initial sketch point pairs, where the first adjacency matrix is ​​used to represent the connection relationship of the initial sketch points; A target sketch graph is generated based on the first adjacency matrix and the position information of the initial sketch points.

7. The method according to claim 6, characterized in that The step of generating a target sketch map based on the first adjacency matrix and the position information of the initial sketch points includes: generating an initial sketch map based on the first adjacency matrix and the position information of the initial sketch points; Determining abnormal lines in the initial sketch according to the initial sketch, wherein the abnormal lines include redundant lines in the initial sketch; Assigning 0 to the elements in the first adjacency matrix corresponding to the redundant lines to obtain a second adjacency matrix; A target sketch map is generated based on the second adjacency matrix and the position information of the initial sketch points.

8. The method according to claim 7, characterized in that The abnormal lines also include disconnected lines in the initial sketch image. Before generating the target sketch image based on the second adjacency matrix and the position information of the initial sketch points, the method further includes: The elements in the first adjacency matrix corresponding to the broken lines are assigned a value of 1 to obtain a second adjacency matrix.

9. A sketch generating device, characterized in that: include: A detection module, configured to perform edge detection on the image to be processed to obtain an edge map of the image to be processed; A first determining module is configured to determine initial sketch points in the edge map, wherein the initial sketch points include sketch points in the edge map that are edge points; a second determining module, configured to determine a connection probability of an initial sketch point pair based on image features of the edge map and position information of the initial sketch points, the initial sketch point pair comprising any two initial sketch points in the edge map; A generation module is used to generate a target sketch map based on the connection probability of the initial sketch point pairs and the position information of the initial sketch points.

10. A sketch generating device, comprising: A transceiver, a memory, a processor, and a program stored in the memory and executable on the processor; characterized in that: The processor is configured to read a program in a memory to implement the steps of the method according to any one of claims 1 to 8.

11. A readable storage medium, characterized in that: The readable storage medium stores a program or instruction, wherein the program or instruction, when executed by a processor, implements the steps of the method according to any one of claims 1 to 8.

Images