Stroke-based scene sketch semantic segmentation method and device

Abstract

The application relates to a stroke-based scene sketch semantic segmentation method and device. The method comprises the following steps: extracting appearance features of strokes in a scene sketch; using the appearance features of the strokes as input, performing time sequence feature coding on the strokes through a recurrent neural network to obtain time sequence features; constructing a graph structure between the strokes through the position relationship of adjacent strokes, taking the time sequence features of each stroke as nodes of the graph, simultaneously calculating the adjacent relationship between the strokes to obtain an adjacency matrix, performing spatial feature coding on the strokes through a graph convolutional neural network to obtain spatial features; fusing the time sequence features and the spatial features, performing category prediction on each stroke according to the fused features, and completing stroke-based scene sketch semantic segmentation. Through fusion of appearance, time sequence and spatial features at three different levels, each stroke in the scene sketch can be classified and recognized, and stroke feature-based scene sketch semantic segmentation is realized.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of computer vision, and particularly relates to a stroke-based scene sketch semantic segmentation method and device. BACKGROUND

[0002] Sketches have been used as a major communication tool since ancient times, which is conducive to intuitive and natural expression of user intent. Hand-drawn sketches are a natural and effective way of human thinking expression and communication. Simple graphics are used to express people's abstract thinking, which has both the semantic characteristics of text and the effect of "one picture is worth a thousand words", so that people can immediately think of the semantic information conveyed by a sketch after seeing it. The characteristics of sketch, such as rapid sketching, abstraction, and symbolization, make it a good information expression medium, which improves the efficiency of user intent conversion. With the increasing popularity of touch screens, human-computer interaction has changed a lot. Touch screens have become the mainstream input method, and people are increasingly frequently drawing sketches through touch screens, generating a large number of electronic sketches. Unlike traditional paper and pen interaction, digital sketches can record stroke timing information during drawing, reproduce the evolution of thinking, and provide more dimensional features for sketch understanding, in addition to the advantages of easy modification, easy saving, and fast dissemination.

[0003] Early sketch datasets are mostly collected for single sketch object. With the emergence of cross-modal retrieval and generation tasks, the construction of sketch datasets has changed in two aspects: from single modality to multi-modality; from single object to multi-object (scene sketch). Scene sketch is composed of multiple foreground objects and background. Compared with single object sketch, it contains more complex and diverse semantic information and has a wider range of applications in real life. Single object sketch datasets are mostly collected by freehand drawing and saved in vector format. Existing SketchyScene (Zou, C., Yu, Q., Du, R., Mo, H., Song, Y. Z., Xiang, T.,... & Zhang, H. (2018). SketchyScene: Richly-annotated scene sketches. In Proceedings of the European Conference on Computer Vision (ECCV) (pp. 421-436).) and SketchyCOCO (Gao, Chengying, et al. "SketchyCOCO: Image generation from freehand scene sketches." Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2020.) contribute two scene sketch datasets containing foreground objects and background. However, they are both constructed by dragging single object sketches to the corresponding positions and saved in image format, losing the vector and timing characteristics unique to sketches. In addition to the intuitive visual features like images, sketches also preserve the order of stroke drawing. At the same time, due to its inherent abstractness and sparsity, directly using deep convolutional models commonly used for images cannot fully exploit the features of sketches. Scene sketch datasets containing stroke timing information can fully exploit fine-grained features such as timing, space, and visual pixels, and play an important role in scene sketch recognition and understanding tasks.

[0004] Sketch semantic segmentation is a basic task of sketch understanding. The results of segmentation can be applied to downstream tasks such as sketch coloring, sketch image generation, sketch image retrieval, etc. According to the granularity of segmentation and the type of semantic label, sketch semantic segmentation can be divided into single object semantic segmentation and scene-level semantic segmentation. Single object sketch contains fewer strokes and part categories, so the segmentation difficulty is lower. Traditional sketch semantic segmentation methods are mostly to segment different parts of the object. In actual use, scene sketches appear more frequently and contain more semantic information, so it is more challenging than single object sketch semantic segmentation. However, existing scene sketch segmentation methods, such as LDP (reference: Ge, Ce, et al. "Exploring Local Detail Perception for Scene Sketch Semantic Segmentation." IEEE Transactions on Image Processing 31 (2022): 1447-1461.), are improved on the basis of image semantic segmentation methods, and cannot fully consider the characteristics inherent in sketches. SUMMARY

[0005] In order to solve the above problems, the present application proposes a scene sketch semantic segmentation method and device based on strokes, which introduces a time sequence feature encoder based on bidirectional long short-term memory network BiLSTM and a spatial feature encoder based on graph convolution neural network (GCN) into the algorithm, and proposes a deep model that can efficiently extract stroke spatio-temporal features and realize semantic segmentation of scene sketches. Due to the abstractness, time sequence and sparsity of sketches, feature learning and fusion of strokes from different angles can achieve better recognition effect than methods based on visual pixels. The present application classifies and identifies each stroke in the foreground and background of the scene sketch by fusing the features of appearance, time sequence and space at three different levels, and realizes scene sketch semantic segmentation based on stroke features.

[0006] To achieve the above purpose, the technical scheme adopted by the present application is as follows:

[0007] A scene sketch semantic segmentation method based on strokes, comprising the following steps:

[0008] extracting the appearance features of strokes in the scene sketch;

[0009] using the appearance features of strokes as input, capturing the context semantic relationship of strokes in the time dimension through recurrent neural network, encoding the time sequence features of strokes, and obtaining the time sequence features;

[0010] The graph structure between strokes is constructed by the positional relationship of adjacent strokes, the time sequence feature of each stroke is taken as the node of the graph, the adjacent relationship between strokes is calculated to obtain an adjacency matrix, and the spatial feature of the strokes is encoded by a graph convolutional neural network to obtain the spatial feature.

[0011] The time sequence feature and the spatial feature are fused, the category of each stroke is predicted according to the fused feature, and the scene sketch semantic segmentation based on strokes is completed.

[0012] Further, the appearance feature of the strokes in the scene sketch includes:

[0013] The geometric and positional features of each stroke in the scene sketch are calculated.

[0014] The texture feature of each stroke is extracted by a convolutional neural network, and the geometric and positional features are combined into the appearance feature of the stroke by splicing.

[0015] Further, the geometric and positional features of the strokes in the scene sketch include: the relative coordinate values of the upper left and lower right of the stroke bounding box are 4-dimensional, the time duration of the stroke from the pen-down to the pen-up is 1-dimensional, and the relative length of the stroke is 1-dimensional.

[0016] Further, the texture feature of each stroke is extracted by a convolutional neural network, and the geometric and positional features are combined into the appearance feature of the stroke by splicing.

[0017] Further, the recurrent neural network used to obtain the time sequence feature of the strokes is a bidirectional long short-term memory network.

[0018] Further, the spatial feature of the strokes is encoded by a graph convolutional neural network, which is a four-layer GCN network for updating the spatial feature of the graph structure of the scene sketch.

[0019] Further, the category of each stroke is predicted according to the fused feature, which is a softmax function processing the spatio-temporal fused feature vector to obtain the category probability corresponding to each stroke.

[0020] Further, a cross-entropy loss function is used to adjust the model parameters of the recurrent neural network and the graph convolutional neural network to complete the model training.

[0021] A scene sketch semantic segmentation device based on strokes includes:

[0022] An appearance feature extraction module is configured to extract appearance features of strokes in the scene sketch;

[0023] A time sequence feature encoding module is configured to use the appearance features of the strokes as input, capture context semantic relationships of the strokes in a time dimension through a recurrent neural network, encode time sequence features of the strokes, and obtain the time sequence features;

[0024] A spatial feature encoding module is configured to construct a graph structure between the strokes through position relationships of adjacent strokes, use the time sequence features of each stroke as nodes of the graph, simultaneously calculate adjacent relationships between the strokes to obtain an adjacency matrix, encode spatial features of the strokes through a graph convolutional neural network, and obtain the spatial features;

[0025] A feature fusion module is configured to fuse the time sequence features and the spatial features;

[0026] A category prediction module is configured to predict categories of each stroke according to the fused features, and complete semantic segmentation of the scene sketch based on the strokes.

[0027] In summary, compared with the prior art, the application has the following advantages and positive effects:

[0028] 1. Compared with the prior single-object sketch segmentation method, the method can process scene sketches containing more strokes and more complex semantic information, and accurately complete the semantic segmentation task of the strokes of the scene sketch.

[0029] 2. Compared with the prior method of converting the sketch into an image format to perform semantic segmentation of the scene sketch, the method fully utilizes the characteristics of the strokes, and avoids the problems of abstraction and sparsity existing in the image format sketch.

[0030] 3. The method breaks through the combination of the bidirectional time sequence memory network and the graph convolutional neural network, learns the context information of the strokes in the time sequence and the adjacent relationships of the strokes in the space, and performs time-space feature fusion, so that the accuracy of the scene sketch semantic segmentation task based on the strokes can be effectively improved. BRIEF DESCRIPTION OF DRAWINGS

[0031] Figure 1 is a flowchart of the scene sketch semantic segmentation method based on the strokes.

[0032] Figure 2 is a network structure diagram of the scene sketch semantic segmentation based on the strokes.

[0033] Figure 3 is a module composition diagram of the scene sketch semantic segmentation device based on the strokes. DETAILED DESCRIPTION

[0034] In order to make the person in the technical field better understand the present application, the technical scheme of the stroke-based semantic segmentation algorithm proposed by the present application is described in further detail below in combination with the drawings, but does not constitute a limitation on the present application.

[0035] The present application proposes a stroke-based scene sketch semantic segmentation method, and the flowchart thereof is as shown in the figure Figure 1 The method mainly includes the processes of extracting appearance features, time sequence feature coding, spatial feature coding of strokes in the scene sketch, and multi-classification of strokes, Figure 2 The network structure diagram of the stroke-based scene sketch semantic segmentation is shown.

[0036] 1. Stroke appearance feature extraction

[0037] It is assumed that the scene sketch is a vector data set saved by m strokes in the drawing order. Each stroke is connected by n sampling points in sequence, and is represented as:

[0038] s=[(x ii y1) i (x zi y z ) i ... i (x ni y n )I

[0039] Where (x k , y k ) is the coordinate of the kth sampling point in the stroke S, and n is the number of sampling points contained in the stroke S.

[0040] The sketch is to connect all the strokes, and the visual pattern composed of strokes of different shapes at different positions on the drawing board is displayed. Convolutional neural network is a set of neural networks containing convolutional calculation, and a pre-trained Resnet50 is used as a basic skeleton network in the present application to extract the texture feature of each stroke. By converting the scene sketch into an image format, the image in the bounding box area of each stroke is intercepted and sent into the Resnet50 to extract a 256-dimensional stroke texture feature vector f i cnn .

[0041] In the present application, other appearance features of the stroke are also extracted:

[0042] (1) Stroke bounding box relative position coordinates, with a feature dimension of 4:

[0043]

[0044] where (bx1,by1) is the coordinate of the top-left corner of the stroke bounding box, (bx2,by2) is the coordinate of the bottom-right corner of the stroke bounding box, width and height are the length and width of the sketch respectively.

[0045] (2) Stroke relative length, feature dimension is 1 dimension:

[0046]

[0047] (3) Stroke duration from pen down to pen up f i dur , feature dimension is 1 dimension.

[0048] Finally, these features are spliced to form the appearance feature of the i-th stroke as a whole, which is a 262-dimensional feature vector:

[0049] f i =concat(f i len , f i dur , f i box , f i cnn )

[0050] Finally, the appearance feature F of the entire sketch can be represented as F={f1,f2,…,f m}, where m represents the number of strokes in the scene sketch.

[0051] 2. Time sequence feature coding module based on bidirectional long short-term memory network

[0052] The sketch data in vector format is time sequence data saved by strokes in the order of drawing time. In the process of drawing a sketch, strokes belonging to the same object are closer in time sequence, so learning the context relationship of strokes in time sequence is crucial for the classification and recognition task of sketch strokes. Generally, long short-term memory network (LSTM) is used for time sequence data, and bidirectional long short-term memory network (BiLSTM) can more effectively extract forward and reverse time sequence features, so the application preferably uses BiLSTM network to learn the correlation of sketch stroke data in the time dimension, to code the stroke features in time sequence and extract the time sequence features of the strokes. The formulas of the forward and reverse modules of BiLSTM are as follows:

[0053]

[0054]

[0055] where and represent the forward and reverse operations of BiLSTM respectively, dh This is the dimension of the hidden layer unit. The output of BiLSTM is H. t =[h1,h2,…,h m ], It is the output of the hidden layer features of BiLSTM.

[0056] The input to BiLSTM is the appearance features extracted from strokes. These appearance features are combined into a feature sequence according to the order in which the strokes were drawn. Different sketches contain different numbers of strokes, and BiLSTM needs to ensure that the stroke sequence length is equal during feature updates. Therefore, this invention uses the sketch with the most strokes in the dataset as a benchmark and pads the input features of other sketches with zeros. After training, only the feature output of each stroke sequence and the final global features are saved, discarding the zero-padding parts.

[0057] 3. Spatial Feature Encoding Module Based on Graph Convolutional Neural Network

[0058] The temporal dependence of strokes is uncertain. For example, the last stroke of a previous object and the first stroke of a subsequent object may be temporally adjacent but spatially belong to different objects. Furthermore, during sketching, users often prioritize drawing the overall outlines of different objects before adding details, further contributing to the temporal discontinuity of strokes. To address the problems inherent in temporal encoding, this invention incorporates spatial information into the algorithm model. The spatial feature encoding module primarily learns the spatial correlations of different strokes through a graph convolutional neural network. For scene sketches, each stroke is treated as a node in a graph structure, constructing a spatial graph structure for the strokes. Since graph convolutional neural networks transmit information and update parameters through nodes, establishing an association matrix is ​​crucial. Given any two nodes v in the graph structure... i and v j If the stroke S i The bounding box and stroke S j The sampling points have an intersection, and the edge e between them ij =1; if stroke S i The bounding box and S j The sampling points have no intersection, and the edge e between them ij =0, the calculation formula is as follows:

[0059]

[0060] Where B(.) is the bounding box of a stroke, b(.) is the set of sampling points of a stroke, and E = {e ij The matrix} is used to represent edges.

[0061] After the spatial graph structure of the sketch is established, a four-layer graph convolutional neural network is used to build the stroke graph structure. The input of each layer of the graph convolutional neural network is the output of the previous layer, and the input feature of the first layer is the sketch time sequence feature encoded by the time sequence feature encoder. The feature is updated by the following formula:

[0062]

[0063]

[0064] wherein, is the adjacency matrix of the graph, I is the unit matrix, W (l) is a learnable weight matrix. P (l) is the feature of the l-th layer graph convolutional instance node, P (0) is the input layer of the graph convolutional neural network, and ReLU is the activation function used.

[0065] Since the size of the graph structure of the graph convolutional neural network is fixed, the same method as in the time sequence feature encoder is used to pad the sketch with fewer strokes to the maximum number of strokes in the data set. In order to learn the spatial information into the features of the strokes, the features in the input layer P (0) of the graph convolutional neural network are the features encoded by the time sequence encoder.

[0066] 4. Stroke segmentation

[0067] After the time sequence feature and the spatial feature learned by the time sequence feature encoder and the spatial feature encoder are fused, they are used for class prediction of each stroke. Since the time sequence information is partially lost in the process of feature learning of the graph convolutional neural network, the feature output of the last layer of the graph convolutional neural network is fused with the forward and reverse global features of the BiLSTM. Common fusion methods include splicing, addition, point multiplication, etc. In order to ensure the maximum preservation of the features, the splicing method is used in the present application to obtain the final spatio-temporal feature R i . After that, a fully connected layer is used for feature mapping, and the mapped features are processed by a softmax function to obtain the probability of each object class

[0068]

[0069]

[0070] wherein, P is the feature output of the last layer of the graph convolutional neural network in the spatial feature encoder, is the global feature output of the BiLSTM, and f c (.) is a fully connected layer.

[0071] The present application adopts a cross-entropy loss function as the overall learning goal of the model.

[0072]

[0073] where Y i represents the true label of the stroke category, represents the probability of stroke classification prediction. In order to solve the problem of data class imbalance, different weights w c are assigned to different categories in the present application, and the category with higher frequency is assigned with smaller weight. In the present application, the following calculation method is used to determine the weight of each category:

[0074]

[0075] where Num(c) is the number of objects belonging to category c appearing in the training set.

[0076] In the present application, two indexes P-metric and C-metric in the prior art (reference: Huang, Z., Fu, H. & Lau, R. W. Data driven segmentation and labeling of freehand sketches. ACM Transactions on Graphics (TOG) 33(6), 1-10 (2014)) are used to evaluate the final scene sketch semantic segmentation effect. P-metric represents the proportion of all correctly classified stroke pixels in the test set to the total stroke pixels; C-metric represents the proportion of all correctly classified strokes in the test set to the total number of strokes. Through these two indexes, the segmentation effect of different sketches can be accurately reflected.

[0077] The present application is described below with an example:

[0078] 1. After the user draws the scene sketch, the time-series stroke vector data is obtained.

[0079] 2. The attribute features of each stroke in the sketch data obtained in step 1 are calculated, including 6-dimensional features such as the bounding box relative coordinates of the stroke, the relative length of the stroke, and the stroke drawing time length. The texture features of 256 dimensions of the stroke are extracted using the pre-trained Resnet50 base network, and the attribute features and texture features are spliced into 262-dimensional appearance features.

[0080] 3. The features obtained in step 2 are arranged in the order of strokes to form a feature sequence, which is input into BiLSTM to obtain the time-series features of each stroke and the global time-series features of the stroke.

[0081] 4. Construct the scene sketch stroke graph structure. Based on the proximity relationships of the strokes, construct an adjacency matrix, and use the temporal features from step 3 as the features of the nodes in the graph structure. Update the graph structure using a 4-layer convolutional neural network to obtain the spatial features of each stroke.

[0082] 5. Concatenate the stroke spatial features obtained in step 4 and the stroke global temporal features obtained in step 3, predict the stroke category probability using the softmax function, and finally train the entire model using the cross-entropy loss function.

[0083] 6. Use P-metric and C-metric to evaluate the results of sketch stroke segmentation.

[0084] Another embodiment of the present invention provides a stroke-based scene sketch semantic segmentation device, such as... Figure 3 As shown, it includes:

[0085] The appearance feature extraction module is used to extract the appearance features of strokes in the scene sketch;

[0086] The temporal feature encoding module is used to use the appearance features of strokes as input, capture the contextual semantic relationship of strokes in the time dimension through a recurrent neural network, encode the strokes into temporal features, and obtain temporal features.

[0087] The spatial feature encoding module is used to construct a graph structure between strokes by the positional relationship of adjacent strokes, take the temporal features of each stroke as nodes of the graph, calculate the adjacency matrix of the adjacency relationship between strokes, and encode the spatial features of the strokes through a graph convolutional neural network to obtain spatial features.

[0088] The feature fusion module is used to fuse temporal and spatial features;

[0089] The category prediction module is used to predict the category of each stroke based on the fused features, thus completing the semantic segmentation of the scene sketch based on strokes.

[0090] For the specific implementation process of each module, please refer to the description of the method of the present invention above.

[0091] Another embodiment of the present invention provides a computer device (computer, server, smartphone, etc.) including a memory and a processor, the memory storing a computer program configured to be executed by the processor, the computer program including instructions for performing the steps of the method of the present invention.

[0092] Another embodiment of the present application provides a computer readable storage medium (such as ROM / RAM, magnetic disk, optical disk) storing a computer program, which, when executed by a computer, implements each step of the method of the present application.

[0093] The present application has been described in detail by the foregoing, but it is obvious that the specific implementation forms of the present application are not limited thereto. Various obvious modifications made by those skilled in the art without departing from the spirit of the method of the present application and the scope of the claims are within the protection scope of the present application.

Claims

1. A stroke-based semantic segmentation method for scene sketch, characterized in that, The method comprises the following steps: extracting appearance features of strokes in the scene sketch; using the appearance features of the strokes as input, capturing the context semantic relationship of the strokes in the time dimension through a recurrent neural network, performing time sequence feature coding on the strokes to obtain time sequence features; constructing a graph structure between the strokes through the positional relationship of adjacent strokes, taking the time sequence features of each stroke as nodes of the graph, simultaneously calculating the adjacent relationship between the strokes to obtain an adjacency matrix, performing spatial feature coding on the strokes through a graph convolutional neural network to obtain spatial features; fusing the time sequence features and the spatial features, and performing category prediction on each stroke according to the fused features to complete semantic segmentation of the scene sketch based on the strokes. For the scene sketch, each stroke is taken as a node in the graph structure, and a spatial graph structure of strokes is constructed. The graph convolutional neural network transmits information through nodes and updates parameters. Given any two nodes in the graph structure, if there is an intersection between the bounding box of stroke S i and the sampling point of stroke S j , the edge e i between stroke S j and stroke S ij is 1; if there is no intersection between the bounding box of stroke S i and the sampling point of stroke S j , the edge e i between stroke S j and stroke S ij is 0, and the calculation formula is as follows: B(.) is a bounding box of a stroke, and b(.) is a sampling point set of a stroke.

2. The method of claim 1, wherein, The appearance features of the strokes in the scene sketch are extracted, comprising: calculating the geometric and positional features of each stroke in the scene sketch, extracting the texture features of each stroke through a convolutional neural network, and combining the geometric and positional features and the texture features through splicing to obtain the appearance features of the strokes.

3. The method of claim 1, wherein, The geometric and positional features of the strokes in the scene sketch comprise: relative coordinate values of the upper left and lower right of the stroke bounding box, the time duration of the stroke from the pen-down to the pen-up, and the relative length of the stroke.

4. The method of claim 1, wherein, The texture features of each stroke are extracted through the convolutional neural network, and the geometric and positional features and the texture features are combined through splicing to obtain the appearance features of the strokes, comprising: using a pre-trained Resnet50 as a basic skeleton network to extract 256-dimensional texture features of the strokes, and splicing and combining the texture features and the geometric and positional features of the strokes to obtain a 262-dimensional appearance feature vector.

5. The method of claim 1, wherein, The recurrent neural network is a bidirectional long short-term memory network.

6. The method of claim 1, wherein, The spatial feature coding of the strokes through the graph convolutional neural network is spatial feature updating of the graph structure of the scene sketch using a four-layer GCN network.

7. The method of claim 1, wherein, The category prediction on each stroke according to the fused features is using a softmax function to process the spatio-temporal fused feature vector to obtain the category probability corresponding to each stroke, and using a cross-entropy loss function to adjust the model parameters of the recurrent neural network and the graph convolutional neural network to complete model training.

8. A stroke-based scene sketch semantic segmentation apparatus employing the method of any one of claims 1 to 7, characterized by It comprises: an appearance feature extraction module configured to extract appearance features of strokes in a scene sketch; a time sequence feature coding module configured to use the appearance features of the strokes as input, capture the context semantic relationship of the strokes in the time dimension through a recurrent neural network, and perform time sequence feature coding on the strokes to obtain time sequence features; a spatial feature coding module configured to construct a graph structure between the strokes through the positional relationship of adjacent strokes, take the time sequence features of each stroke as nodes of the graph, simultaneously calculate the adjacent relationship between the strokes to obtain an adjacency matrix, and perform spatial feature coding on the strokes through a graph convolutional neural network to obtain spatial features; a feature fusion module configured to fuse the time sequence features and the spatial features; and a category prediction module configured to perform category prediction on each stroke according to the fused features to complete semantic segmentation of the scene sketch based on the strokes.

9. A computer device, comprising: comprising a memory storing a computer program configured to be executed by a processor, the computer program comprising instructions for performing the method of any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, which, when executed by a computer, implements the method of any one of claims 1-7.

Images