Manipulator sketch semantic segmentation method based on fusion of graph structure and time sequence information

Abstract

The invention relates to the technical field of computer vision, image processing and sketch semantic segmentation, in particular to a manipulator sketch semantic segmentation method based on graph structure and time sequence information fusion, which comprises the following steps of: converting a sketch mode into a graph mode and a sequence mode, respectively inputting a graph structure branch and a sequence branch, therefore, the shape structure information and the time sequence information of the mechanical sketch are fully explored. The graph structure branch adopts graph convolution to carry out feature aggregation and extraction of node features and edge features, the sequence branch adopts a graph enhanced self-attention mechanism module to carry out feature extraction, the module carries out edge coding and stroke coding on features in the graph structure branch, the features serve as induction bias terms and are fused into a self-attention mechanism of the sequence branch, and the self-attention mechanism of the sequence branch is obtained. Centralized coding is added to a sequence input position of a graph enhanced self-attention module, so that manipulator sketch time sequence features and graph structure features are fully extracted and fused, and the capability of an algorithm for semantic segmentation of a manipulator sketch is improved.

Description

Technical Field

[0001] The present invention relates to the technical fields of computer vision, image processing and sketch semantic segmentation, and in particular to a component-level semantic segmentation method for a mechanical hand-drawn sketch. Background Art

[0002] The research and development of industrial software has always been an important and challenging field in China, especially in terms of how to efficiently model human-computer interaction. Sketches, as an important tool for recording and expressing ideas and concepts, have been used by humans since ancient times to reflect real-world objects and ideas. From primitive cave paintings to modern designs, hand-drawn sketches have become a popular and popular form of design. Figure 1 It is an important way for designers to quickly capture inspiration and communicate design ideas in the initial stage.

[0003] Drawing hand-drawn sketches of mechanical parts on multimedia devices and engineering modeling based on artificial intelligence have become a new trend. Sketches usually contain the time sequence information and image shape information drawn by the user. Compared with ordinary hand-drawn sketches, mechanical sketches are more sensitive to shape structures and contain more small decorative strokes. Often, some small stroke differences contain different semantic information. Existing methods cannot fully utilize the sequence information and shape information to distinguish the semantics of mechanical sketch parts when facing mechanical hand-drawn sketches, thus affecting the accuracy of mechanical sketch part-level analysis, which leads to difficulties in downstream tasks such as editing and reconstruction of mechanical hand-drawn sketches. Summary of the invention

[0004] In order to solve the above technical problems existing in the prior art, explore the fine-grained understanding of mechanical sketches and perform component-level semantic segmentation, the present invention combines the temporal recurrent network and the graph convolutional network to fully mine mechanical hand-drawn sketches to achieve high-precision component-level semantic segmentation. The specific technical solution is as follows:

[0005] A semantic segmentation method for mechanical hand-drawn sketches based on the fusion of graph structure and temporal information includes the following steps:

[0006] Step 1: Use a digital drawing board and a hand-drawn pen to collect hand-drawn sketches of different mechanical parts, retain the stroke coordinates and timing information. In order to ensure the alignment of the graph convolution and clarify the abnormal points, the number of key points of each image is set to 512 through RDP downsampling, and then the 512 sampling points of each sample are marked with the component category to which the point belongs. A total of 20 major categories are marked, each of which contains 3-5 minor categories, forming a mechanical hand-drawn sketch semantic segmentation dataset.

[0007] Step 2: Convert the sketch into a graph structure, that is, use the sampling points as the nodes of the graph structure, each stroke as a connected subgraph, and multiple strokes form a graph, which is sent into the graph convolution (GCN) layer. Each layer of graph convolution will aggregate the sketch points and edges, and use the graph structure expansion strategy to add new edges to update the graph structure, so as to extract the sketch shape structure features.

[0008] Step 3: The sketch sequence contains the coordinate information, category information and stroke status information of each point, and is arranged as an array in the drawing order. It is input into the LSTM to extract high-dimensional temporal information, and then sent into the Transformer Encoder to extract the temporal features of the mechanical hand-drawn sketch.

[0009] Step 4: In the graph structure branch, add the edge encoding and stroke encoding of the updated graph structure of each layer to the self-attention of the temporal branch, and add the graph structure centrality encoding after the position plane of the temporal branch, so that the temporal information can recognize and utilize the structural relationship between nodes. When calculating the attention, not only consider the temporal features, but also consider the position and distance of the nodes in the graph.

[0010] Step 5: Add the features of the two branches, pass through LBR (liner, batch normal, relu), and finally output the part prediction probability of each key point through the MLP layer.

[0011] Furthermore, in Step 1, collect the hand-drawn sketches of different mechanical parts through the tablet and the hand-drawing pen, retain the stroke coordinates and temporal information. To ensure the alignment of graph convolution and clear the outliers, set the number of key points of each graph to 512 through RDP downsampling, and then mark the part category to which the 512 sampling points of each sample belong.

[0012] Specifically, for the sequence information, the sequence of the mechanical sketch takes a two-dimensional structured set of N points as input. The i-th point of this sketch can be represented as P i ={(x i ,y i ,s i ,l i ), 0 < i < N}, where (x i ,y i ) represents the coordinate information of the i-th point, N is the total number of key points, s i represents the drawing action, s i = 0 means that the point is starting to draw, and the drawing point is connected to the next one. s i = 1 means that the point is lifting the pen and the drawing is finished, and the drawing point is not connected to the next point. l i represents the manual annotation label of the i-th coordinate point.

[0013] Furthermore, in step 2, each key point of the sketch is taken as a point. Two adjacent points in the same stroke are connected by edges, and different strokes are not connected by edges. The mechanical sketch is thus transformed into a graph structure, which is input into the graph convolution network. Each layer of the graph convolution includes a graph convolution operation and a graph structure update operation, as follows:

[0014] Graph convolution operation: For each layer of graph G l =(V l , E l , f l ), where V l It is Figure G l The node set, E l It is Figure G l The edge set, f l It is Figure G l The node feature set in the lth layer, the feature f of node i i l Through the public

[0015] Update of formula 1:

[0016]

[0017] Where Θ is a learnable weight parameter, the feature update of node i is realized by the feature of neighbor node j, and the maximum value of all neighbor node features after update is taken, and the update operation h Θ as follows:

[0018] h Θ (f i ,f j )=ReLU(MLP Θ (concat(f i ,f j -f i ))) Formula 2

[0019] Graph update operation: A single graph in a sketch is used as the basic unit to update the graph structure by adding new edges using the graph structure expansion strategy. The definition of the graph at the lth layer is:

[0020]

[0021] Among them, ε is the set of edges, To add a new edge structure to the graph:

[0022]

[0023] in is the vertex v i The d-expanded neighborhood of i All nodes within d.

[0024] Furthermore, in step 3, the sketch sequence is input into LSTM to extract high-dimensional time series information, and then sent to Transformer Encoder to extract the time series features of the mechanical hand-drawn sketch. Each layer of Transformer contains 8 multi-head self-attention mechanisms and a forward linear layer with a dimension of 1024. For the sketch sequence at each position, the same linear transformation is performed to obtain three Q, K, and V feature matrices with dimensions of 100*512. The calculation method of self-attention is as follows:

[0025]

[0026] Furthermore, in step 4, in the graph structure branch, the edge coding and stroke coding of each updated graph structure are added to the self-attention of the temporal branch, and the graph structure centrality coding is added to the position surface of the temporal branch, so that the temporal information can identify and utilize the structural relationship between nodes. The centrality coding assigns two real-valued embedding vectors to each node according to its in-degree and out-degree, and adds them to the node features as input:

[0027]

[0028] For edge encoding, each ordered node pair (V i , V j ), we find that from V i To V j The shortest path is calculated and the average dot product of the edge features and the learnable embedding is calculated. This edge encoding is added to the attention mechanism through the bias coefficient, as follows:

[0029]

[0030] in: is the feature in the nth edge, is the nth weight.

[0031] Furthermore, in step 5, the features of the two branches are added together to fuse the features of each stroke point, and then passed through LBR (liner, batch normal, relu), and finally through the MLP layer to output the component prediction probability of each key point. The two-dimensional tensors of the feature outputs of the two branches are globally average pooled based on the sequence length, and the average is added together, input into the linear layer, and normalized using Softmax.

[0032] In summary, we have invented an efficient method for component-level semantic segmentation of mechanical sketches. By representing the sketches as graph structures and temporal information, extracting structural and temporal features for fine-grained understanding, we achieve component-level semantic segmentation. Description of the Drawings

[0033] Figure 1 is a schematic diagram of the overall process of a method for component-level semantic segmentation of mechanical hand-drawn sketches based on the interaction of graph and temporal information according to an embodiment of the present invention;

[0034] Figure 2 is the software interface of the annotation integrating the downsampling algorithm and the visualization function;

[0035] Figure 3 is the feature interaction model diagram of the graph structure information and the self-attention temporal branch;

[0036] Figure 4 is a schematic diagram of the preprocessing of mechanical hand-drawn sketches;

[0037] Figure 5 is the component-level segmentation result of the mechanical hand-drawn sketch;

[0038] Figure 6 is the experimental result graph of the segmentation accuracy of the mechanical hand-drawn sketch. Detailed Embodiment

[0039] In order to make the objectives, technical solutions, and technical effects of the present invention clearer and more understandable, the following further elaborates on the present invention in detail in conjunction with the specification drawings and embodiments.

[0040] As Figure 1 shown, the method for semantic segmentation of mechanical hand-drawn sketches based on the fusion of graph structure and temporal information according to the present invention includes the following steps:

[0041] Step 1: Dataset construction.

[0042] For the construction of the mechanical hand-drawn sketch dataset, it is mainly divided into two parts, namely the collection of hand-drawn sketch information and data preprocessing. The collection of hand-drawn sketch information is mainly recorded through a digital drawing board and a drawing pen, retaining the stroke coordinates and temporal information. The sketch information is saved in the format of P i ={(x i ,y i ,s i ,l i ), 0 < i < N}, where (x i ,y i ) represents the coordinate information of the i-th point, N is the total number of key points, s i represents the drawing action, 0 means drawing, 1 means the end of drawing, and l iRepresents the manual annotation label of the i-th coordinate point. Data preprocessing mainly uses the RDP algorithm to control the sampling points to 512, and uses normalization to scale the coordinate points to a canvas of size 224.

[0043] Step 2: Graph convolution branch.

[0044] In the graph convolution branch, each key point of the sketch is first taken as a point. Two adjacent points in the same stroke are connected by edges, and there is no edge connection between different strokes. The sketch information is converted into a graph structure. The graph structure is input into the graph convolution network. Each layer of the graph convolution includes a graph convolution operation and a graph structure update operation.

[0045] Step 3: Timing branching.

[0046] The sketch sequence is input into LSTM to extract high-dimensional time series information, and then sent to the Transformer decoder to extract the time series features of the mechanical hand-drawn sketch. Each layer of Transformer contains 8 multi-head self-attention mechanisms and a forward linear layer with a dimension of 1024. For the sketch sequence at each position, the same linear transformation is performed to obtain three Q, K, and V feature matrices with dimensions of 100*512, and finally the time series feature output is obtained.

[0047] Step 4: Information exchange.

[0048] In information interaction, the edge coding, stroke coding and centrality coding in the graph convolution branch are mainly used to assist the time series information to converge faster. The edge coding and stroke coding of the graph structure updated at each layer are added to the self-attention of the time series branch. After the graph structure centrality coding is added to the position surface of the time series branch, the time series information can recognize and utilize the structural relationship between nodes. The centrality coding assigns two real-valued embedding vectors to each node according to its in-degree and out-degree, and adds them to the node features as input. For edge coding, each ordered node pair (V i , V j ), we find that from V i To V j The shortest path is obtained by calculating the average dot product of the edge features and the learnable embedding on the path. This edge encoding is added to the attention mechanism through the bias coefficient. The result of adding the attention mechanism as a whole is:

[0049] Step 5: Feature fusion.

[0050] The features of the graph structure branch and the time series branch are fused, and the category probability of each key point is predicted to achieve the effect of component segmentation. The specific operation is to perform global average pooling on the two-dimensional tensors output by the two features based on the sequence length, add the averaged tensors, input them into the linear layer, and use Softmax for normalization, and finally output the prediction results.

Claims

1. A semantic segmentation method for mechanical hand-drawn sketches based on the fusion of graph structure and temporal information, characterized in that: include: Step 1: Use a digital drawing board and a hand-drawn pen to collect hand-drawn sketches of different mechanical parts, retain the stroke coordinates and timing information, set the number of key points of each image to 512 through RDP downsampling, and then manually annotate the component category to which each key point belongs to form a mechanical hand-drawn sketch semantic segmentation dataset; Step 2: Convert the sketch into a graph structure and send it to the graph convolution layer. Each layer of graph convolution will aggregate the sketch points and edges, and use the graph structure expansion strategy to add new edges to update the graph structure and extract the shape structure features of the sketch. Step 3: Input the sketch sequence into LSTM to extract high-dimensional temporal information, and then send it to the Transformer decoder to extract the temporal features of the mechanical hand-drawn sketch; Step 4: Add the edge coding and stroke coding of each updated graph structure to the self-attention of the temporal branch, and add the graph structure centrality coding to the position plane of the temporal branch, so that the temporal information can identify and utilize the structural relationship between nodes; Step 5: Add the graph structure branch and the time series branch features, pass through LBR, and finally pass through the MLP layer to output the component prediction probability of each key point.

2. The method for semantic segmentation of mechanical hand-drawn sketches based on fusion of graph structure and temporal information as claimed in claim 1, characterized in that: In step 1, a digital drawing board is used to collect the stroke trajectory information drawn by the painter. The storage format of the sequence trajectory information is {(x i ,y i ,s i ,l i ), 0 < i < n}, where (x i ,y i ) is the coordinate information of the sampling point, n is the total number of key points, set to 512, s i ,t i represent the drawing actions. If s i = 0, it means that the stroke at this state point has not been lifted, indicating that this point is connected to the next point. If s i = 1, it means that the stroke at this state point has been lifted, indicating that this point is the last point of this stroke, that is, the drawing stops.

3. The method for semantic segmentation of mechanical hand-drawn sketches based on the fusion of graph structure and temporal information as claimed in claim 1, characterized in that: In step 2, the sketch record points are used as graph nodes, the stroke sequence is connected as edges, one stroke constitutes a subgraph, and a sketch is composed of non-connected subgraphs composed of different strokes. They are sent to the graph convolution layer. Each layer of graph convolution will aggregate the sketch points and edges, and use the graph structure expansion strategy to add new edges to update the graph structure, so as to extract the shape structure features of the sketch.

4. The method for component-level semantic segmentation of mechanical hand-drawn sketches based on the interaction of graph and temporal information as claimed in claim 1, characterized in that: In step 3, the sketch sequence information is composed of the coordinate values ​​of each point in order. The sketch sequence is input into the LSTM to extract the high-dimensional time series information, and then sent to the Transformer decoder to extract the time series features of the mechanical hand-drawn sketch.

5. The method for semantic segmentation of mechanical hand-drawn sketches based on fusion of graph structure and temporal information as claimed in claim 1, characterized in that: In step 4, the centrality encoding assigns two real-valued embedding vectors to each node according to its in-degree and out-degree, and adds them to the node features as input:

6. The method for semantic segmentation of mechanical hand-drawn sketches based on fusion of graph structure and temporal information as claimed in claim 1, characterized in that: In step 4, the edge encoding uses the shortest path to find the points in the connected graph and calculates the average of the dot product of the edge features on the path and the learnable embedding. This edge encoding is added to the attention mechanism through the bias coefficient.

7. The method for semantic segmentation of mechanical hand-drawn sketches based on fusion of graph structure and temporal information as claimed in claim 1, characterized in that: In step 5, the two-dimensional tensors output by the graph structure branch and the time series branch are globally averaged pooled based on the sequence length, the averaged tensors are added together, input into the linear layer, and normalized using Softmax, and finally the prediction results are output.

Images