A mechanical hand sketch semantic segmentation method based on graph structure and time sequence information fusion

Abstract

This invention relates to the fields of computer vision, image processing, and sketch semantic segmentation, and particularly to a method for semantic segmentation of mechanical hand-drawn sketches based on the fusion of graph structure and temporal information. The method includes the following steps: transforming the sketch modality into a graph modality and a sequence modality, and inputting them into a graph structure branch and a sequence branch respectively, thereby fully exploring the shape structure information and temporal information of the mechanical sketch. The graph structure branch uses graph convolution to aggregate and extract node and edge features, while the sequence branch uses a graph-enhanced self-attention mechanism module for feature extraction. This module integrates the features from the graph structure branch into the self-attention mechanism of the sequence branch as inductive bias terms through edge encoding and stroke encoding, and adds them to the sequence input of the graph-enhanced self-attention module through centrality encoding. This fully extracts and fuses the temporal and graph structure features of the mechanical hand-drawn sketch, improving the algorithm's ability to semantically segment mechanical hand-drawn sketches.

Description

Technical Field

[0001] This invention relates to the fields of computer vision, image processing, and sketch semantic segmentation, and in particular to a component-level semantic segmentation method for mechanical hand-drawn sketches. Background Technology

[0002] The development of industrial software has always been an important and challenging field in China, especially in how to efficiently perform human-computer interaction modeling, which has been a focus of attention. Sketches, as an important tool for recording and expressing thoughts and concepts, have been used by humans since ancient times to reflect objects and ideas in the real world. From primitive cave paintings to modern design, hand-drawn sketches... Figure 1 Straight lines are an important way for designers to quickly capture inspiration and communicate design ideas in the early stages.

[0003] Drawing hand-drawn sketches of mechanical parts on multimedia devices, and using artificial intelligence for engineering modeling, has become a new trend. Sketches typically contain temporal and shape information drawn by the user. Compared to ordinary hand-drawn sketches, mechanical sketches are more sensitive to shape and structure and contain more subtle decorative strokes. Often, these minor stroke differences contain different semantic information. Existing methods cannot fully utilize the temporal and shape information to distinguish the semantics of mechanical sketch components, thus affecting the accuracy of component-level analysis. This leads to difficulties in downstream tasks such as editing and reconstructing mechanical sketches. Summary of the Invention

[0004] To address the aforementioned technical problems in existing technologies, and to explore fine-grained understanding and component-level semantic segmentation of mechanical sketches, this invention combines temporal recurrent networks and graph convolutional networks to fully mine mechanical hand-drawn sketches, thereby achieving high-precision component-level semantic segmentation. The specific technical solution is as follows:

[0005] A semantic segmentation method for mechanical hand-drawn sketches based on graph structure and temporal information fusion includes the following steps:

[0006] Step 1: Collect hand-drawn sketches of different mechanical parts using a digital drawing board and a pen, preserving stroke coordinates and temporal information. To ensure the alignment of the graph convolution and to clearly identify outliers, the number of key points in each graph is set to 512 through RDP downsampling. Then, the 512 sampling points of each sample are labeled with the part category to which the point belongs, resulting in a total of 20 major categories, each containing 3-5 subcategories, forming a semantic segmentation dataset of mechanical hand-drawn sketches.

[0007] Step 2: Convert the sketch into a graph structure. That is, take the sampling points as the nodes of the graph structure, each stroke as a connected subgraph, and multiple strokes form a graph, which is sent into the graph convolution (GCN) layer. Each layer of graph convolution will aggregate the sketch points and edges, and adopt a graph structure expansion strategy to add new edges to update the graph structure, so as to extract the sketch shape structure features.

[0008] Step 3: The sketch sequence contains the coordinate information, category information and stroke status information of each point, and is arranged as an array in the drawing order. It is input into the LSTM to extract high-dimensional temporal information, and then sent into the Transformer Encoder to extract the temporal features of the mechanical hand-drawn sketch.

[0009] Step 4: In the graph structure branch, add the edge encoding and stroke encoding of the updated graph structure of each layer to the self-attention of the temporal branch, and add the graph structure centrality encoding after the position plane of the temporal branch, so that the temporal information can recognize and utilize the structural relationship between nodes. When calculating the attention, not only consider the temporal features, but also consider the position and distance of the nodes in the graph.

[0010] Step 5: Add the features of the two branches, pass through LBR (liner, batch normal, relu), and finally output the part prediction probability of each key point through the MLP layer.

[0011] Furthermore, in Step 1, collect the hand-drawn sketches of different mechanical parts through the tablet and the hand-drawing pen, retain the stroke coordinates and temporal information. To ensure the alignment of graph convolution and clear the outliers, set the number of key points of each graph to 512 through RDP downsampling, and then mark the part category to which the 512 sampling points of each sample belong.

[0012] Specifically, for the sequence information, the sequence of the mechanical sketch takes a two-dimensional structured set of N points as the input. The i-th point of this sketch can be represented as P i ={(x i ,y i ,s i ,l i ), 0 < i < N}, where (x i ,y i ) represents the coordinate information of the i-th point, N is the total number of key points, s i represents the drawing action, s i = 0 means that the point is starting to draw, and the drawing point is connected to the next one. s i = 1 means that the point is lifting the pen and the drawing is ended, and the drawing point is not connected to the next point. l i represents the manual annotation label of the i-th coordinate point.

[0013] Furthermore, in step 2, each key point of the sketch is treated as a point. Adjacent points within the same stroke are connected by edges, while different strokes are not connected by edges. The mechanical sketch is thus transformed into a graph structure. This graph structure is then input into a graph convolutional network. Each layer of the graph convolutional network includes a graph convolution operation and a graph structure update operation, as detailed below:

[0014] Graph convolution operation: For each layer of graph G l =(V l E l f l ), where V l Is it Figure G? l The set of nodes, E l Is it Figure G? l The set of edges, f l Is it Figure G? l In the node feature set in the l-th layer, the feature f of node i is... i l Through public

[0015] Update to Formula 1:

[0016]

[0017] Where Θ is a learnable weight parameter, the feature update of node i is achieved by using the features of its neighbor node j, and the maximum value of the updated features of all neighbor nodes is taken. The update operation h Θ as follows:

[0018] h Θ (f i ,f j =ReLU(MLP) Θ (concat(f i ,f j -f i ))) Formula 2

[0019] Graph update operation: The graph structure is updated by adding new edges using a graph structure expansion strategy, taking a single graph in a sketch as the basic unit. The graph at level l is defined as follows:

[0020]

[0021] Where ε is the set of edges, To add a new edge structure to the graph:

[0022]

[0023] in It is vertex v i The d-extended neighborhood means that it includes v. i All nodes within d.

[0024] Further, in step 3, the sketch sequence is input into an LSTM to extract high-dimensional temporal information, and then fed into a Transformer Encoder to extract the temporal features of the mechanical hand-drawn sketch. Each Transformer layer contains eight multi-head self-attention mechanisms and a 1024-dimensional forward linear layer. For each position of the sketch sequence, the same linear transformation is performed to obtain three 100*512-dimensional Q, K, and V feature matrices. The self-attention is calculated as follows:

[0025]

[0026] Furthermore, in step 4, in the graph structure branch, the edge encoding and stroke encoding of the updated graph structure at each layer are added to the self-attention of the temporal branch. After adding the graph structure centrality encoding to the position plane of the temporal branch, the temporal information can be used to identify and utilize the structural relationships between nodes. The centrality encoding assigns two real-valued embedding vectors to each node based on its in-degree and out-degree, and adds them to the node features as input.

[0027]

[0028] For edge encoding, each ordered node pair (V) i V j We found from V i To V j The shortest path is found, and the average of the dot products of the edge features on the path and the learnable embeddings is calculated. This edge encoding is added to the attention mechanism through a bias coefficient, as shown in the following formula:

[0029]

[0030] in: The features of the nth edge, It is the nth weight.

[0031] Further, in step 5, the features of the two branches are added together to fuse the features of each stroke point. This is then processed through LBR (linear, batch normal, ReLU) and finally through an MLP layer to output the component prediction probability for each keypoint. The two-dimensional tensors output by the two features are globally averaged based on the sequence length, and the averages are added together before being input into a linear layer and normalized using Softmax.

[0032] In summary, we have invented an efficient method for component-level semantic segmentation of mechanical sketches. By representing the sketches as graph structures and temporal information, extracting structural and temporal features for fine-grained understanding, we can achieve component-level semantic segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 FIG. is a schematic diagram of the overall process of a method for component-level semantic segmentation of mechanical hand-drawn sketches based on the interaction of graph and temporal information according to an embodiment of the present invention;

[0034] Figure 2 is a software interface for annotation integrating a downsampling algorithm and visualization functions;

[0035] Figure 3 is a feature interaction model diagram of graph structure information and self-attention temporal branches;

[0036] Figure 4 is a schematic diagram of the preprocessing of mechanical hand-drawn sketches;

[0037] Figure 5 is the component-level segmentation result of mechanical hand-drawn sketches;

[0038] Figure 6 is a graph of the experimental results of the segmentation accuracy of mechanical hand-drawn sketches. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0039] In order to make the objectives, technical solutions, and technical effects of the present invention clearer and more understandable, the present invention will be further described in detail below with reference to the accompanying drawings of the specification and embodiments.

[0040] As Figure 1 shown, the method for semantic segmentation of mechanical hand-drawn sketches based on the fusion of graph structure and temporal information according to the present invention includes the following steps:

[0041] Step 1: Dataset construction.

[0042] For the construction of a mechanical hand-drawn sketch dataset, it is mainly divided into two parts, namely the collection of hand-drawn sketch information and data preprocessing. The collection of hand-drawn sketch information is mainly recorded through a digital drawing board and a drawing pen, retaining the stroke coordinates and temporal information. The sketch information is saved in the format of P i ={(x i , y i , s i , l i ), 0 < i < N}, where (x i , y i ) represents the coordinate information of the i-th point, N is the total number of key points, s i represents the drawing action, 0 means drawing, 1 means ending drawing, and l iThis represents the manually added label for the i-th coordinate point. Data preprocessing primarily uses the RDP algorithm to control the number of sampling points to 512, and normalization is used to scale the coordinate points to a canvas of size 224.

[0043] Step 2: Graph convolution branch.

[0044] In the graph convolution branch, each key point of the sketch is first treated as a point. Adjacent points within the same stroke are connected by edges, while different strokes are not connected by edges. This transforms the sketch information into a graph structure. The graph structure is then input into the graph convolutional network, where each layer includes a graph convolution operation and a graph structure update operation.

[0045] Step 3: Timing branch.

[0046] The sketch sequence is input into an LSTM to extract high-dimensional temporal information, and then fed into a Transformer decoder to extract the temporal features of the mechanical hand-drawn sketch. Each Transformer layer contains eight multi-head self-attention mechanisms and a 1024-dimensional forward linear layer. For each position of the sketch sequence, the same linear transformation is performed to obtain three 100*512-dimensional Q, K, and V feature matrices, and finally, the temporal feature output is obtained.

[0047] Step 4: Information exchange.

[0048] In information interaction, edge encoding, stroke encoding, and centrality encoding in the graph convolution branch are mainly used to assist temporal information in achieving faster convergence. The edge and stroke encodings of the updated graph structure at each layer are added to the self-attention of the temporal branch. After adding the graph structure centrality encoding to the positional plane of the temporal branch, the temporal information can recognize and utilize the structural relationships between nodes. The centrality encoding assigns two real-valued embedding vectors to each node based on its in-degree and out-degree, and adds them to the node features as input. For edge encoding, each ordered node pair (V... i V j We found from V i To V j The shortest path is found, and the average of the dot products of the edge features on the path and the learnable embeddings is calculated. This edge encoding is added to the attention mechanism via a bias coefficient. The overall result after incorporating it into the attention mechanism is:

[0049] Step 5: Feature fusion.

[0050] The graph structure branch and temporal branch features are fused to predict the class probability of each keypoint, thus achieving component segmentation. Specifically, the two-dimensional tensors output by both features are globally averaged based on the sequence length, the averages are summed, input into a linear layer, normalized using Softmax, and finally the prediction result is output.

Claims

1. A semantic segmentation method for mechanical hand-drawn sketches based on the fusion of graph structure and temporal information, characterized in that, include: Step 1: Collect hand-drawn sketches of different mechanical parts using a digital drawing board and a hand-drawing pen, retain the stroke coordinates and temporal information, set the number of key points of each image to 512 through RDP downsampling, and then manually label the component category to which each key point belongs to form a semantic segmentation dataset of mechanical hand-drawn sketches. Step 2: Convert the sketch into a graph structure and feed it into a graph convolutional layer. Each graph convolutional layer will aggregate the sketch points and edges, and use a graph structure expansion strategy to add new edges to update the graph structure and extract the sketch shape and structure features. Step 3: Input the sketch sequence into the LSTM to extract high-dimensional temporal information, and then send it into the Transformer decoder to extract the temporal features of the mechanical hand-drawn sketch; Step 4: Add the edge and stroke codes of the updated graph structure at each layer to the self-attention of the temporal branch, and add the graph structure centrality code to the position surface of the temporal branch so that the temporal information can identify and utilize the structural relationships between nodes. Step 5: Add the features of the graph structure branch and the temporal branch, pass them through LBR, and finally pass them through the MLP layer to output the component prediction probability of each key point; In step 1, a digital drawing board is used to collect the stroke trajectory information drawn by the artist. The sequence trajectory information is stored in the following format: ,in It is the coordinate information of the sampling point. This is the total number of key points, set to 512. Represents the drawing action. This represents the manually added annotation label for the i-th coordinate point. This indicates that the stroke at this state point has not been lifted, and that this point is connected to the next point. This indicates that the stroke has been lifted at this state point, and that this state point is the last point of this stroke, meaning that drawing has stopped. In step 4, the centrality encoding assigns two real-valued embedding vectors to each node based on its in-degree and out-degree, and adds them to the node features as input: ; In step 4, the edge encoding uses the shortest path to find the points in the connected graph and calculates the average value of the dot product of the edge features on the path and the learnable embedding. This edge encoding is added to the attention mechanism through the bias coefficient. In step 5, the two-dimensional tensors output by the graph structure branch and the temporal branch are globally averaged based on the sequence length, the averaged tensors are summed, input into the linear layer, and normalized using Softmax. Finally, the prediction results are output.

2. The semantic segmentation method for mechanical hand-drawn sketches based on graph structure and temporal information fusion as described in claim 1, characterized in that: In step 2, sketch recording points are used as graph nodes, stroke sequence connections are used as edges, one stroke constitutes a subgraph, and a sketch consists of non-connected subgraphs composed of different strokes. These are fed into a graph convolutional layer. Each graph convolutional layer aggregates sketch points and edges, and a graph structure expansion strategy is used to add new edges to update the graph structure, thereby extracting sketch shape and structural features.

3. The semantic segmentation method for mechanical hand-drawn sketches based on graph structure and temporal information fusion as described in claim 1, characterized in that: In step 3, the sketch sequence information is composed of the coordinate values ​​of each point in sequence. The sketch sequence is input into LSTM to extract high-dimensional temporal information, and then sent to the Transformer decoder to extract the temporal features of the mechanical hand-drawn sketch.

Images