Contour line recognition method and device based on bidirectional long short-term memory neural network

Abstract

The application discloses a contour line recognition method and device based on a bidirectional long short-term memory neural network, which comprises the following steps: acquiring a contour of a target picture, extracting key points on a contour line from the target picture based on a key point extraction algorithm, acquiring a feature sequence of the key points and inputting the feature sequence into a trained first neural network model to detect corner points, identifying corner point information from the feature sequence of the key points of different shapes, segmenting the contour line according to the corner point information, segmenting two side line segments / curves adjacent to the corner points to obtain a point set sequence of the segmented line segments / curves, acquiring the point set sequence of the line segments / curves and inputting the point set sequence into a trained second neural network model to recognize the types of the line segments / curves, and obtaining a recognition result of the contour line. The method combines deep learning and a traditional contour line recognition method, is favorable for fast recognition of the contour line, and improves the recognition effect.

Description

Technical Field

[0001] This invention relates to the field of industrial manufacturing automation technology, and in particular to a contour line recognition method and apparatus based on a bidirectional long short-term memory neural network. Background Technology

[0002] In the field of high-energy beam processing (waterjet, laser, plasma, etc.), a common problem is the need to extract the contours of an image, perform feature recognition on the extracted contours, and then plan the processing path based on the recognized feature lines. Contour extraction methods are relatively mature in this workflow. The key step is how to perform feature recognition on the extracted contours. Researching rapid contour recognition can not only reduce human-computer interaction in automated programming systems but also avoid the problem of inconsistent standards in manually judging contour shapes.

[0003] In 2D CNC programming for high-energy beam machining, it is often necessary to convert the 2D graphic contour of the workpiece into a high-level description of the machining contour to facilitate machining path planning. In actual machining, the workpiece contour may originate from non-CAD data such as images, thus requiring contour recognition. The result of contour recognition directly affects the accuracy of the high-level description of the machining contour, thereby impacting the efficiency and quality of workpiece machining. For non-CAD data such as images, the extracted contour lines consist of multiple line segments / curves. Therefore, contour recognition involves two steps: 1) Contour line segmentation: segmenting different line segments / curves using segmentation algorithms. 2) Line segment recognition: identifying the segmented line segments / curves to determine their type (straight line segment, circular arc, spline curve, etc.).

[0004] The current methods are as follows: 1. Nguyen et al. (2011) proposed a contour line segmentation method based on support regions. This method traverses every key point on the curve and determines the support regions on both sides of the key point by including the key point and its consecutive points on both sides in the same rectangular area as much as possible. Then, it calculates the angle formed by the two support regions. When the angle shows a continuous monotonically increasing / decreasing trend and then decreasing / increasing again as the key points are traversed, the points in the common area formed by the continuously increasing / decreasing starting support region and the ending support region are considered to be corner points. Its drawback is that it is difficult to completely segment the arc segment, and the changing trend of the angle formed by the support regions on both sides of the key point in the arc segment is not obvious, so it is difficult to identify the starting and ending points of the arc segment.

[0005] 2. (2020) proposed a corner filtering method based on minimizing the error criterion. This method first estimates the corner curvature of each point on the curve using a combination of support region analysis and principal component analysis (PCA). Then, points with corner curvature greater than a given threshold are extracted as candidate corners. Finally, a suboptimal feature selection method based on minimizing the error criterion is used to progressively filter the candidate corners, resulting in a fixed set of corners. The drawbacks of this method are as follows: the suboptimal solution iterative method requires a fixed number of corners, thus requiring parameter adjustments for different curves to achieve good segmentation results. Furthermore, the search method itself has high complexity, resulting in low segmentation efficiency.

[0006] 3. Ding et al. (2016) proposed a direction transformation algorithm based on principal component analysis. Addressing the problem in corner-based contour segmentation methods where thresholding can segment multiple line segments into a single curve, they introduced the tangent direction of key points. By analyzing changes in the tangent direction, they determined whether a key point was a corner. The drawbacks of this method are as follows: it struggles to handle curve types other than circular arcs. For such curves, the tangent direction does not exhibit a uniform increasing or decreasing trend, making corner identification difficult.

[0007] In summary, most existing contour line recognition methods employ corner detection based on curvature and tangent direction. Building upon previous research, they have improved corner detection accuracy by introducing methods such as support region denoising and PCA analysis for threshold selection. However, these methods either require manual parameter adjustment for different curve types and scenarios or have high computational complexity, making it difficult to achieve rapid contour line segmentation and recognition.

[0008] Currently, with the development of deep learning, neural network-based algorithms have achieved great success in fields such as image processing and natural language processing. The advantage of neural networks lies in their ability to automatically learn features, thus avoiding the problem of experts manually extracting features for different scenarios. However, in industrial fields such as computer-aided manufacturing, the application of deep learning remains relatively limited due to the limited amount of data and high cost of data annotation in most scenarios.

[0009] Therefore, it is necessary to combine deep learning and traditional contour recognition methods. A fast contour recognition method based on bidirectional long short-term memory network (Bi-LSTM) is proposed to solve the problems of existing contour recognition algorithms that rely on manual parameter adjustment, have high computational complexity, and have unsatisfactory recognition results. Summary of the Invention

[0010] The purpose of this invention is to provide a contour line recognition method and device based on a bidirectional long short-term memory neural network. Combining deep learning and traditional contour line recognition methods, this invention proposes a fast contour line recognition method based on a bidirectional long short-term memory network (Bi-LSTM) to solve the problems of existing contour line recognition algorithms that rely on manual parameter adjustment, have high computational complexity, and have unsatisfactory recognition results. At the same time, it provides a new approach that combines deep learning and traditional methods, avoiding the disadvantage of deep learning being difficult to train in scenarios with limited data.

[0011] This invention provides a contour line recognition method based on a bidirectional long short-term memory neural network, comprising:

[0012] Obtain the outline of the target image, and extract key points on the outline line from the target image based on the key point extraction algorithm;

[0013] The feature sequences of key points are obtained and input into the trained first neural network model for corner detection, and corner information is identified from the feature sequences of variable-length key points that make up contours of different shapes.

[0014] The contour line is segmented according to the corner point information, and the line segments / curves on both sides adjacent to the corner point are segmented to obtain a point set sequence of line segments / curves after the contour line is segmented.

[0015] Obtain the point set sequence of line segments / curves and input it into the trained second neural network model to perform line segment / curve type recognition, and obtain the contour line recognition result.

[0016] Preferably, both the first neural network model and the second neural network model integrate convolutional neural networks and bidirectional long short-term memory networks, and the first neural network model is a line segmentation model, while the second neural network model is a line segment category recognition model.

[0017] The first neural network model and the second neural network model share a backbone network structure, which includes two or more convolutional units and bidirectional LSTM units.

[0018] The convolutional unit includes a first convolutional unit and a second convolutional unit, which are connected to each other to take the key point sequence as low-level feature input and extract high-level features.

[0019] The bidirectional LSTM unit includes a first LSTM unit and a second LSTM unit respectively connected to the convolutional unit, used to extract features of the keypoint sequence from both forward and backward directions, so that the backbone network can extract features covering the entire keypoint sequence.

[0020] Preferably, the step of obtaining the feature sequence of key points and inputting it into the trained first neural network model for corner detection further includes:

[0021] In the first neural network model after training, the input is a feature sequence of key points belonging to the same contour, and the output is the probability value of each key point being predicted as a corner point. The feature sequence includes the coordinates of the key point in the x-axis direction, the coordinates in the y-axis direction, the curvature, and the tangent angle.

[0022] If the output probability value is greater than or equal to the preset threshold, then the key point is determined to be a corner point;

[0023] If the output probability value is less than the preset threshold, the key point is determined to be a non-corner point.

[0024] After traversing all key points to obtain the categories of corner points or non-corner points, the corner points in the contour line are cut off to obtain multiple line segments / curves with the corner points as endpoints, thus completing the segmentation of the contour line.

[0025] Preferably, the step of obtaining the point set sequence of line segments / curves and inputting it into the trained second neural network model for line segment / curve type recognition includes:

[0026] The second neural network model after training inputs the feature sequence of key points of line segment / curve and outputs the first probability, second probability and third probability of the line segment / curve being classified into different categories, including straight line segment, circular arc and B spline;

[0027] If the first probability is greater than the second probability and the first probability is greater than the third probability, then the network predicts that the line segment / curve is a straight line segment.

[0028] If the second probability is greater than the first probability and the second probability is greater than the third probability, then the network predicts that the line segment / curve is an arc.

[0029] If the third probability is greater than the first probability and the third probability is greater than the second probability, then the network predicts that the category of the line segment / curve is B-spline;

[0030] The probability that a line segment / curve is classified into different categories satisfies the following condition: m represents the number of categories of line segments / curves.

[0031] Preferably, the first neural network model includes a backbone network unit, a third convolutional unit, and a point-by-point local connection unit connected in sequence. The feature sequence of key points of the same contour is input from the input layer, and the low-level feature is output from the input layer as the input of the backbone network unit. The high-level feature is output by the backbone network unit and used as the input of the third convolutional unit. The corresponding high-level feature is output by the third convolutional unit and used as the input of the point-by-point local connection unit. When the input feature sequence is of variable length, the output layer finally outputs the prediction result of whether each input key point in the feature sequence of arbitrary length is a corner point.

[0032] The second neural network model includes a backbone network unit, a fourth convolutional unit, a first fully connected unit, and a second fully connected unit connected in sequence. The feature sequence of key points on the same line segment / curve after contour line segmentation is input from the input layer. This low-level feature is output from the input layer and used as input to the backbone network unit. The backbone network unit outputs high-level features, which are then used as input to the fourth convolutional unit. Based on the high-level features of each key point, the input feature sequence is padded to a fixed length using the fourth convolutional unit, the first fully connected unit, and the second fully connected unit, even when the input feature sequence is of variable length. Finally, the output layer outputs the predicted probability of the line segment / curve being classified into different categories.

[0033] Preferably, the process of obtaining the outline of the target image includes:

[0034] Determine the corner points on the outline, randomly generate N points on the two-dimensional plane, and generate the endpoints of N-1 line segments / curves of different types. Connect the different line segments / curves end to end to form a complete curve.

[0035] Determine the line segment / curve category, randomly select the line segment / curve category between two consecutive points, and randomly select the parameters of the line segment / curve;

[0036] Based on the line segment / curve category and parameters, construct the target equation between the two corresponding corner points. Select multiple points uniformly and discretely on the line segment / curve segment expressed by the target equation, and arrange the selected points and corner points in sequence to form a sequence of key points.

[0037] Noise is added to the coordinate sequence of key points to increase the bias, thus generating the coordinate sequence of key points;

[0038] Label the key point sequence. If the key point is a selected corner point, label the key point as a corner point. If the key point is a discretely selected point on the generated line segment / curve, label the key point as the line segment / curve type label.

[0039] The coordinate sequence of the above key points and the label of each key point together form a curve sample. Repeat the above steps to generate a certain number of curve sample data.

[0040] The label types include corner points, line segment key points, arc key points, and B-spline curve key points.

[0041] Preferably, the training steps for the first neural network model and the second neural network model include:

[0042] The training dataset is obtained and preprocessed. The preprocessing method is to obtain the coordinates of the key points in the x-axis direction and y-axis direction, normalize them, calculate the curvature feature and tangent angle feature of each key point, and combine the feature values ​​of the key points with the coordinates of the key points as the features of the input sequence.

[0043] The dataset is divided into a training set, a validation set, and a test set according to a preset ratio;

[0044] The training set is input into the first neural network model and the second neural network model respectively to train the model and obtain the trained first neural network model and the second neural network model.

[0045] The trained model was validated using a test set and a validation set, respectively, to obtain the optimized first neural network model / second neural network model.

[0046] Preferably, the second neural network model training step further includes:

[0047] The dataset is segmented into line segments based on whether the key point category is a corner point, and the line segment categories are encoded using one-hot encoding to obtain the training data for the second neural network model.

[0048] Fill the sequence of key points of line segments in the dataset with zero values ​​to a preset length;

[0049] Using cross-entropy as the loss function, the model parameters are updated by gradient descent based on the difference between the model's predictions and the true labels on the training dataset, resulting in the model that performs best on the validation dataset.

[0050] Input a test set and evaluate the model's prediction results based on the mean squared error. Further optimize the model based on the prediction results.

[0051] The present invention also provides a contour line recognition device based on a bidirectional long short-term memory neural network, comprising:

[0052] The key point extraction module is used to obtain the outline of the target image and extract key points on the outline line from the target image based on the key point extraction algorithm.

[0053] The corner detection module is used to acquire the feature sequence of key points and input it into the trained first neural network model to detect corner points, and to identify corner information from the feature sequence of variable-length key points that make up contours of different shapes.

[0054] The line segment segmentation module is used to segment the contour line according to the corner point information, and to segment the line segments / curves on both sides adjacent to the corner point to obtain a point set sequence of line segments / curves after the contour line is segmented.

[0055] The line segment recognition module is used to acquire the point set sequence of line segments / curves and input it into the trained second neural network model to identify the type of line segment / curve and obtain the recognition result of the contour line.

[0056] The present invention also provides an electronic device, comprising:

[0057] The memory is used to store the processing program;

[0058] The processor, when executing the processing program, implements the contour line recognition method based on a bidirectional long short-term memory neural network as described in the embodiments of the present invention.

[0059] Compared with the prior art, the present invention has the following beneficial effects:

[0060] This invention provides a contour recognition method based on a bidirectional long short-term memory neural network. Combining deep learning and traditional contour recognition methods, it proposes a fast contour recognition method based on a bidirectional long short-term memory network (Bi-LSTM) to solve the problems of existing contour recognition algorithms that rely on manual parameter adjustment, have high computational complexity, and have unsatisfactory recognition results. At the same time, it provides a new approach that combines deep learning and traditional methods, avoiding the disadvantage of deep learning being difficult to train in scenarios with limited data.

[0061] This invention uses a neural network model to identify corner points from a sequence of variable-length key points that make up a shape's contour, thereby achieving rapid contour line identification without the need for manual feature extraction by experts. The model's training samples are generated through a designed data generation algorithm, thus avoiding the problems of limited data volume and high data annotation costs in industrial scenarios. Attached Figure Description

[0062] Figure 1 This is a flowchart illustrating the contour line recognition method based on a bidirectional long short-term memory neural network according to an embodiment of the present invention.

[0063] Figure 2 This is a schematic diagram of the overall steps of the contour line recognition method based on a bidirectional long short-term memory neural network according to an embodiment of the present invention;

[0064] Figure 3 This is a backbone network structure diagram in one embodiment of the present invention;

[0065] Figure 4 This is a schematic diagram of the structure of the first neural network model in one embodiment of the present invention;

[0066] Figure 5 This is a schematic diagram of the structure of the second neural network model in one embodiment of the present invention;

[0067] Figure 6 This is a schematic diagram illustrating the training principle of the first neural network model in one embodiment of the present invention. Detailed Implementation

[0068] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0069] The term "comprising" and its variations as used herein are open-ended inclusion, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below.

[0070] It should be noted that the concepts of "first" and "second" mentioned in this application are only used to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0071] It should be noted that the terms "a" and "a plurality of" used in this application disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0072] Example 1

[0073] like Figure 1 As shown, this invention provides a contour line recognition method based on a bidirectional long short-term memory neural network, comprising:

[0074] S1: Obtain the outline of the target image, and extract key points on the outline line from the target image based on the key point extraction algorithm; the obtained target image can be an actual image or an image set generated by a data generation algorithm. This image set is the training data generated by randomly generating outline lines as required for model training in this invention; the key point extraction algorithm used in this embodiment is implemented by existing technology.

[0075] S2: Obtain the feature sequence of key points and input it into the trained first neural network model for corner detection. Identify corner information from the feature sequence of variable-length key points that make up contours of different shapes. The corner information is the probability that each corresponding key point is predicted to be a corner. The feature sequence of the key point includes x / y coordinates, curvature, and tangent angle. A corner is a dividing point connecting two different line segments / curves. The first neural network model used in this embodiment is used for line segment / curve segmentation.

[0076] S3: Segment the contour line according to the corner point information, and segment the line segments / curves on both sides adjacent to the corner point to obtain a point set sequence of line segments / curves after contour line segmentation;

[0077] S4: Obtain the point set sequence of line segments / curves and input it into the trained second neural network model for line segment / curve type recognition to obtain the contour line recognition result. The second neural network model used in this embodiment is used for line segment / curve category recognition. In this embodiment, the inputs of the two neural network models are consistent in feature dimensions, namely x / y coordinates, curvature, and tangent angle.

[0078] See Figure 2 As shown, in this embodiment, for the input image, the key points on the contour line to be identified are first extracted from the image using a traditional key point extraction algorithm. Then, the coordinate sequence of the key points is used as input, and a neural network model consisting of a Convolutional Neural Network (CNN) and a Bidirectional Long Short-Term Memory Network (Bi-LSTM) is used to identify the corner points. The point set sequence that forms the line segments / curves after segmentation by corner points is then used as input to a neural network model for identifying the type of line segments / curves, such as straight lines and arcs. This embodiment uses a neural network model to identify corner points from the variable-length key point sequence that makes up the contour, thus achieving rapid contour line identification without the need for manual feature extraction by experts. The training samples for the model are generated using a designed data generation algorithm, thereby avoiding the problems of limited data volume and high data annotation costs in industrial scenarios.

[0079] The key components of this invention are the first neural network model, namely contour line segmentation, also known as the corner detection model, and the second neural network model, namely the line segment category recognition model. Both models are neural network models and share a portion of their network structure. The two neural network models share a backbone network structure, which includes the following components (see details). Figure 3 As shown. Specifically, both the first neural network model and the second neural network model integrate convolutional neural networks and bidirectional long short-term memory networks. The first neural network model is a line segmentation model, and the second neural network model is a line segment category recognition model. The first neural network model and the second neural network model share a backbone network structure, which includes two or more convolutional units and bidirectional LSTM units. The convolutional units include a first convolutional unit and a second convolutional unit, which are connected to each other and used to take the keypoint sequence as low-level feature input to extract high-level features. The bidirectional LSTM unit includes a first LSTM unit and a second LSTM unit connected to the convolutional units respectively, and is used to extract features of the keypoint sequence from both forward and backward directions, so that the backbone network can extract features covering the entire keypoint sequence.

[0080] The convolutional unit used in this embodiment is a convolutional layer (CNN): the input keypoint sequence only contains low-dimensional information such as coordinates and curvature, and the convolutional layer can extract higher-dimensional features based on local information. Initially, the four features input to the convolutional unit are: x and y coordinates, curvature, and tangent angle. However, in the backbone network, the number of features in each layer gradually increases with the network depth. The bidirectional LSTM unit used in this embodiment is a bidirectional long short-term memory network (Bi-LSTM): LSTM networks have memory units that can remember previous sequence information and, combined with the current input, output results associated with the previous sequence information. Keypoints on the contour line have significant sequence relationships, and the sequence length is variable. Using the LSTM network structure can fully utilize the information of long sequences. There are certain sequence relationships between keypoints on the contour line, and for keypoints on the contour line, reversing the sequence order does not change the information contained in the keypoints. The bidirectional LSTM network can extract features from the keypoint sequence in both forward and backward directions, enabling the backbone network to extract features covering the entire keypoint sequence.

[0081] In several embodiments of this application, the first neural network model includes a backbone network unit, a third convolutional unit, and a point-by-point local connection unit connected in sequence. The feature sequence of key points of the same contour is input from the input layer, and the low-level feature is output from the input layer as the input of the backbone network unit. The high-level feature is output from the backbone network unit and used as the input of the third convolutional unit. The corresponding high-level feature is output from the third convolutional unit and used as the input of the point-by-point local connection unit. When the input feature sequence is of variable length, the output layer finally outputs the prediction result of whether each input key point in the feature sequence of arbitrary length is a corner point.

[0082] Those skilled in the art will understand that the first neural network model used in this embodiment, namely the contour line segmentation model, consists of a backbone network structure and additional convolutional layers and local connection layers, the structure of which is described in [reference needed]. Figure 4 As shown, the network input is a sequence of features from key points belonging to the same contour, such as x / y coordinates, curvature, tangent angle, and P. i =(x i y i c i a i ), i = 1…n, the network output is a sequence of the same length as the input, where each element represents the probability (p) of whether the point corresponding to the input is a corner point. i (i = 1…n). The contour segmentation model adds a convolutional layer after the backbone network, which outputs higher-dimensional features based on local features. The output of the convolutional layer serves as the input to a point-wise local connection layer, which outputs the probability of whether a point is a corner point based on its high-dimensional features. Using point-wise local connections also allows the network's output layer to output the probability of whether each keypoint corresponding to each input is a corner point. The entire network can predict the probability of every point in a sequence of arbitrary length without using zero-value padding.

[0083] In several embodiments of this application, the second neural network model includes a backbone network unit, a fourth convolutional unit, a first fully connected unit, and a second fully connected unit connected in sequence. The feature sequence of key points on the same line segment / curve after contour line segmentation is input from the input layer. This low-level feature is output from the input layer as input to the backbone network unit. The backbone network unit outputs high-level features as input to the fourth convolutional unit. Based on the high-level features of each key point, the input feature sequence is padded to a fixed length using the fourth convolutional unit, the first fully connected unit, and the second fully connected unit, even when the input feature sequence is of variable length. Finally, the output layer outputs the predicted probability of the line segment / curve being classified into different categories.

[0084] Those skilled in the art will understand that the second neural network model used in this embodiment, namely the line segment recognition model, also includes a backbone network, the structure of which can be found in [reference needed]. Figure 5 As shown. The network input is the feature sequence of key points belonging to the same line segment after corner points are identified by the contour line segmentation model, such as x / y coordinates, curvature, and tangent angle. P i =(x i y i c i a i For each line segment, i = 1…n, the network output is the probability that a single line segment is classified into different line segment categories, i.e., p. j , j = 1…m, where m represents the number of categories of the curve, satisfying That is, the sum of the probabilities of a line segment being classified into different categories is 1. Since the prediction layer of this network uses a fully connected layer, the network has a fixed input sequence length. This invention uses padding to fill the input sequence to a fixed length, thereby enabling the network to handle input sequences of variable length.

[0085] In various embodiments of this application, step S2, which involves obtaining the feature sequence of key points and inputting it into the trained first neural network model for corner detection, further includes:

[0086] In the first neural network model after training, the input is a feature sequence of key points belonging to the same contour, and the output is the probability value of each key point being predicted as a corner point. The feature sequence includes the coordinates of the key point in the x-axis direction, the coordinates in the y-axis direction, the curvature, and the tangent angle. If the output probability value is greater than or equal to a preset threshold, the key point is determined to be a corner point. The preset threshold is such as 0.5. If the output probability value is less than the preset threshold, the key point is determined to be a non-corner point. After traversing all key points to obtain the category of corner point or non-corner point, the corner point position in the contour line is cut off to obtain multiple line segments / curves with the corner point as the endpoint, thus completing the contour line segmentation.

[0087] Those skilled in the art will understand that the principle of the first neural network model used in this embodiment for segmenting line segments is as follows: The input is a feature sequence of key points belonging to the same contour; for each point, the network outputs a probability value p. i , representing the corresponding key point input P i The probability of a point being predicted as a corner point ranges from [0,1]. If the output probability value is greater than 0.5, the point is considered a corner point; if the output probability value is less than 0.5, the point is considered a key point within the line segment, i.e., not a corner point. After obtaining the categories of all key points, the outline is cut at the corner point position to obtain multiple line segments with the corner point as the endpoint, thus completing the outline segmentation.

[0088] In several embodiments of this application, step S4, which involves obtaining a sequence of point sets of line segments / curves and inputting it into a trained second neural network model for line segment / curve type recognition, includes:

[0089] The second neural network model after training inputs the feature sequence of key points of line segment / curve and outputs the first probability p1, the second probability p2 and the third probability p3 of the line segment / curve being classified into different categories, including straight line segments, circular arcs and B splines.

[0090] If the first probability p1 > the second probability p2 and the first probability p1 > the third probability p3, then the network predicts that the line segment / curve is a straight line segment.

[0091] If the second probability p2 > the first probability p1 and the second probability p2 > the third probability p3, then the network predicts that the line segment / curve is an arc.

[0092] If the third probability p3 > the first probability p1 and the third probability p3 > the second probability p2, then the network predicts that the category of the line segment / curve is B-spline;

[0093] The probability that a line segment / curve is classified into different categories satisfies the following condition: m represents the number of categories of line segments / curves.

[0094] Those skilled in the art will understand that the principle behind the second neural network model used in this embodiment for predicting line segment categories is as follows: When predicting the category of a line segment, the input feature sequence of key points of the line segment can yield the probability that the line segment is classified into different categories. For example, if there are three categories: straight line segment, arc, and B-spline, the network will output three probability values, p1, p2, and p3, and the sum of the three probabilities is 1. If p1 is the largest of the three probabilities, the network predicts that the category of the line segment is a straight line segment.

[0095] The neural network model used in this embodiment requires a large amount of labeled data for training. However, in industrial scenarios, the amount of data is limited, and data labeling is costly. Therefore, this embodiment designs a data generation algorithm to generate training data by randomly generating contour lines. The specific process of the data generation algorithm is as follows:

[0096] Determine the corner points on the outline, randomly generate N points on the two-dimensional plane, and generate the endpoints of N-1 line segments / curves of different types. Connect the different line segments / curves end to end to form a complete curve.

[0097] The line segment / curve category is determined, and the line segment / curve category between two consecutive points is randomly selected, along with the parameters of the line segment / curve. In this embodiment, the line segment / curve category between two consecutive points is randomly selected, such as a straight line segment, an arc, or a B-spline curve, and the parameters of the line segment / curve are randomly selected, including the number of key points common to all line segment / curve types, as well as parameters specific to the line segment / curve type, such as the center position of an arc, the number / position of control points in a B-spline, etc.

[0098] Based on the line segment / curve category and parameters, a target equation is constructed between the corresponding two corner points. Multiple points are uniformly and discretely selected on the line segment / curve segment expressed by the target equation, and the selected points and corner points are arranged in sequence to form a sequence of key points. In this embodiment, the target equation can be a straight line equation, a curve equation, etc. If the line segment / curve category is a straight line, the two endpoints are directly determined, and the constructed target equation is a straight line equation y = kx + b, where x and y represent the abscissa and ordinate of a point on the straight line, respectively. Determining the endpoints allows direct calculation of the values ​​of k and b. If the line segment / curve category is an arc, a point is selected on the perpendicular bisector of the two endpoints as the center of the circle, and the arc can be directly drawn, resulting in a curve equation. If the line segment / curve category is a B-spline, 3-5 control points are determined in the plane, and the curve can be directly generated through the B-spline equation, and the curve equation can be constructed.

[0099] Noise is added to the coordinate sequence of key points to increase the deviation, thereby generating the coordinate sequence of key points. In this embodiment, generating the coordinate sequence of key points means extracting the horizontal and vertical coordinates of the key points. The added noise includes Gaussian noise, which is used to make the horizontal and vertical coordinates shift relative to the actual position. Key points are randomly removed to remove some points in the sequence. Key points are randomly added by randomly selecting two key points and linearly inserting a key point between them.

[0100] Label the key point sequence. If the key point is a selected corner point, then the key point type is labeled as a corner point. If the key point is a discretely selected point on the generated line segment / curve, then the key point label is set as the line segment / curve type label. For example, if three key points are selected on a line segment, then these key points are all marked as line segment key points, while the endpoints at both ends of the line segment are marked as corner points.

[0101] The coordinate sequence of the above key points and the label of each key point together form a curve sample. Repeat the above steps to generate a certain number of curve sample data.

[0102] The label types include: corner points, line segment key points, arc key points, and B-spline curve key points.

[0103] In this embodiment, since the two networks reuse the backbone network structure, and the goal of the second neural network model, i.e., the line segment category recognition model, is to output the category of the line segment, the backbone network part of this network can extract features with a high correlation to the line segment category. The first neural network model, i.e., the line segment segmentation model, can identify corner points based on the line segment category-related features extracted by this network. Based on this, this embodiment of the invention designs a transfer learning training method, and the transfer learning training process is as follows:

[0104] 1. Data Preprocessing: Obtain the training dataset and perform data preprocessing. This preprocessing involves normalizing the x-axis and y-axis coordinates of keypoints, calculating the curvature and tangent angle features of each keypoint, and combining these features with the keypoint coordinates as the features of the input sequence. This can be understood as normalizing the (x,y) coordinates of the training data to the (0,1] interval. The curvature and tangent angle of each keypoint are calculated, and these two types of feature values, along with the keypoint coordinates, are used as the features of the input sequence.

[0105] 2. Dataset Splitting: Split the dataset into a training set, a validation set, and a test set according to a preset ratio; the preset ratio is, for example, 8:1:1.

[0106] 3. Training process of the first neural network model / second neural network model: Input the training set into the first neural network model / second neural network model respectively to train the model and obtain the trained first neural network model / second neural network model; use the test set and validation set respectively to validate the trained model to obtain the optimized first neural network model / second neural network model.

[0107] 4. Model Parameter Tuning: Because the dataset used for model training is randomly generated, its data distribution may differ from that of data in industrial scenarios. Therefore, after model training, this invention uses a dataset from an industrial scenario to tune the model's parameters to improve its performance in industrial environments. The model parameter tuning process is as follows:

[0108] a) Select a small amount of data from industrial scenarios and use a contour key point extraction algorithm to extract key points to obtain a key point sequence.

[0109] b) Add data labels to the keypoint sequence. Label types include: corner points, line segment keypoints, arc keypoints, B-spline curve keypoints, etc.

[0110] c) Following steps 2-4, train both models using a dataset from an industrial scenario, setting a small learning rate and number of training epochs to fine-tune the model parameters.

[0111] d) Evaluate the two models using an industrial dataset to assess their performance in an industrial setting.

[0112] The training process for the second neural network model used in this embodiment for line segment category recognition is as follows:

[0113] a) Segment the dataset into line segments according to whether the key point category is a corner point, and encode the line segment category one-hot to obtain the training data for the second neural network model, which is the training data required for line segment category recognition.

[0114] b) Input sequence padding: Fill the sequence of key points of line segments in the dataset with zero values ​​to the preset length;

[0115] c) Model training: Cross-entropy is used as the loss function. Based on the difference between the model's prediction results on the training dataset and the true labels, gradient descent is used to update the model parameters to obtain the model that performs best on the validation dataset.

[0116] d) Model evaluation: Input the test set and evaluate the model's prediction results based on the mean squared error (MSE). Further optimize the model based on the prediction results.

[0117] The training process for the first neural network model used in this embodiment for line segment segmentation is as follows: This model is based on a transfer learning method: the parameters of the backbone network portion of the second neural network model used to extract specific line segment features are directly transferred to the backbone network of the first neural network model, and a smaller learning rate is set for the backbone network during training. The transfer learning used in this example aims to leverage the similarity between data, tasks, or models to transfer knowledge learned in an old domain to a new domain. Transfer learning involves using the network structure of a successfully trained model, or a portion thereof, for training in a new scenario, thereby reducing training time and improving model performance.

[0118] In this embodiment, the first neural network model needs to predict the probability of whether a point is a corner point based on the contextual feature information of the keypoint sequence. However, it is quite difficult to train the model so that the backbone network can accurately extract this information. (See below.) Figure 6As shown, this invention employs a transfer learning method similar to that used in object detection. In object detection tasks, models are typically trained first through image classification tasks, enabling the backbone network to extract high-dimensional features for each image category. Then, the backbone network structure and its model parameters are directly used in the object detection model. The line segmentation model has the exact same backbone network structure as the line segment classification model. After training the line segment classification model, the backbone network can extract high-dimensional features related to the line segment category. This reduces the training overhead of the line segmentation network while improving prediction accuracy. Specifically, each keypoint in the input dataset is connected by a line segment, with the diameter of the keypoint slightly larger than the line width. Figure 6 The points in the corner recognition results are the identified corner points, with other key points within the line segments removed.

[0119] Example 2

[0120] Based on the same concept, the present invention provides a contour line recognition device based on a bidirectional long short-term memory neural network, comprising:

[0121] The key point extraction module is used to obtain the outline of the target image and extract key points on the outline line from the target image based on the key point extraction algorithm.

[0122] The corner detection module is used to acquire the feature sequence of key points and input it into the trained first neural network model to detect corner points, and to identify corner information from the feature sequence of variable-length key points that make up contours of different shapes.

[0123] The line segment segmentation module is used to segment the contour line according to the corner point information, and to segment the line segments / curves on both sides adjacent to the corner point to obtain a point set sequence of line segments / curves after the contour line is segmented.

[0124] The line segment recognition module is used to acquire the point set sequence of line segments / curves and input it into the trained second neural network model to identify the type of line segment / curve and obtain the recognition result of the contour line.

[0125] The implementation principles of the aforementioned key point extraction module, corner detection module, line segmentation module, and line segment recognition module have been described in the previous embodiments, and therefore will not be repeated here.

[0126] Example 3

[0127] Based on the same concept, an electronic device is also provided in some embodiments of this application. This electronic device includes a memory and a processor, wherein the memory stores a processing program, and the processor executes the processing program according to instructions. When the processor executes the processing program, the contour line recognition method based on a bidirectional long short-term memory neural network described in the foregoing embodiments is implemented.

[0128] In some embodiments of this application, a readable storage medium is also provided, which can be a non-volatile readable storage medium or a volatile readable storage medium. The readable storage medium stores instructions that, when executed on a computer, cause an electronic device containing such a readable storage medium to perform the aforementioned contour line recognition method based on a bidirectional long short-term memory neural network.

[0129] It is understood that, for the aforementioned contour recognition methods based on bidirectional long short-term memory neural networks, if they are all implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes: USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, optical disks, and other media capable of storing program code.

[0130] Computer-readable storage media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable storage medium may also be any readable medium other than a readable storage medium that can transmit, propagate, or transfer a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0131] The program code for executing the technical solutions disclosed in this application can be written in any combination of one or more programming languages. These programming languages ​​include object-oriented programming languages—such as Java and C++—and conventional procedural programming languages—such as C or similar languages. The program code can be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0132] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A contour line recognition method based on a bidirectional long short-term memory neural network, characterized in that, include: Obtain the outline of the target image, and extract key points on the outline line from the target image based on the key point extraction algorithm; The feature sequences of key points are obtained and input into the trained first neural network model for corner detection, and corner information is identified from the feature sequences of variable-length key points that make up contours of different shapes. The contour line is segmented according to the corner point information, and the line segments / curves on both sides adjacent to the corner point are segmented to obtain a point set sequence of line segments / curves after the contour line is segmented. Obtain the point set sequence of line segments / curves and input it into the trained second neural network model to identify the type of line segments / curves, and obtain the recognition result of the contour line; Both the first neural network model and the second neural network model integrate convolutional neural networks and bidirectional long short-term memory networks, and the first neural network model is a line segmentation model, while the second neural network model is a line segment category recognition model. The first neural network model and the second neural network model share a backbone network structure, which includes two or more convolutional units and bidirectional LSTM units. The convolutional unit includes a first convolutional unit and a second convolutional unit, which are connected to each other to take the key point sequence as low-level feature input and extract high-level features. The bidirectional LSTM unit includes a first LSTM unit and a second LSTM unit respectively connected to the convolutional unit, used to extract features of the keypoint sequence from both forward and backward directions, so that the backbone network can extract features covering the entire keypoint sequence.

2. The contour line recognition method based on a bidirectional long short-term memory neural network according to claim 1, characterized in that, The step of obtaining the feature sequence of key points and inputting it into the trained first neural network model for corner detection further includes: In the first neural network model after training, the input is a feature sequence of key points belonging to the same contour, and the output is the probability value of each key point being predicted as a corner point. The feature sequence includes the coordinates of the key point in the x-axis direction, the coordinates in the y-axis direction, the curvature, and the tangent angle. If the output probability value is greater than or equal to the preset threshold, then the key point is determined to be a corner point; If the output probability value is less than the preset threshold, the key point is determined to be a non-corner point. After traversing all key points to obtain the categories of corner points or non-corner points, the corner points in the contour line are cut off to obtain multiple line segments / curves with the corner points as endpoints, thus completing the segmentation of the contour line.

3. The contour line recognition method based on a bidirectional long short-term memory neural network according to claim 1, characterized in that, The step of obtaining the point set sequence of line segments / curves and inputting it into the trained second neural network model for line segment / curve type recognition includes: The second neural network model after training inputs the feature sequence of key points of line segment / curve and outputs the first probability, second probability and third probability of the line segment / curve being classified into different categories, including straight line segment, circular arc and B spline; If the first probability is greater than the second probability and the first probability is greater than the third probability, then the network predicts that the line segment / curve is a straight line segment. If the second probability is greater than the first probability and the second probability is greater than the third probability, then the network predicts that the line segment / curve is an arc. If the third probability is greater than the first probability and the third probability is greater than the second probability, then the network predicts that the category of the line segment / curve is B-spline; The probability that a line segment / curve is classified into different categories satisfies the following condition: , where m represents the number of categories of line segments / curves.

4. The contour line recognition method based on a bidirectional long short-term memory neural network according to claim 1, characterized in that, The first neural network model includes a backbone network unit, a third convolutional unit, and a point-by-point local connection unit connected in sequence. The feature sequence of key points of the same contour is input from the input layer, and the low-level feature is output from the input layer as the input of the backbone network unit. The high-level feature is output by the backbone network unit and used as the input of the third convolutional unit. The corresponding high-level feature is output by the third convolutional unit and used as the input of the point-by-point local connection unit. When the input feature sequence is of variable length, the output layer finally outputs the prediction result of whether each input key point in the feature sequence of arbitrary length is a corner point. The second neural network model includes a backbone network unit, a fourth convolutional unit, a first fully connected unit, and a second fully connected unit connected in sequence. The feature sequence of key points on the same line segment / curve after contour line segmentation is input from the input layer. This low-level feature is output from the input layer and used as input to the backbone network unit. The backbone network unit outputs high-level features, which are then used as input to the fourth convolutional unit. Based on the high-level features of each key point, the input feature sequence is padded to a fixed length using the fourth convolutional unit, the first fully connected unit, and the second fully connected unit, even when the input feature sequence is of variable length. Finally, the output layer outputs the predicted probability of the line segment / curve being classified into different categories.

5. The contour line recognition method based on a bidirectional long short-term memory neural network according to claim 1, characterized in that, The process of obtaining the outline of the target image includes: Determine the corner points on the contour line, randomly generate N points on the two-dimensional plane, and generate N-1 line segments / curves of different types. Connect the different line segments / curves end to end to form a complete curve. Determine the line segment / curve category, randomly select the line segment / curve category between two consecutive points, and randomly select the parameters of the line segment / curve; Based on the line segment / curve category and parameters, construct the target equation between the two corresponding corner points. Select multiple points uniformly and discretely on the line segment / curve segment expressed by the target equation, and arrange the selected points and corner points in sequence to form a sequence of key points. Noise is added to the coordinate sequence of key points to increase the bias, thus generating the coordinate sequence of key points; Label the key point sequence. If the key point is a selected corner point, label the key point as a corner point. If the key point is a discretely selected point on the generated line segment / curve, label the key point as the line segment / curve type label. The coordinate sequence of the above key points and the label of each key point together form a curve sample. Repeat the above steps to generate a certain number of curve sample data. The label types include: corner points, line segment key points, arc key points, and B-spline curve key points.

6. The contour line recognition method based on a bidirectional long short-term memory neural network according to claim 1, characterized in that, The training steps for the first neural network model and the second neural network model include: The training dataset is obtained and preprocessed. The preprocessing method is to obtain the coordinates of the key points in the x-axis direction and y-axis direction, normalize them, calculate the curvature feature and tangent angle feature of each key point, and combine the feature values ​​of the key points with the coordinates of the key points as the features of the input sequence. The dataset is divided into a training set, a validation set, and a test set according to a preset ratio; The training set is input into the first neural network model and the second neural network model respectively to train the model and obtain the trained first neural network model and the second neural network model. The trained model was validated using a test set and a validation set, respectively, to obtain the optimized first neural network model / second neural network model.

7. The contour line recognition method based on a bidirectional long short-term memory neural network according to claim 1, characterized in that, The second neural network model training step further includes: The dataset is segmented into line segments based on whether the key point category is a corner point, and the line segment categories are encoded using one-hot encoding to obtain the training data for the second neural network model. Fill the sequence of key points of line segments in the dataset with zero values ​​to a preset length; Using cross-entropy as the loss function, the model parameters are updated by gradient descent based on the difference between the model's predictions and the true labels on the training dataset, resulting in the model that performs best on the validation dataset. Input a test set and evaluate the model's prediction results based on the mean squared error. Further optimize the model based on the prediction results.

8. A contour line recognition device based on a bidirectional long short-term memory neural network, characterized in that, include: The key point extraction module is used to obtain the outline of the target image and extract key points on the outline line from the target image based on the key point extraction algorithm. The corner detection module is used to acquire the feature sequence of key points and input it into the trained first neural network model to detect corner points, and to identify corner information from the feature sequence of variable-length key points that make up contours of different shapes. The line segment segmentation module is used to segment the contour line according to the corner point information, and to segment the line segments / curves on both sides adjacent to the corner point to obtain a point set sequence of line segments / curves after the contour line is segmented. The line segment recognition module is used to acquire the point set sequence of line segments / curves and input it into the trained second neural network model to identify the type of line segment / curve and obtain the recognition result of the contour line. Both the first neural network model and the second neural network model integrate convolutional neural networks and bidirectional long short-term memory networks, and the first neural network model is a line segmentation model, while the second neural network model is a line segment category recognition model. The first neural network model and the second neural network model share a backbone network structure, which includes two or more convolutional units and bidirectional LSTM units. The convolutional unit includes a first convolutional unit and a second convolutional unit, which are connected to each other to take the key point sequence as low-level feature input and extract high-level features. The bidirectional LSTM unit includes a first LSTM unit and a second LSTM unit respectively connected to the convolutional unit, used to extract features of the keypoint sequence from both forward and backward directions, so that the backbone network can extract features covering the entire keypoint sequence.

9. An electronic device, characterized in that, include: The memory is used to store the processing program; A processor, which, when executing the processing program, implements the contour line recognition method based on a bidirectional long short-term memory neural network as described in any one of claims 1 to 7.

Images