Weld seam positioning method based on LSTM model for linear array camera scanning data

Abstract

The present invention relates to a weld positioning method for linear array camera scanning data based on LSTM model, comprising the following steps: scanning a workpiece for which weld position positioning needs to be performed with a linear array camera to obtain multiple lines of scanning data; annotating each line of scanning data obtained with a weld, and inputting the annotated data into a deep learning model for training to obtain a weld positioning model; collecting image data of the workpiece in real time through a linear array camera, preprocessing it and inputting it into a weld positioning model for classification to determine whether it contains a weld area; after the classification is completed, data splicing is performed according to the classification result and the real-time sampling of the image data to form a classification result graph to locate the weld. In the present invention, since the linear array camera can collect sequence data in real time and the LSTM model is a sequence model, the real-time data of the line structure can be classified in real time, and the weld area and the non-weld area can be classified to complete the positioning of the weld.

Description

Technical Field

[0001] The present invention belongs to the field of 3D visual image processing, and specifically is a weld positioning method for linear array camera scanning data based on LSTM deep learning. Background Art

[0002] Welding technology is an indispensable part of advanced manufacturing. At present, a considerable part of high-precision welding process requires manual positioning of the robotic arm. Vision-based welding robots are an important development direction for this technology, and weld positioning is an important process in vision-based mechanical welding.

[0003] At present, the weld positioning method adopts a combination of two-dimensional images and deep learning. Due to the limitation that two-dimensional images are easily disturbed during acquisition, their applicability is not wide. Weld positioning based on workpiece point cloud information is difficult to apply to real-time detection because point cloud computing requires a large amount of calculation and time. The template matching based on structured light features and two-dimensional image features is often not as good as the features learned by deep learning, resulting in low image processing accuracy, poor algorithm generalization ability, and low robustness for weld recognition and positioning. Summary of the invention

[0004] The purpose of the present invention is to provide a weld location method based on LSTM deep learning of linear array camera scanning data. Aiming at the industrial weld identification scene, the present invention combines the deep learning sequence classification model with the sequence data collected in real time by the linear array camera, and proposes a new method for real-time weld extraction based on the LSTM (long short term memory) model combined with the linear array camera scanning data. The method has the advantages of fast speed, simple application, good effect, etc., and can be used to obtain real-time weld position information after the workpiece is welded in real industrial scenes.

[0005] The technical solution adopted by the present invention to achieve the above-mentioned purpose is: a weld positioning method for linear array camera scanning data based on LSTM deep learning, comprising the following steps:

[0006] The workpiece that needs to be welded is scanned by a linear array camera to obtain multiple lines of scanning data;

[0007] Label each line of scanned data to identify the weld, and input the labeled data into a deep learning model for training to obtain a weld positioning model.

[0008] The image data of the workpiece is collected in real time by a linear array camera, pre-processed and input into the weld positioning model for classification to determine whether there is a weld area;

[0009] After the classification is completed, data splicing is performed based on the classification results and the real-time sampling of the image data to form a classification result map to locate the weld.

[0010] The workpiece that needs to be welded is scanned by a linear array camera to obtain multiple lines of scanning data, specifically as follows:

[0011] The linear array camera moves along the scanning direction at a fixed height, irradiates the laser vertically onto the surface of the workpiece, and obtains multiple lines of scanning data; the scanning data is a depth map; the value of each pixel in the depth map is the height information between the corresponding position of the pixel on the workpiece surface and the camera.

[0012] The method of labeling the weld seam of each row of scanned data obtained and inputting the labeled data into the deep learning model for training includes the following steps:

[0013] Serialize the multiple lines of scan data of the linear array camera: sort each line of scan data in sequence to form a sequence, and mark each line of scan data in the sequence to indicate whether it is a weld area;

[0014] Each row of data and annotations in the sequence are intercepted and serialized: for the i-th row of scan data, the i-th to i+n-th row of scan data are selected to intercept and form sample i for training; where i is 1...N, N is the number of scan data rows, and n is the set value;

[0015] The samples are sequentially formed into a sample sequence, normalized, and input into a deep learning model for training. The preprocessing includes the following steps:

[0016] The collected image data is intercepted in real time with a fixed length, that is, for the i-th row of data, the i-th row to the i+n-th row of scan data are selected to form intercepted real-time sequence data;

[0017] Normalize the truncated sequence data.

[0018] After the classification is completed, data splicing is performed according to the classification result and the real-time sampling of the image data to form a classification result graph, including the following steps:

[0019] After completing the sequence data classification, a result sequence set is obtained. According to the classification order of the sequence data and the real-time sampling of the data, the result sequence set is spliced ​​to obtain a classification result map, so as to determine the weld position according to the classification result map.

[0020] The deep learning model is an LSTM model.

[0021] The weld positioning system based on LSTM deep learning of linear array camera scanning data includes:

[0022] A scanning data acquisition module is used to scan a workpiece that needs to be welded by a linear array camera to obtain multiple lines of scanning data;

[0023] A weld positioning model building module is used to annotate each line of scan data obtained as a weld, and input the annotated data into a deep learning model for training to obtain a weld positioning model;

[0024] The weld seam identification and classification module is used to collect image data of the workpiece in real time through a linear array camera, perform preprocessing and input it into the weld seam positioning model for classification to determine whether there is a weld seam area;

[0025] The classification result display module, after completing the classification, performs data splicing based on the classification results and the real-time sampling of the image data to form a classification result map to locate the weld.

[0026] The present invention has the following beneficial effects and advantages:

[0027] 1) The two-dimensional image information in industrial scenes is easily affected by other scene factors such as light and color, making it difficult to achieve ideal results. Linear array camera scanning is less affected by other factors and can be applied to industrial scenes.

[0028] 2) Since the linear array camera can collect sequence data in real time and the LSTM model is a sequence model, it can classify the real-time data in real time.

[0029] 3) The application of deep learning models is simple, and after completing model training, there is no need for complicated parameter adjustment in actual applications.

[0030] 4) The features used in the deep learning method adopted in the present invention are all learned by the model itself. The deep network can learn sufficiently complex high-level features, which are often more effective than the artificially designed features of the traditional method. In addition, the data-dimensional depth map data used in the present invention is less affected by the scene interference than the traditional 2D image. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 Provide a flow chart for data classification and location processing;

[0032] Figure 2 a is the data of the weld workpiece to be inspected collected by the linear array camera Figure 1 ;

[0033] Figure 2 b is the data of the weld workpiece to be inspected collected by the linear array camera Figure 2 ;

[0034] Figure 3 a is the area partition of the depth map Figure 1 ;

[0035] Figure 3 b is the area partition of the depth map Figure 2 ;

[0036] Figure 4 a is the result of the weld position obtained by LSTM model classification of data and visualized using Python Figure 1 ;

[0037] Figure 4 b is the result of the weld position obtained by LSTM model classification of data using Python visualization Figure 2 . DETAILED DESCRIPTION

[0038] The present invention is further described in detail below in conjunction with the accompanying drawings and embodiments.

[0039] Figure 1 This is the algorithm flow chart of the present invention, in which the entire process can be divided into five steps: data acquisition, model training, data preprocessing, model real-time classification, and data post-processing. The first two steps are pre-completed work, and the last three steps are real-time work.

[0040] Step 1: Get the line scan camera data

[0041] The workpiece that needs to be welded is scanned by a line array camera to obtain scanning data. The line array camera emits laser light to the surface of the workpiece, detects the reflection of the laser light on the object at a fixed angle, and then determines the height and width information of the object surface through the principle of triangulation. The image collected by the camera is a depth map, in which the value of each pixel in the image is the height information between this position in the object and the camera.

[0042] After completing step 1, the depth information map of the target is obtained.

[0043] Step 2: Training the model

[0044] The acquired scan data is used to mark the welds, and then the marked data is input into the LSTM model for parameter training. LSTM is a special RNN network that solves the problem of gradient diffusion in the RNN model and can learn long-term rules in the sequence.

[0045] The main steps of model training are:

[0046] 1) The imaged line array camera scanning data is serialized and the pre-collected line array camera scanning data is converted into a depth image. The annotation information is also based on the depth image. The depth image is annotated row by row to indicate whether it is a weld area.

[0047] 2) For training, the depth image and annotation information are serialized by row (20 rows are used as a fixed window length to perform fixed-step sliding window capture in the height direction of the image. The step length of the present invention is set to 1, that is, 20 rows of image information are captured each time as a sequence segment. After completing a sequence segment capture, the sliding window moves one step along the height direction of the image, and the sequence segment is captured again, and repeated until the edge of the image).

[0048] 3) Normalize the serialized data and input it into the LSTM model for training.

[0049] After completing step 2, you can get a trained classification model and a labeled data set (the data set has completed data normalization, that is, the data is projected into the range of 0-1 according to the mean and variance of the original data).

[0050] Step 3: Data preprocessing

[0051] After completing the model training, the camera needs to perform simple preprocessing after collecting the depth map data of the workpiece in real time before it can be input into the model for classification.

[0052] The specific steps are:

[0053] 1) The data is cut into fixed length in real time (the collected image information is cut into fixed length of 20 rows in real time. At this time, each row of image is each frame of data in the sequence segment, and each column of data corresponding to each frame of data is the feature of the input model of the frame of data) to meet the model's requirements for the sequence format.

[0054] 2) Normalize the sequence data after the interception.

[0055] After completing step three, the pre-processed real-time sequence data can be obtained.

[0056] Step 4: Real-time detection

[0057] The preprocessed data can be directly input into the LSTM model for classification to determine whether the sequence data contains a weld area.

[0058] After completing step 4, the sequence segment that has completed the detection can be obtained.

[0059] Step 5: Data post-processing

[0060] After completing the classification of each sequence, the result sequence set is spliced ​​and reorganized into a classification result graph according to the classification order of the sequence and the real-time sampling of the data. The starting and ending points of the weld in the depth map are determined based on the classification result graph and the specific position of the weld is output.

[0061] After completing step five, the positions of the starting and ending points of the weld in the depth map can be obtained.

[0062] Embodiment: The present invention can be applied to the welding seam grinding process in which robots are involved in reality. After the line laser device is installed on the robot arm, the line array camera scans the welding seam workpiece to obtain the depth map data set of the workpiece surface. Figure 2 a. Figure 2 As shown in b, here Figure 2 a. Figure 2 b is the depth map data obtained by scanning the dipper arm of the excavator with line structured light. The depth map data is processed by the patented algorithm to obtain the weld area and the position information of the weld in the camera coordinate system. Figure 4 a. Figure 4 b is the effect of the depth map data being processed by the patented algorithm. The weld position information under the camera system is transformed into the position matrix obtained by hand-eye calibration to the world coordinate system to obtain the position of the weld on the actual workpiece, which facilitates the robot to perform subsequent high-precision grinding and other tasks.

[0063] effect: Figure 3 a. Figure 3 b is a schematic diagram of the composition of each depth area collected. The horizontal black part in the middle is the weld area, which is the depth information collected based on the workpiece weld. This patented algorithm extracts the upper and lower position boundaries of the weld to facilitate subsequent polishing and grinding.

[0064] Effect: The original depth map does not contain information about the specific location of the weld. After the LSTM model serializes and classifies the depth data, the specific location information of the weld in the depth map is obtained. Figure 4 a. Figure 4 b Python is used to output the weld position information in the form of a picture compared with the original image, and the weld and non-weld areas can be clearly separated.

Claims

1. A weld positioning method for linear array camera scanning data based on LSTM model, characterized in that: The following steps are involved: The workpiece that needs to be welded is scanned by a linear array camera to obtain multiple lines of scanning data; Label each line of scanned data to identify the weld, and input the labeled data into a deep learning model for training to obtain a weld positioning model. The image data of the workpiece is collected in real time by a linear array camera, pre-processed and input into the weld positioning model for classification to determine whether there is a weld area; After the classification is completed, data splicing is performed based on the classification results and the real-time sampling of the image data to form a classification result map to locate the weld; The method of labeling the weld seam of each row of scanned data obtained and inputting the labeled data into the deep learning model for training includes the following steps: Serialize the multiple lines of scan data of the linear array camera: sort each line of scan data in sequence to form a sequence, and mark each line of scan data in the sequence to indicate whether it is a weld area; Each row of data and annotations in the sequence are intercepted and serialized: for the i-th row of scan data, the i-th to i+n-th row of scan data are selected to intercept and form sample i for training; where i is 1...N, N is the number of scan data rows, and n is the set value; After the samples are sequentially organized into a sample sequence, they are normalized and input into the deep learning model for training; The pre-processing comprises the following steps: The collected image data is intercepted in real time with a fixed length, that is, for the i-th row of data, the i-th row to the i+n-th row of scan data are selected to form intercepted real-time sequence data; Normalize the truncated sequence data.

2. The weld positioning method based on LSTM model for linear array camera scanning data according to claim 1 is characterized in that: The workpiece that needs to be welded is scanned by a linear array camera to obtain multiple lines of scanning data, specifically as follows: The linear array camera moves along the scanning direction at a fixed height, irradiates the laser vertically onto the surface of the workpiece, and obtains multiple lines of scanning data; the scanning data is a depth map; the value of each pixel in the depth map is the height information between the corresponding position of the pixel on the workpiece surface and the camera.

3. The weld positioning method based on LSTM model for linear array camera scanning data according to claim 1, characterized in that: After the classification is completed, data splicing is performed according to the classification result and the real-time sampling of the image data to form a classification result graph, including the following steps: After completing the sequence data classification, a result sequence set is obtained. According to the classification order of the sequence data and the real-time sampling of the data, the result sequence set is spliced ​​to obtain a classification result map, so as to determine the weld position according to the classification result map.

4. According to the weld positioning method based on LSTM model for linear array camera scanning data according to claim 1, the deep learning model is an LSTM model.

5. The positioning system of the weld positioning method based on the LSTM model for the linear array camera scanning data according to claim 1 is characterized in that: include: A scanning data acquisition module is used to scan a workpiece that needs to be welded by a linear array camera to obtain multiple lines of scanning data; A weld positioning model building module is used to annotate each line of scan data obtained as a weld, and input the annotated data into a deep learning model for training to obtain a weld positioning model; The weld seam identification and classification module is used to collect image data of the workpiece in real time through a linear array camera, perform preprocessing and input it into the weld seam positioning model for classification to determine whether there is a weld seam area; The classification result display module, after completing the classification, performs data splicing based on the classification results and the real-time sampling of the image data to form a classification result map to locate the weld.

Images