An Automatic Segmentation Method for Connector Profile Images Based on Deep Learning

Abstract

The present invention discloses an automatic segmentation method for the sectional images of connectors based on deep learning. For lightweight connection processes, sectional images of connectors are collected as sample images. Each sample image is labeled through a labeling tool to mark the positions of different plates, fasteners, and other regions of interest in the image, generating a mask image and a text file corresponding to the labeled sample image, and taking them together with the original image as training samples; the training samples are subjected to image enhancement processing, and corresponding mask images and text files are generated; a deep learning model based on image instance segmentation is selected or built; a training environment for the selected or built deep learning network model is set up; the training parameters of the deep learning network model are set; the training samples are added to the deep learning network model for training, and the resulting deep learning model after training is used for automatic segmentation of the sectional images of connectors.

Description

Technical Field

[0001] The present invention relates to the fields of image segmentation and computer vision, and particularly to an automatic segmentation method for the cross-sectional images of connectors based on deep learning. Background Art

[0002] Lightweighting has become a development trend in the machinery industry. Lightweight materials generally include light alloys, high-strength metal materials, engineering plastics, carbon fiber reinforced composite materials, and ceramic materials, etc. Considering cost, performance, and lightweighting effects comprehensively, the use of multi-material hybrid design is the most important lightweighting means adopted in the machinery industry. However, traditional process technologies are difficult to meet the hybrid connection between various materials, making the assembly of lightweight multi-material components face huge challenges. Currently, for the material and structural design of components, the mainstream lightweight connection processes include resistance spot welding (RSW) of aluminum alloy, self-piercing riveting (SPR), non-rivet self-piercing riveting (Clinching), flow drill screw (FDS), structural adhesive (Adhesive), etc. The cross-sectional images generated during different connection processes are important data for studying connection processes.

[0003] Image segmentation is a basic computer vision technology, an important link in automated image processing, and the basis for image analysis and understanding. From the initial traditional image segmentation algorithms to the current deep learning-based image segmentation algorithms, image segmentation has made great progress. Its goal is to classify each pixel in the image to understand the content in the image and make the analysis of each part easier.

[0004] In the field of multi-material connection, connection technology engineers need to manually count and analyze specific regions and key index parameters in the cross-sectional images one by one. The disadvantage of this method is that it is time-consuming and laborious, and is greatly affected by subjective factors, and cannot achieve a standardized process. Therefore, for a large number of connector cross-sectional images, if image segmentation can be performed quickly and accurately, and the true content represented by each pixel in the connector cross-sectional image can be understood through a deep learning model, it has important guiding significance for studying connection processes. Summary of the Invention

[0005] In order to solve the deficiencies existing in the prior art, the present application proposes an automatic segmentation method for the cross-sectional images of connectors based on deep learning, which uses deep learning algorithms and image enhancement techniques to perform image segmentation on the cross-sectional images generated during the material connection process, overcomes the disadvantages of the current engineers manually counting cross-sectional images, and provides technical support for realizing the automated, intelligent, and standardized analysis of connector cross-sectional images.

[0006] The technical solution adopted by the present invention is as follows:

[0007] An automatic segmentation method for the cross-sectional images of connectors based on deep learning, the method comprising the following steps:

[0008] Step 1, training set production: For the lightweight connection process, collect the cross-sectional images of the connectors as sample images, annotate each sample image through an annotation tool, mark the positions of different plates, fasteners and other areas of concern in the image, generate the corresponding mask image and text file for the annotated sample image, and use them together with the original Figure 1 as training samples;

[0009] Step 2, perform image enhancement processing on the training samples, and generate the corresponding mask image and text file;

[0010] Step 3, select or build a deep learning model based on image instance segmentation;

[0011] Step 4, build the training environment for the selected or built deep learning network model;

[0012] Step 5, set the training parameters of the deep learning network model;

[0013] Step 6, add the training samples to the selected deep learning network model in Step 3 for training, and after the training is completed, obtain a deep learning model applicable to the automatic segmentation of the cross-sectional images of the connectors;

[0014] Step 7, input the image to be detected into the deep learning network model trained in Step 6, the deep learning network model outputs a predicted mask image, and according to the segmentation effect of the instances in the mask image, repeat Step 5 and Step 6 for model optimization.

[0015] Furthermore, the lightweight connection process includes one or a combination of spot welding, self-piercing riveting, non-rivet self-piercing riveting, hot melt self-tapping screws, and structural adhesives.

[0016] Furthermore, the annotation tool for annotating the sample images in Step 1 selects one or a combination of Labelme, CVAT, VIA, PixlAnnotationTool, and EISeg.

[0017] Furthermore, the text file obtained in Step 1 is in the JSON or XML format, and is used to record the specific information of the plates and fasteners and the manual segmentation results of the regions formed after their connection.

[0018] Furthermore, the image enhancement tool in Step 2 selects one or a combination of OpenCV, Imgaug, Skimage, PIL, Augmentor, and Albumentations.

[0019] Furthermore, using the above enhancement tools, the image enhancement method for the image is one or a combination of translation, flipping, rotation, adding noise, blurring, sharpening, cropping, and scaling.

[0020] Furthermore, the deep learning framework in step 3 is selected from one of TensorFlow, Keras, PyTorch, Caffe, Theano, PaddlePaddle, MXNet, CNTK, Chainer, and Deeplearning4j.

[0021] Furthermore, the deep learning network model for image instance segmentation is one of FCN, U-Net, SegNet, Mask-RCNN, PolarMask, TensorMask, Mask Scoring RCNN, YOLACT series, SOLO series, DeepLab series, and derivative models based on them.

[0022] Furthermore, the parameter settings of the deep learning network model in step 5 include one or a combination of learning rate, optimizer, batch size, activation functions, and number of epochs.

[0023] Furthermore, the optimizer of the deep learning network model in step 5 is selected from one or a combination of Adam, Adamax, Nadam, BGD, SGD, MBGD, Momentum, Adagrad, Adadelta, or RMSprop.

[0024] Furthermore, the activation function of the deep learning network model in step 5 is selected from Softmax, Sigmoid, Tanh, Relu, Leaky Relu, PRelu, RRelu, Elu, Selu, Swish, or Maxout.

[0025] The present invention has the following technical effects and advantages:

[0026] The automatic segmentation method for the connector profile image based on deep learning proposed by the present invention can achieve the automatic segmentation of a batch of connector profile images and reduce the manual participation process;

[0027] In addition, this method can achieve the unification and standardization of the segmentation process of the connector profile image;

[0028] Compared with the segmentation method using traditional algorithms, the segmentation of this invention has higher accuracy, can adapt to more complex scenarios, and has stronger scalability. Relatively good prediction results can also be obtained even when the cross-sectional profiles of some connectors are not obvious. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 It is a schematic flow diagram of the method of this invention.

[0030] Figure 2 It is a test diagram in Embodiment 1.

[0031] Figure 3 It is the test effect diagram of the model obtained by using the method in Embodiment 1.

[0032] Figure 4 It is a test diagram in Embodiment 2.

[0033] Figure 5 It is the test effect diagram of the model obtained by using the method in Embodiment 2 - rivet.

[0034] Figure 6 It is the test effect diagram of the model obtained by using the method in Embodiment 2 - upper plate.

[0035] Figure 7 It is the test effect diagram of the model obtained by using the method in Embodiment 2 - lower plate. DETAILED DESCRIPTION OF THE INVENTION

[0036] In order to make the objectives, technical solutions and advantages of this invention clearer, the following further elaborates on this invention in conjunction with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this invention and are not used to limit this invention.

[0037] As Figure 1 shown, this invention adopts an automatic segmentation method for connector cross-sectional images based on deep learning, including the following steps:

[0038] Step 1, production of the training set: For a specific connection process, such as spot welding, self-piercing riveting, non-rivet self-piercing riveting, hot melt self-tapping screws, structural adhesives, or a combination of one or more of them, collect a certain number of cross-sectional images of the connector as sample images; annotate each sample image through an annotation tool to mark the positions of different plates, fasteners and other areas of concern in the sample image, generate the corresponding mask image and text file for the annotated sample image, and use the mask image and text file together with the original Figure 1 sample image as the training sample; repeat the above processing of the sample image to obtain the training samples corresponding to all sample images, and then obtain the training set;

[0039] Step 2: Perform image enhancement on the training samples obtained in Step 1, and generate corresponding mask images and their text files. The objects of different image enhancement processes can be the original enhanced images, and then the corresponding text files need to be modified according to the enhancement method. However, some directly perform enhancement on the Mask images. The enhancement process in Step 2 is not limited to being after Step 1.

[0040] Step 3: Select or build a deep learning network model for image instance segmentation.

[0041] Step 4: Set up the training environment for the selected or built deep learning network model. Environment setup generally refers to the CUDA version, VisualStudio version, python version, framework version (such as the versions of tensorflow and pytorch), and other dependent packages, etc.

[0042] Step 5: Set the training parameters of the deep learning network model. The network parameters are set as one or more combinations of learning rate, optimizer, batch size, activation function, and number of epochs. The optimizer is one or more combinations of Adam, Adamax, Nadam, BGD, SGD, MBGD, Momentum, Adagrad, Adadelta, RMSprop. The activation functions are Softmax, Sigmoid, Tanh, Relu, Leaky Relu, PRelu, RRelu, Elu, Selu, Swish, Maxout, etc.

[0043] Step 6: Use the training set obtained in Step 1 to train the deep learning network model selected in Step 3. After training, obtain a deep learning network model suitable for automatic segmentation of the cross-sectional images of the connector.

[0044] Step 7: Input the image to be detected into the deep learning network model trained in Step 6. The deep learning network model outputs a predicted mask image. According to the segmentation effect of the instances in the mask image, repeat Step 5 and Step 6 for model optimization.

[0045] The specific connection processes targeted by this method include one or more combinations of spot welding, self-piercing riveting, non-rivet self-piercing riveting, hot melt self-tapping screws, and structural adhesives.

[0046] In this embodiment, as the annotation tool for annotating the sample image in step 1, one or a combination of image annotation software or tools such as Labelme, CVAT, VIA (VGG Image Annotator), PixlAnnotationTool, and EISeg can be selected.

[0047] In this embodiment, the text file obtained in step 1 is in the format of JSON or XML, etc., and is used to record specific information about the board, connectors, and the manual segmentation results of the regions formed after their connection.

[0048] In this embodiment, as the enhancement tool for enhancing the image in step 2, one or a combination of OpenCV, Imgaug, Skimage, PIL, Augmentor, and Albumentations is selected. Using the above enhancement tools, the image enhancement methods for the image are one or a combination of translation, flipping, rotation, adding noise, blurring, sharpening, cropping, and scaling.

[0049] In this embodiment, the deep learning framework in step 3 can be selected from one of Tensorflow, Keras, Pytorch, Caffe, Theano, PaddlePaddle, MXNet, CNTK, Chainer, and Deeplearning4j.

[0050] In this embodiment, the deep learning network model in step 3 is one of FCN, U-Net, SegNet, Mask-RCNN, PolarMask, TensorMask, Mask Scoring RCNN, YOLACT series, SOLO series, DeepLab series, and derivative models based on them.

[0051] In this embodiment, the improvements to the deep learning network model selected in step 3 are as follows:

[0052] (1) For different connection processes, the convolutional layer, pooling layer, and fully connected layer of the neural network need to be adjusted to achieve the expected segmentation effect;

[0053] (2) For the prediction branch of the deep learning network model, a mask smoothing processing layer needs to be added to reduce the impact of noise on engineering measurements;

[0054] (3) For the output segmentation image, for the same instance, the mask color needs to be fixed according to the specific instance position and prediction result for convenient specific engineering analysis;

[0055] (4) Map the predicted mask to the original image for intuitive comparison of the segmentation effect. At the same time, the mask image needs to be output separately to reduce the color conflict between the shooting background and the mask image, which may cause errors in the later automatic engineering analysis.

[0056] Example 1:

[0057] Randomly select a deep learning instance segmentation network to achieve automatic segmentation of the cross-sectional image of aluminum alloy resistance spot welding (RSW).

[0058] Collect 18 cross-sectional images of RSW as the training set, annotate this sample through the Labelme tool, record the annotation information in a JSON file, then convert the annotated sample into a mask image, and use them together as the training set to input into the network for training. After training for 10 Epochs, a deep learning segmentation model is obtained.

[0059] Figure 2 The test image of the deep learning segmentation model obtained by the technical solution of this embodiment is given.

[0060] Figure 3 The test effect diagram of the deep learning segmentation model obtained according to the technical solution of this embodiment is given. From the segmentation effect, the nugget area in the RSW cross-sectional image can be accurately identified.

[0061] Example 2:

[0062] Randomly select a deep learning instance segmentation network to achieve automatic segmentation of the cross-sectional image of self-piercing riveting (SPR).

[0063] Collect 40 cross-sectional images of SPR as the training set, annotate this sample through the Labelme tool, record the annotation information in a JSON file, then convert the annotated sample into a mask image, and use them together as the training set to input into the network for training. After training for 36 Epochs, a deep learning segmentation model is obtained.

[0064] Figure 4 The test image of the deep learning segmentation model obtained by the technical solution of this embodiment is given.

[0065] Figures 5 - 7 The test effect diagram of the deep learning segmentation model obtained according to the technical solution of this embodiment is given. From the segmentation effect, the rivet, upper plate, and lower plate areas in the SPR cross-sectional image can be accurately identified.

[0066] The above embodiments are only used to illustrate the design ideas and features of the present invention, and their purpose is to enable those skilled in the art to understand the content of the present invention and implement it accordingly. The protection scope of the present invention is not limited to the above embodiments. Therefore, all equivalent changes or modifications made according to the principles and design ideas disclosed by the present invention are within the protection scope of the present invention.

Claims

1. An automatic segmentation method for the cross-sectional image of a connector based on deep learning, characterized in that, The method includes the following steps: Step 1, Training set production: For the lightweight connection process, collect the cross-sectional images of the connecting parts as sample images, annotate each sample image with an annotation tool, mark the positions of different plates, fasteners and other areas of concern in the image, generate the mask image and text file corresponding to the annotated sample image, and use them together with the original image as training samples; The annotation tool for annotating the sample images can be one or a combination of Labelme, CVAT, VIA, PixlAnnotationTool, and EISeg; The obtained text file is in JSON or XML format, and is used to record the specific information of the plates and fasteners and the manual segmentation results of the areas formed after their connection; Step 2, Perform image enhancement processing on the training samples, and generate the corresponding mask image and text file; Step 3, Select or build a deep learning model based on image instance segmentation; The deep learning framework can be one of Tensorflow, Keras, Pytorch, Caffe, Theano, PaddlePaddle, MXNet, CNTK, Chainer, Deeplearning4j; The deep network model for image instance segmentation is one of FCN, U-Net, SegNet, Mask-RCNN, PolarMask, TensorMask, Mask Scoring RCNN, YOLACT series, SOLO series, DeepLab series and their derivative models; The improvements to the deep learning network model selected in Step 3 are as follows: (1) For different connection processes, it is necessary to adjust the convolutional layer, pooling layer, and fully connected layer of the neural network to achieve the expected segmentation effect; (2) For the prediction branch of the deep learning network model, a mask smoothing processing layer needs to be added to reduce the impact of noise on engineering measurements; (3) For the output segmentation image, for the same instance, the mask color needs to be fixed according to the specific instance position and prediction result for specific engineering analysis; (4) Map the predicted mask to the original image for intuitive comparison of the segmentation effect. At the same time, the mask image needs to be output separately to reduce the color conflict between the shooting background and the mask image and avoid causing errors in later engineering automatic analysis; Step 4, Build the training environment for the selected or built deep learning network model; The environment construction refers to the CUDA version, VisualStudio version, python version, framework version and other dependent packages; Step 5, set the training parameters of the deep learning network model; the parameter setting of the deep learning network model includes one or more combinations of learning rate, optimizer, batch size, activation function, number of iterations; the optimizer of the deep learning network model selects one or more combinations of Adam, Adamax, Nadam, BGD, SGD, MBGD, Momentum, Adagrad, Adadelta or RMSprop; the activation function of the deep learning network model selects Softmax, Sigmoid, Tanh, Relu, LeakyRelu, PRelu, RRelu, Elu, Selu, Swish or Maxout; Step 6, add the training samples to the deep learning network model selected in Step 3 for training, and obtain a deep learning model suitable for automatic segmentation of the connector profile image after training; Step 7, input the image to be detected into the deep learning network model trained in Step 6, the deep learning network model outputs a predicted mask image, and according to the segmentation effect of the instances in the mask image, repeat Step 5 and Step 6 for model optimization.

2. The automatic segmentation method of the connector profile image based on deep learning according to claim 1, wherein The lightweight connection process includes one or more combinations of spot welding, self-piercing riveting, non-rivet self-piercing riveting, hot melt self-tapping screws, structural adhesives.

3. The automatic segmentation method for the connector profile image based on deep learning according to claim 1, characterized in that, In Step 2, select one or more combinations of OpenCV, Imgaug, Skimage, PIL, Augmentor, Albumentations for the image enhancement tool.

4. The automatic segmentation method for the connector profile image based on deep learning according to claim 3, characterized in that Using the above enhancement tools, the image enhancement methods for the image are one or more combinations of translation, flipping, rotation, adding noise, blurring, sharpening, cropping, scaling.

Images