An engine main bearing cap image segmentation method based on FCN and ELM

Abstract

An image segmentation method based on the combination of FCN and ELM is mainly used to solve the problem of low computational efficiency, long training time, and insufficient accuracy when the image semantic segmentation of FCN is performed. After the image features are collected, a sliding window is used to traverse the entire image to be classified for classification, and then the feature map is upsampled. Therefore, an image segmentation algorithm based on the combination of FCN and ELM is proposed, which uses the good classification of ELM to accelerate the extraction and classification of image features. The beneficial effects of the present invention are: changing the upsampling method in the traditional FCN. After retaining the operations of pooling layer 3 and pooling layer 4 in the previous method, the upsampling method is modified, and after the pooling layer 5, it is connected to the ELM network structure to optimize the classification of the extracted feature information. Then the upsampling process is performed to restore the dimension of the original image. The experimental results show that the proposed method can improve the accuracy of image segmentation and provide a new method and idea for improving the accuracy of image segmentation of the engine main bearing cover.

Description

Technical Field

[0001] The invention belongs to the field of computer graphics and computer vision, and relates to an image segmentation algorithm based on the combination of FCN and ELM, which is mainly used for the segmentation and recognition of optical images. Background Art

[0002] Image segmentation is a classic problem in the field of computer vision, and has now become a hot topic in the field of image understanding. Image segmentation is usually the first step in image analysis, one of the foundations of computer vision, and an important part of image understanding. Image segmentation refers to dividing an image into several non-overlapping regions based on features such as grayscale, color, spatial texture, and geometric shape, so that these features show consistency or similarity in the same region, and show obvious differences between different regions, thereby segmenting the image into a set of non-overlapping regions and extracting the target region of interest. Simply put, in an image, the target is separated from the background. How to perform image segmentation and accurately extract the target region is one of the basic problems in the field of computer vision. Regarding image segmentation technology, due to the importance and difficulty of the problem itself, the image segmentation problem has attracted many researchers to make great efforts since the 1970s. Although there is no universal and perfect image segmentation method so far, there is basically a consensus on the general rules of image segmentation, and a considerable number of research results and methods have been produced. The accuracy of image segmentation has a significant impact on the recognition performance and measurement accuracy of the machine vision system. For example, in a system with recognition as its main function, the result of image segmentation has a significant impact on the shape characteristics of the target, which determines the success or failure.

[0003] One of the most widely used networks in image segmentation is FCN (Fully Convolutional Neural Network). It is an improvement on CNN (Convolutional Neural Network), which removes the three fully connected layers after the convolution layer of CNN and replaces them with three convolution layers. While the original CNN only accepts input images of a single fixed size, FCN can accept input images of any size, which is more efficient and avoids the problems of repeated calculations and space waste caused by the use of neighborhoods. Finally, the output classification map of the output layer is upsampled to the input image size to obtain the predicted output of the network. Among them, the upsampling operation of the feature can actually be completed by the convolution kernel deconvolution operation on the feature, and this process is also called deconvolution. FCN solves the problem of repeated storage and calculation convolution caused by the use of pixel blocks in traditional segmentation networks, but it still has some obvious defects: for example, it is relatively rough in the deconvolution process and is not sensitive to image details; the training is complex, the amount of calculation is large, and it cannot achieve real-time performance. Summary of the invention

[0004] The problem to be solved by the present invention is that the image semantic segmentation of FCN is mainly used to solve the problem of insufficient accuracy caused by using a sliding window to traverse the entire image to be classified for classification after image feature collection, and then upsampling the feature map. Therefore, an image segmentation algorithm based on the combination of FCN and ELM is proposed, which uses the good classification of ELM to accelerate the extraction and classification of image features.

[0005] The technical solution adopted by the present invention is: combining FCN (Fully Convolutional Neural Network) and ELM (Extreme Learning Machine) to realize the image segmentation algorithm, that is, using the convolution layer in FCN to mainly undertake the function of feature extraction, extracting the image features of the input image, removing the fully connected layer of the convolutional neural network, and then using ELM to replace the fully connected layer as a classifier to quickly classify the image pixels, and then connect the upsampling operation to the extreme learning machine, and use the classified image pixels as the input of the upsampling operation. The generated feature thumbnail needs to be restored to the dimension of the original image. Thereby shortening the specific training time and improving the accuracy. Specifically including the following steps:

[0006] Step 1: Collection of original image data: Use the high-resolution industrial digital camera in the MV-VS1600 teaching innovation experiment platform to take 500 images for training data and 50 images for testing data.

[0007] Step 2: Image preprocessing, convert the collected images into grayscale images, and then perform image enhancement. First, the image is grayscale converted, and then the median filter is used to remove some noise on the image surface.

[0008] Step 3: To ensure that the trained model has better accuracy, the images are uniformly made into [224*224] size. They are placed in the folder images, with the training set and validation set placed in different folders. The folders are marked as train (training set) and validation (validation set).

[0009] Step 4: Use labelme software to create a data set. Label the obtained training data and test data. Then put the prepared 8-bit depth labeled images in the annotations folder, where the training set and validation set are placed in different folders. The folders are marked as train (training set) and validation (validation set).

[0010] Step 5: Initialize the FCN network using the VGG16 convolutional neural network model and perform feature extraction.

[0011] a. The convolutional layer in FCN is mainly responsible for image feature extraction.

[0012] The image input size is:

[0013] W1×H1×D1 (1)

[0014] Among them, W1: image width, H1: image height, D1: image depth (number of channels)

[0015] b. The output image size after convolution is:

[0016]

[0017] W is the width of the input image; H is the length of the input image; D is the image depth (number of channels); F is the dimension of the convolution kernel; S is the stride; P is the number of padded pixels. n represents the order.

[0018] c. Convolution and activation function operations:

[0019]

[0020] The part in brackets is the main operation of the convolutional layer to extract features: l is the current convolutional layer of the network, j is one of the feature maps of the current convolutional layer of the network, is the feature map, is the value of the i-th pixel in the receptive field, M j represents the set of selected input maps, is the convolution kernel connecting the i-th map in layer l-1 and the j-th map in layer l, is the bias term, and f() is the activation function.

[0021] d. The qth layer is a subsampling layer (pooling layer), and its hth feature map is:

[0022]

[0023] dowm() is the downsampling function (using the maximum value method for downsampling), is the weight, For bias.

[0024] Step 6: Replace the last fully connected layer of the constructed convolutional neural network with the ELM network structure to realize the combination of FCN and ELM.

[0025] a. Use the training set feature n extracted by the convolutional neural network to construct an ELM network, where the input layer of the ELM network has n input variables (i.e., training set feature n). i ,t i),i=1,2,…,N,where x i =[x i1 ,x i2 ,…,x m ]∈R n , t i =[t i1 ,t i2 ,…,t m ]∈R m , then the objective function of the extreme learning machine is

[0026] Hβ=T′ (5)

[0027] Where H is the output matrix of the hidden layer of the neural network; T is the expected output, and T' is the transposed matrix of T.

[0028] bH is in the form of:

[0029] The form of cT is: T = [t1, t2, ..., t N ] T (7)

[0030] d. where h (x) The form is:

[0031] h(x)=N(W.x+b) (8)

[0032] W∈R L×n is the weight from the input layer to the hidden layer, b∈R L is the bias of the hidden layer, and N() is the activation function.

[0033] e. The connection weight W between the input layer and the hidden layer and the bias b of the hidden layer are randomly selected before training. The calculation formula of β is:

[0034] β=H + T (9)

[0035] Among them, H + is the Moore-Penrose generalized inverse matrix of H.

[0036] Step 7: After the ELM is constructed, the fully connected layer of the output part is replaced with a convolutional layer, and the deconvolution layer construction operation (deconvolution) is performed, which is an upsampling process. In FCN, the pooling operation reduces the size of the input image. However, in the image segmentation process, each pixel needs to be classified, and finally a segmented image with the same dimension as the input image is obtained. Therefore, the generated feature thumbnail needs to be restored to the dimension of the original image. Deconvolution can achieve the effect of outputting the reconstructed input through reverse training, so that the output image is restored to the same dimension as the input image.

[0037] Put the dataset folder images and dataset annotations folder into the Data folder;

[0038] Input the training set into the built FCN and ELM combined model for training;

[0039] Input the test set into the built FCN and ELM combination model, and test the trained FCN and ELM combination model;

[0040] The beneficial effects of the present invention are as follows: a method for image segmentation based on the combination of FCN and ELM is proposed, which changes the upsampling method in the traditional FCN. After retaining the operations of pooling layer 3 and pooling layer 4 in the previous method, the upsampling method is modified, and after pooling layer 5, it is connected to the ELM network structure to optimize the classification of the extracted feature information. Then the upsampling process is performed to restore the dimension of the original image. The experimental results show that the proposed method can improve the accuracy of image segmentation and provide a new method and idea for improving the accuracy of engine main bearing cover image segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 The overall flow chart of the engine main bearing cap image segmentation method based on the combination of FCN and ELM;

[0042] Figure 2 Schematic diagram of upsampling of the engine main bearing cap image segmentation method based on the combination of FCN and ELM

[0043] Figure 3 Engine main bearing cap physical picture

[0044] Figure 4 Engine main bearing cap label diagram

[0045] Figure 5 Engine main bearing cap prediction results DETAILED DESCRIPTION

[0046] The following is further described in conjunction with the accompanying drawings;

[0047] like Figure 1 The overall flow chart shows the flow chart of the engine main bearing cap image segmentation method based on the combination of FCN and ELM. The steps shown in the figure are:

[0048] Step 1: Collection of original image data: Use the high-resolution industrial digital camera in the MV-VS1600 teaching innovation experiment platform to take images. 500 training data and 50 test data.

[0049] Step 2: Image preprocessing, convert the collected images into grayscale images, and then perform image enhancement. First, the image is grayscale converted, and then the median filter is used to remove some noise on the image surface.

[0050] Step 3: To ensure that the trained model has better accuracy, the images are uniformly made into [224*224] size. They are placed in the folder images, with the training set and validation set placed in different folders. The folders are marked as train (training set) and validation (validation set).

[0051] Step 4: Use labelme software to create a data set. Label the obtained training data and test data. Then put the prepared 8-bit depth labeled images in the annotations folder, where the training set and validation set are placed in different folders. The folders are marked as train (training set) and validation (validation set).

[0052] Step 6: Construct FCN network initialization, using VGG16 convolutional neural network model to perform feature extraction.

[0053] a. The convolutional layer in FCN is mainly responsible for image feature extraction.

[0054] The image input size is:

[0055] W1×H1×D1 (1)

[0056] Among them, W1 is the image width, H1 is the image height, and D1 is the image depth (number of channels)

[0057] b. The output image size after convolution is:

[0058]

[0059] W is the width of the input image; H is the length of the input image; D is the image depth (number of channels); K is a specific value; F is the dimension of the convolution kernel; S is the step size; P is the number of padded pixels. n represents the order.

[0060] c. Convolution and activation function operations:

[0061]

[0062] The part in brackets is the main operation of the convolutional layer to extract features: l is the current convolutional layer of the network, j is one of the feature maps of the current convolutional layer of the network, is the feature map, is the value of the i-th pixel in the receptive field, Mj represents the set of selected input maps, is the convolution kernel connecting the i-th map in layer l-1 and the j-th map in layer l, is the bias term, and f() is the activation function.

[0063] d. The qth layer is a subsampling layer (pooling layer), and its hth feature map is:

[0064]

[0065] dowm() is the downsampling function (using the maximum value method for downsampling), is the weight, For bias.

[0066] Step 6: Replace the last fully connected layer of the constructed convolutional neural network with the ELM network structure to achieve the combination of FCN and ELM.

[0067] a. Use the training set feature n extracted by the convolutional neural network to construct an ELM network, where the input layer of the ELM network has n input variables (i.e., training set feature n). i ,t i ),i=1,2,…,N,where x i =[x i1 ,x i2 ,…,x m ]∈R n , t i =[t i1 ,t i2 ,…,t m ]∈R m , then the objective function of the extreme learning machine is

[0068] Hβ=T′ (5)

[0069] Where H is the output matrix of the hidden layer of the neural network; T is the expected output, and T' is the transposed matrix of T.

[0070] bH is in the form of:

[0071] The form of cT is: T = [t1, t2, ..., t N ] T (7)

[0072] where h (x) The form is:

[0073] h(x)=N(W.x+b) (8)

[0074] W∈R L×nis the weight from the input layer to the hidden layer, b∈R L is the bias of the hidden layer, and N() is the activation function.

[0075] d. The connection weight W between the input layer and the hidden layer and the bias b of the hidden layer are randomly selected before training. The calculation formula of β is:

[0076] β=H + T (9)

[0077] Among them, H + is the Moore-Penrose generalized inverse matrix of H.

[0078] Step 7: After the ELM is constructed, perform the deconvolution layer construction operation (deconvolution). It is the inverse process of pooling. In FCN, the pooling operation reduces the size of the input image. However, in the image segmentation process, each pixel needs to be classified, and the final segmented image with the same dimension as the input image is obtained. Therefore, the generated feature thumbnail needs to be restored to the dimension of the original image. Deconvolution can achieve the effect of outputting the reconstructed input through reverse training, so that the output image is restored to the same dimension as the input image.

[0079] Put the dataset folder images and dataset annotations folder into the Data folder;

[0080] Input the training set into the built FCN and ELM combined model for training;

[0081] The test set is input into the built FCN and ELM combined model, and the trained FCN and ELM combined model is tested; as shown in Table 1 below:

[0082]

[0083] Among them, PA (pixel accuracy), MPA (mean pixel accuracy), MIoU (mean intersection over union). The experimental results show that the proposed method can improve the accuracy of image segmentation to a certain extent, and provides a new method and idea for improving the accuracy of engine main bearing cap image segmentation.

Claims

1. The image segmentation method based on the combination of FCN and ELM is characterized by: The specific steps include: Step 1: Collection of original image data: Use the high-resolution industrial digital camera in the MV-VS1600 teaching innovation experiment platform to take 500 images for training data and 50 images for testing data; Step 2: Image preprocessing: convert the collected images into grayscale images and then perform image enhancement. First, perform grayscale conversion on the images and then use median filtering to remove some noise on the image surface. Step 3: To ensure that the trained model has better accuracy, the images are uniformly made into [224*224] size and placed in the folder images. The training set and validation set are placed in different folders, marked as train and validation. Step 4: Use labelme software to create a data set; label the obtained training data and test data; then put the prepared 8-bit depth labeled images in the annotations folder, where the training set and validation set are placed in different folders, and the folders are marked as train and validation; Step 5: Build the FCN network initialization, using the VGG16 convolutional neural network model for feature extraction; Step 6: Replace the last fully connected layer of the constructed convolutional neural network with the ELM network structure to achieve the combination of FCN and ELM; Step 7: After the ELM is constructed, replace the fully connected layer of the output part with the convolution layer, and perform the deconvolution layer construction operation, which is an upsampling process; in FCN, the pooling operation reduces the size of the input image, but in the image segmentation process, each pixel needs to be classified, and finally a segmented image with the same dimension as the input image is obtained, so the generated feature thumbnail needs to be restored to the dimension of the original image; deconvolution can achieve the effect of output reconstruction input through reverse training, so that the output image is restored to the same size as the input image; put the dataset folder images and the dataset annotations folder into the Data folder; input the training set into the built FCN and ELM combined model for training; input the test set into the built FCN and ELM combined model, and test the trained FCN and ELM combined model; The specific method of step 6 is: a. The training set feature n extracted by the convolutional neural network is used to construct the ELM network, where the input layer of the ELM network has n input variables, i.e., the training set feature n. i , t i ), i = 1, 2, ..., N, where x i =[x i1 , x i2 , …, x m ]∈R n , t i =[t i1 , t i2 ,…t m ]∈R m , then the objective function of the extreme learning machine is: Hβ=T′ (1) Where H is the output matrix of the hidden layer of the neural network; T is the expected output, and t′ is the transposed matrix of T; bH is in the form of: The form of cT is: T=[t1,t2,…,t N ] T (3) where h (x) The form is: h(x) = N(W.x + b) (4) W∈R L×n is the weight from the input layer to the hidden layer, b∈R L is the bias of the hidden layer, N() is the activation function; d. The connection weight W between the input layer and the hidden layer and the bias b of the hidden layer are randomly selected before training. The calculation formula of β is: β=H + T (5) Among them, H + H is the Moore-Penrose generalized inverse matrix.

2. The image segmentation method based on the combination of FCN and ELM according to claim 1, characterized in that: The specific method of step 5 is: a. The convolution layer in FCN is mainly responsible for image feature extraction; The image input size is: W1×H1×D1 (6) Where W1 is the image width, H1 is the image height, and D1 is the image depth b. The output image size after convolution is: W is the width of the input image; H is the length of the input image; D is the image depth; K is the specific value; F is the dimension of the convolution kernel; S is the step size; P is the number of padded pixels, and n represents the order; c. Convolution and activation function operations: The part in brackets is the main operation of the convolutional layer to extract features: l is the current convolutional layer of the network, j is one of the feature maps of the current convolutional layer of the network, is the feature map, is the value of the i-th pixel in the receptive field, M j represents the set of selected input maps, is the convolution kernel connecting the i-th map in layer l-1 and the j-th map in layer l, is the bias term, f() is the activation function, and the qth layer is the subsampling layer, then its hth feature map is: down() is a downsampling function, which uses the maximum value method for downsampling. is the weight, For bias.

Images