A Large Workpiece Detection Method Based on Few-Shot Learning

Abstract

The present invention discloses a large workpiece detection method based on few-shot learning. A six-axis robotic arm is arranged on a freely liftable platform, and a first industrial camera is arranged on the six-axis robotic arm through a rotating cloud platform. The intelligent rotating arm is provided with a rotatable rotating column on the chassis, and a guide rail is arranged at the top of the rotating column. A second industrial camera is arranged on the guide rail through another rotating cloud platform. The method includes: 1. Photographing the workpiece areas corresponding to each detection point on the side and top surfaces of the workpiece, so as to obtain A image data of the side surface of the workpiece and B image data of the top surface of the workpiece; 2. Converting the image data into grayscale images and then performing preprocessing, and generating multiple grayscale sample images; 3. Constructing a few-shot image recognition model based on the InceptionV3 network and training it to be used for classifying the sample images to be detected. The present invention has a high recognition rate, is convenient for product update, and can meet the requirements for surface detection of large workpieces.

Description

Technical Field

[0001] The present invention belongs to the technical field of defect detection, and relates to a vision detection method for large workpieces. Background Art

[0002] During the production and processing of large workpieces, unavoidable appearance defects exist. The appearance detection of large workpieces has always been a difficult problem for many enterprises. When large workpieces leave the factory, their appearance needs to be detected to ensure that there are no defective products. Manual detection not only takes a long time, but also is difficult to accurately control in terms of accuracy, and there may be relatively large deviations. The existing instrument detection is carried out by manually holding, which cannot achieve intelligent detection and has poor real-time performance.

[0003] The existing vision detection devices perform real-time detection on small parts in the production line and cannot meet the requirements of the surface of large workpieces. Large workpieces need to rely on manual detection, and people are prone to fatigue and other situations during the detection process. The intelligent detection device for the surface of large workpieces is very important for enterprises. Summary of the Invention

[0004] The present invention is to solve the above-mentioned deficiencies existing in the prior art, and proposes a detection method for large workpieces based on few-shot learning, in order to meet the detection function of the surface of large workpieces, thereby improving the recognition rate of appearance detection.

[0005] In order to achieve the above invention purpose, the present invention adopts the following technical solutions:

[0006] The detection method for large workpieces based on few-shot learning of the present invention is characterized in that it is applied to a detection environment composed of a six-axis robotic arm and an intelligent rotating arm. The six-axis robotic arm is arranged on a freely lifting platform, and a first industrial camera is arranged on the six-axis robotic arm through a rotating cloud platform for rotating and photographing the side of the workpiece within a height range. The intelligent rotating arm is provided with a rotatable rotating column on the chassis, and a guide rail is arranged on the top of the rotating column. A second industrial camera is arranged on the guide rail through another rotating cloud platform. The second industrial camera moves on the guide rail and rotates and photographs the top of the workpiece. The detection method for large workpieces is carried out according to the following steps:

[0007] Step 1): For the side shape of the workpiece, according to the side detection requirements, A detection points are set, and thus the motion trajectory of the six-axis robotic arm is planned according to the A detection points, so that while the six-axis robotic arm drives the first industrial camera to move according to the planned motion trajectory, the workpiece areas corresponding to the respective detection points on the side of the workpiece are photographed, thereby obtaining A image data of the side of the workpiece;

[0008] Step 2) According to the shape of the top of the workpiece and the top detection requirements, B detection points are pre-determined, and based on these B detection points, the motion trajectory of the intelligent rotating arm is planned. While the intelligent rotating arm drives the second industrial camera to move according to the planned motion trajectory, it captures the workpiece areas corresponding to each detection point on the top of the workpiece, so as to obtain B image data of the top surface of the workpiece.

[0009] Step 3) Determine the color space of the image data. If it is a color image, it is directly converted into a grayscale image; otherwise, it is retained, indicating that the original image is a grayscale image.

[0010] Step 4) Use an image processing algorithm to enhance a single sample of the grayscale image:

[0011] Step 4.1) According to the position of the workpiece in the grayscale image with a gray background, use Equation (1) to perform multi-target rough segmentation on any grayscale image to obtain a segmented binary image:

[0012]

[0013] In Equation (1), f(x, y) represents the pixel value of the grayscale image at the position point (x, y), b0 represents the black pixel value, b1 represents the white pixel value, t represents the segmentation threshold; f′(x, y) represents the pixel value of the segmented binary image at the position point (x, y).

[0014] Step 4.2) Use the bilinear interpolation method to perform two linear interpolations on the segmented binary image in the x direction and then one linear interpolation in the y direction to obtain a standardized binary image.

[0015] Step 4.3) Perform data enhancement processing on the standardized binary image, including: mirroring and symmetry operations, scale transformation operations, rotation operations, and translation operations, so as to generate multiple sample images to complete the enhancement of a single sample.

[0016] Step 5) Use an image processing algorithm to generate an adversarial network GAN to enhance a single sample of the grayscale image:

[0017] Construct a generative adversarial network GAN composed of a generative network G and a discriminative network D, and process any grayscale image to generate multiple sample images.

[0018] Step 6) Combine the multiple sample images generated in Step 4 and Step 5 into a sample library, and set corresponding class labels for each image sample in the sample library.

[0019] Step 7) Construct a small-sample image recognition model based on the InceptionV3 network, including: six convolutional layers, two pooling layers, a mixed layer, a fully connected layer, and a dropout layer.

[0020] Any sample image F in the sample library is input into the small-sample image recognition model, and the first convolutional layer performs a convolution operation on the sample image F to obtain the downsampled image data F1;

[0021] The second convolutional layer performs a convolution operation on the image data F1 to obtain the downsampled image data F2;

[0022] The third convolutional layer performs a convolution operation on the image data F2 to obtain the downsampled image data F3;

[0023] The first pooling layer performs a max pooling operation on the image data F3 to obtain the pooled image data F4;

[0024] The fourth convolutional layer performs a convolution operation on the image data F4 to obtain the downsampled image data F5;

[0025] The fifth convolutional layer performs a convolution operation on the image data F5 to obtain the downsampled image data F6;

[0026] The sixth convolutional layer performs a sixth convolution operation on the image data F6 to obtain the downsampled image data F7;

[0027] The hybrid layer performs eight convolution operations on the image data F7 to obtain the image feature data F8;

[0028] The second pooling layer performs a max pooling operation on the image feature data F8 to obtain the pooled image feature data F9;

[0029] The fully connected layer performs a fully connected operation on the image feature data F9 to obtain the image feature data T;

[0030] The dropout layer operates on the image feature data T to obtain the feature data G;

[0031] The softmax function is used to classify the feature data G to obtain the probability values of each class, and the maximum probability value is used as the class corresponding to the sample image F;

[0032] Step 8) With the goal of minimizing the cross-entropy loss function, the small-sample image recognition model is trained by the Adam optimization algorithm, and its network parameters are updated until the loss function converges, so as to obtain a trained small-sample image recognition model for realizing the class recognition of the sample image to be detected.

[0033] An electronic device according to the present invention includes a memory and a processor, characterized in that the memory is used to store a program for supporting the processor to execute the method for large workpiece detection, and the processor is configured to execute the program stored in the memory.

[0034] A computer-readable storage medium of the present invention stores a computer program on the computer-readable storage medium, characterized in that the computer program executes the steps of the large workpiece detection method when run by a processor.

[0035] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0036] (1) The present invention uses a six-axis robotic arm and an intelligent rotating platform to cooperate with a mobile cloud platform to control the movement trajectory of the camera. A freely elevating platform is designed between the six-axis robotic arm and the base, and the adjustable range is N centimeters, realizing the function of the industrial camera moving freely on the surface of large parts and solving the problem of real-time image acquisition on the surface of large workpieces.

[0037] (2) The structure of the present invention is simple. Different models of cameras can be replaced for different parts, and the movement trajectory can be reset, which is convenient to operate and can be applied to the surface detection of most large workpieces. It has the characteristics of simple structure, convenient operation, high recognition rate, and convenient product update.

[0038] (3) The number of defective products on the surface of large workpieces is small, which has a certain impact on visual inspection. The present invention adopts the method of small sample learning to avoid situations such as inability to train caused by too few bad samples, and uses the small sample learning model to identify appearance features, improving the recognition rate of appearance inspection. Description of the Drawings

[0039] Figure 1 is a schematic side view of the large workpiece detection device and method of the present invention;

[0040] Figure 2a is a schematic side view of the top detection of the large workpiece detection device and method of the present invention;

[0041] Figure 2b is a schematic top view of the top detection of the large workpiece detection device and method of the present invention;

[0042] Figure 3 is a block diagram of the small sample recognition of the large workpiece detection device and method of the present invention. Detailed Embodiments

[0043] In this embodiment, a large workpiece detection method based on small sample learning is as Figure 1As shown in the figure, it is applied to a detection environment composed of a six-axis robotic arm and an intelligent rotating arm. The six-axis robotic arm is set on a free-lifting platform, and a first industrial camera is set on the six-axis robotic arm through a rotating cloud platform, which is used to rotate and photograph the side of the workpiece within a height range. On the intelligent rotating arm, a rotatable rotating column is set on the chassis, and a guide rail is set on the top of the rotating column. A second industrial camera is set on the guide rail through another rotating cloud platform. The second industrial camera moves on the guide rail and rotates and photographs the top of the workpiece; to complete the work of detecting the appearance defects on the side of large workpieces. When the model of the large workpiece changes, only the motion trajectory of the six-axis robotic arm needs to be modified. In this way, not only the number of cameras is saved, but also the appearance of various models of large workpieces can be detected flexibly. The top of the large workpiece is detected by a separate intelligent rotating arm. As Figure 2a and Figure 2b shown, a separate detection station is used for the entire top. The intelligent rotating arm cooperates with the guide rail to design a polar coordinate motion trajectory, and the camera acquisition area covers the entire top position.

[0044] In this embodiment, the detection method of the large workpiece is carried out according to the following steps:

[0045] Step 1) For the shape of the side of the workpiece, according to the side detection requirements, A detection points are set, and then the motion trajectory of the six-axis robotic arm is planned according to the A detection points, so that while the six-axis robotic arm drives the first industrial camera to move according to the planned motion trajectory, the workpiece areas corresponding to the respective detection points on the side of the workpiece are photographed, thereby obtaining A image data of the side of the workpiece;

[0046] Step 2) For the shape of the top of the workpiece, according to the top detection requirements, B detection points are set in advance, and then the motion trajectory of the intelligent rotating arm is planned according to the B detection points, so that while the intelligent rotating arm drives the second industrial camera to move according to the planned motion trajectory, the workpiece areas corresponding to the respective detection points on the top of the workpiece are photographed, thereby obtaining B image data of the top surface of the workpiece;

[0047] Step 3) Judge the color space of the image data. If it is a color image, it is directly converted into a grayscale image. Otherwise, the original image is retained, which is the grayscale image;

[0048] Step 4) Use an image processing algorithm to enhance a single sample of the grayscale image.

[0049] Step 4.1) According to the position of the workpiece in the gray-bottom image, use Equation (1) to perform multi-target rough segmentation on any grayscale image to obtain a segmented binary image:

[0050]

[0051] In formula (1), f(x, y) represents the pixel value of the grayscale image at the position point (x, y), b0 represents the black pixel value, b1 represents the white pixel value, and t represents the segmentation threshold; f′(x, y) represents the pixel value of the binary image after segmentation at the position point (x, y);

[0052] Step 4.2) Use the bilinear interpolation method to perform two linear interpolations on the segmented binary image in the x direction and then one linear interpolation in the y direction to obtain the standardized binary image. The specific steps are as follows:

[0053] First, f(x, y) is the image function after threshold processing. Assume that the values of four points f(x0, y0), f(x1, y1), f(x0, y1), and f(x1, y0) are known. These four points determine a rectangle, and the function value of any point within the rectangle is obtained through interpolation. First, perform two linear interpolations in the x direction to obtain:

[0054]

[0055] Then perform one linear interpolation in the y direction to obtain an image with a size of 299*299.

[0056]

[0057] Step 4.3) Perform data augmentation on the standardized binary image, including: mirroring and symmetry operations, scale transformation operations, rotation operations, and translation operations, so as to generate multiple sample images to complete the augmentation of a single sample;

[0058] Step 5) Use an image processing algorithm to generate an adversarial network GAN to perform single-sample augmentation on the grayscale image:

[0059] Construct a generative adversarial network GAN composed of a generator network G and a discriminator network D, and process any grayscale image to generate multiple sample images;

[0060] Step 6) Combine the multiple sample images generated in Step 4 and Step 5 into a sample library, and set corresponding class labels for each image sample in the sample library;

[0061] Step 7) As Figure 3As shown in the figure, a few-shot image recognition model based on the InceptionV3 network is constructed. The detection task for large workpiece targets is taken as a specific task, and a large number of task sets are constructed using easily obtainable large workpiece targets to train the model, so as to realize the category prediction of large workpiece targets in a few-shot scenario. Since the InceptionV3 network has a structure with multiple hidden layers, each hidden layer can perform a non-linear transformation on the output of the previous layer and can express complex functional relationships in a more concise way. Therefore, this network is used for part recognition. The few-shot image recognition model based on the InceptionV3 network includes: six convolutional layers, two pooling layers, a mixed layer, a fully connected layer, and a dropout layer; the specific steps are as follows:

[0062] Step 7.1) Collect M qualified images and N defective images according to the processes of Step 1 and Step 2, and set category labels for the total M + N images respectively.

[0063] Step 7.2 Enhance the M + N images according to the processes of Step 3 and Step 4 to obtain (M + N) × num image samples. After normalizing the (M + N) × num image samples using the bilinear interpolation method, a normalized image dataset is obtained. Let any one of the normalized image samples be denoted as F; where num represents the multiple; in this embodiment, Num = 100;

[0064] The InceptionV3 is divided into 3 blocks, corresponding to images with dimensions of 35×35, 17×17, and 8×8. There are many parts in the middle of each block. The adaptive pooling is 1×1×2048, corresponding to different feature layer depths for feature extraction, and a 1000-dimensional vector is output. The specific steps are as follows:

[0065] Any sample image F in the sample library is input into the few-shot image recognition model, and the first convolutional layer performs a convolution operation on the sample image F to obtain the downsampled image data F1;

[0066] The second convolutional layer performs a convolution operation on the image data F1 to obtain the downsampled image data F2;

[0067] The third convolutional layer performs a convolution operation on the image data F2 to obtain the downsampled image data F3;

[0068] The first pooling layer performs a max pooling operation on the image data F3 to obtain the pooled image data F4;

[0069] The fourth convolutional layer performs a convolution operation on the image data F4 to obtain the downsampled image data F5;

[0070] The fifth convolutional layer performs a convolution operation on the image data F5 to obtain the downsampled image data F6;

[0071] The sixth convolutional layer performs a sixth convolutional operation on the image data F6 to obtain the downsampled image data F7;

[0072] The mixing layer performs eight convolutional operations on the image data F7 to obtain the image feature data F8;

[0073] The second pooling layer performs a max pooling operation on the image feature data F8 to obtain the pooled image feature data F9;

[0074] The fully connected layer performs a fully connected operation on the image feature data F9 to obtain the image feature data T;

[0075] The dropout layer operates on the image feature data T to obtain the feature data G;

[0076] The softmax function is used to classify the feature data G to obtain the probability values of each class, and the class corresponding to the sample image F is taken as the class with the maximum probability value; after softmax, the probability values of G belonging to each class are obtained. The formula of the softmax function is shown in Equation (4), where the input is the result obtained from K different linear functions, and the probability that the feature data G belongs to the j-th classification is P.

[0077]

[0078] In step 8), with the goal of minimizing the cross-entropy loss function, the specific formula of the cross-entropy function is shown in Equation (5). In Equation (5), c is the cost function, G is the sample, y is the actual value, n is the output value, and n is the total number of samples. When the small sample image recognition module uses the InceptionV3 to train its own model and selects to use the cross-entropy loss function, the greater the error, the greater the adjustment amplitude of the weights and biases, and when the error becomes smaller, the adjustment amplitude of the weights and biases also becomes smaller.

[0079]

[0080] The small sample image recognition model is trained through the Adam optimization algorithm, and its network parameters are updated until the loss function converges, so as to obtain a trained small sample image recognition model for realizing the classification of the sample image to be detected.

[0081] In this embodiment, an electronic device includes a memory and a processor. The memory is used to store a program that supports the processor to execute the above method, and the processor is configured to execute the program stored in the memory.

[0082] In this embodiment, a computer-readable storage medium stores a computer program thereon, and when the computer program is run by a processor, it executes the steps of the above method.

Claims

1. A large workpiece detection method based on few-shot learning, characterized in that, Applied to the detection environment composed of a six-axis robotic arm and an intelligent rotating arm, the six-axis robotic arm is arranged on a free-lifting platform, and a first industrial camera is arranged on the six-axis robotic arm through a rotating cloud platform, which is used to rotate and photograph the side of the workpiece within a height range. The intelligent rotating arm is provided with a rotatable rotating column on the chassis, and a guide rail is arranged at the top of the rotating column. A second industrial camera is arranged on the guide rail through another rotating cloud platform. The second industrial camera moves on the guide rail and rotates and photographs the top of the workpiece; the large workpiece detection method is carried out according to the following steps: Step 1) For the side shape of the workpiece, according to the side detection requirements, set A detection points, and thus plan the movement trajectory of the six-axis robotic arm according to the A detection points, so that while the six-axis robotic arm drives the first industrial camera to move according to the planned movement trajectory, photograph the workpiece areas corresponding to each detection point on the side of the workpiece, thereby obtaining A image data of the side of the workpiece; Step 2) For the top shape of the workpiece, according to the top detection requirements, preset B detection points in advance, and thus plan the movement trajectory of the intelligent rotating arm according to the B detection points, so that while the intelligent rotating arm drives the second industrial camera to move according to the planned movement trajectory, photograph the workpiece areas corresponding to each detection point on the top of the workpiece, thereby obtaining B image data of the top surface of the workpiece; Step 3) Judge the color space of the image data. If it is a color image, directly convert it to a grayscale image. Otherwise, retain the original image, which is the grayscale image; Step 4) Use an image processing algorithm to enhance a single sample of the grayscale image: Step 4.1) According to the position of the workpiece in the grayscale image, use formula (1) to perform multi-target rough segmentation on any grayscale image to obtain a segmented binary image; (1) In formula (1), represents the pixel value of the grayscale image at the position point ; represents the black pixel value, represents the white pixel value, represents the segmentation threshold; represents the pixel value of the binary image after segmentation at the position point ; Step 4.2) Use the bilinear interpolation method to perform two linear interpolations on the segmented binary image in the x direction and then perform one linear interpolation in the y direction, thereby obtaining a standardized binary image; Step 4.3) Perform data enhancement processing on the standardized binary image, including: mirroring and symmetry operations, scale transformation operations, rotation operations, and translation operations, thereby generating multiple sample images to complete the enhancement of a single sample; Step 5) Use an image processing algorithm to generate an adversarial network GAN to enhance a single sample of the grayscale image: Construct a generative adversarial network GAN composed of a generative network G and a discriminative network D, and process any grayscale image to generate multiple sample images; Step 6) Combine the multiple sample images generated in Step 4 and Step 5 into a sample library, and set corresponding class labels for each image sample in the sample library; Step 7) Construct a small-sample image recognition model based on the InceptionV3 network, including: six convolutional layers, two pooling layers, a mixed layer, a fully connected layer, and a dropout layer; Any sample image F in the sample library is input into the small-sample image recognition model, and the first convolutional layer performs a convolutional operation on the sample image F to obtain the downsampled image data F1; The second convolutional layer performs a convolution operation on the image data F1 to obtain the downsampled image data F2; The third convolutional layer performs a convolution operation on the image data F2 to obtain the downsampled image data F3; The first pooling layer performs a max pooling operation on the image data F3 to obtain the pooled image data F4; The fourth convolutional layer performs a convolution operation on the image data F4 to obtain the downsampled image data F5; The fifth convolutional layer performs a convolution operation on the image data F5 to obtain the downsampled image data F6; The sixth convolutional layer performs a sixth convolution operation on the image data F6 to obtain the downsampled image data F7; The mixing layer performs eight convolution operations on the image data F7 to obtain the image feature data F8; The second pooling layer performs a max pooling operation on the image feature data F8 to obtain the pooled image feature data F9; The fully connected layer performs a fully connected operation on the image feature data F9 to obtain the image feature data T; The dropout layer operates on the image feature data T to obtain the feature data G; The softmax function is used to classify the feature data G to obtain the probability values of each class, and the class corresponding to the sample image F is determined by the maximum probability value; In step 8), with the goal of minimizing the cross-entropy loss function, the small-sample image recognition model is trained by the Adam optimization algorithm, and its network parameters are updated until the loss function converges, thereby obtaining a trained small-sample image recognition model for realizing the class recognition of the sample image to be detected.

2. An electronic device, comprising a memory and a processor, characterized in that, The memory is used to store a program for supporting the processor to execute the large workpiece detection method described in claim 1, and the processor is configured to execute the program stored in the memory.

3. A computer-readable storage medium, on which a computer program is stored, characterized in that, When the computer program is run by the processor, it executes the large workpiece detection method described in claim 1.

Images