Hydropower maintenance tool sorting method based on SqueezeNet convolutional neural network model

Abstract

The invention discloses a hydroelectric overhaul tool sorting method based on a SqueezeNet convolutional neural network model. The method is suitable for various tools such as wrenches, screwdrivers and sleeves. The method comprises the following steps: firstly, acquiring a maintenance tool image in a backlight environment, and removing background interference through an image preprocessing technology to generate a clear target binary image; in order to solve the problem of insufficient samples, data enhancement means such as expansion, corrosion and rotation are adopted to expand a training set. And then, deep learning training is carried out by using a SqueezeNet model, parameters are adjusted in combination with a cross entropy loss function and a gradient descent optimization algorithm, and finally, a high-precision recognition model is obtained. In addition, hand-eye calibration of the visual system and the mechanical arm is completed through a nine-point calibration method, and accurate conversion from image coordinates to mechanical arm action coordinates is ensured. In actual operation, stacked tools are separated through the flexible vibration disc, then the visual system collects images of the single tools, the types and the positions of the single tools are recognized through the model, and the mechanical arm is guided to conduct accurate grabbing.

Description

Technical Field

[0001] The invention belongs to the field of machine vision, and in particular relates to a water and electricity maintenance tool sorting method based on a SqueezeNet convolutional neural network model. Background Art

[0002] In the daily maintenance of turbine units, various maintenance tools are frequently used. Therefore, orderly management of maintenance tools is an important part of improving maintenance efficiency. And the sorting of maintenance tools is a prerequisite for orderly management of maintenance tools.

[0003] Existing sorting methods usually rely on manual operations, which have problems of high error rate and low efficiency. The sorting method based on 3D vision has the problem of high investment cost. With the development of industrial technology, there is an urgent need for a high-efficiency, high-accuracy, low-cost and highly automated method to sort and repair tools. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide a water and electricity maintenance tool sorting method based on the SqueezeNet convolutional neural network model, so as to solve the problems of high sorting error rate, low efficiency and high cost in the prior art, and achieve the goals of improving sorting efficiency, improving sorting accuracy and reducing costs.

[0005] In order to solve the above technical problems, the technical solution adopted by the present invention is: The water and electricity maintenance tool sorting method based on the SqueezeNet convolutional neural network model has the following steps: Step S1: collecting images of the maintenance tool in a backlight environment and performing preprocessing operations; Step S2: preprocessing the data set; Step S3: training a maintenance tool recognition model M0 based on a SqueezeNet convolutional neural network; Step S4: Use the visual system Cam and the robot arm Rob to perform hand-eye calibration and calculate the affine transformation matrix Mat0; Step S5: Separating the stacked maintenance tools using a flexible vibration plate; Step S6: collecting an image of the maintenance tool in a backlight environment to obtain a pre-processed image I0; Step S7: using the maintenance tool recognition model M0, identifying the maintenance tool T0 in the image I0, and calculating its center coordinate P0; Step S8: Calculate the target position P1 of the robot arm using the center coordinate P0 of the maintenance tool T0 and the relationship matrix Mat0.

[0006] Preferably, in step S1, image difference and mean filtering algorithms are first used to remove image background and noise interference, and then the maximum inter-class variance method is used to extract the target area to generate a binary image.

[0007] Preferably, in step S2, morphological algorithms including image dilation and erosion and geometric transformation algorithms including image rotation are used to randomly process the binary image in step S1 to expand the data set; the expanded data set is divided into a training set, a validation set and a test set according to the ratio of 70%, 15% and 15%.

[0008] Preferably, in step S3, a learning rate decay strategy is adopted to train the SqueezeNet convolutional neural network-based maintenance tool identification model M0, that is, the entire training process is divided into 4 stages, and the learning rate decays by 0.5 times in sequence.

[0009] Preferably, the SqueezeNet convolutional neural network uses convolutional layers conv1 and conv10 as the starting layer and the ending layer respectively, and is stacked with Fire modules in the middle; each Fire module includes two parts, squeeze and expand; During the training process, the input image is propagated forward through the network structure of the SqueezeNet model to obtain the model prediction result. ; As shown in formula (1), the cross entropy loss function can be used to calculate the prediction result and the true label The loss between: (1); According to the back propagation algorithm, find the gradient of the loss to the i-th layer model , ; As shown in formula (2), the gradient descent optimization algorithm is used to update the model parameters: (2); in, is the learning rate, , are the weight matrix and bias vector of the i-th layer respectively; Through multiple iterations of the above process, and after each training round, the model M is tested on the validation set. n The evaluation is performed until all training rounds are completed; then, the maintenance tool identification model M0 with the best prediction result is selected.

[0010] Preferably, in step S4, the visual system Cam and the robot arm Rob perform hand-eye calibration and calculate the affine transformation matrix Mat0; the method is as follows: The purpose of hand-eye calibration is to calculate the affine transformation matrix of the coordinate system from the visual system Cam to the robot arm Rob; the nine-point calibration method is used, and the implementation method is as follows: The robot arm Rob moves nine points in sequence, and the visual system Cam also moves the coordinates of the above points in sequence and records them according to the corresponding relationship; As shown in formula (3), using the least squares algorithm, the matrix Mat0 can be obtained according to the above coordinate group; (3); In the formula, , are the X and Y coordinates of the i-th point collected in the robot arm Rob coordinate system; , are the X and Y coordinates of the i-th point collected in the Cam coordinate system of the visual system; where i is an integer, and .

[0011] Preferably, in step S5, a flexible vibration plate is used to vibrate the stacked maintenance tools, and algorithms including image difference, mean filtering and maximum inter-class variance method are used to obtain the pre-processed image I0.

[0012] Preferably, step S6: collecting an image of the maintenance tool in a backlight environment to obtain a preprocessed image I0; collecting an image of the maintenance tool in a backlight environment, and obtaining a binary image I0 of the maintenance tool using the same method as in step S1.

[0013] Preferably, step S7: using the inspection tool recognition model M0, identifying the inspection tool T0 in the image I0, and calculating the center coordinate P0; according to the image I0 obtained in step S6, inputting the model M0 described in step S4 o In the above example, the specific category of the maintenance tool in the image I0 can be calculated; at the same time, the threshold segmentation algorithm can be used to extract the target area and calculate its center coordinate P0.

[0014] Preferably, step S8: calculating the target position P1 of the robot arm through the center coordinate P0 of the maintenance tool T0 and the relationship matrix Mat0; As shown in formula (4), according to Mat0 in step S5 and coordinate P0 in step S7, the grasping position P1 in the robot base coordinate system can be obtained; (4).

[0015] The present invention can achieve the following beneficial effects: 1. By adopting the SqueezeNet convolutional neural network model and combining image preprocessing technologies such as image acquisition in backlight environments, differential algorithms, and mean filtering, the outline of the maintenance tool can be clearly captured, reducing the impact of metal surface reflections, thereby improving the accuracy of maintenance tool recognition.

[0016] 2. Use data augmentation techniques (such as dilation, erosion, rotation, etc.) to increase the number of samples, making the training set more diverse, which helps prevent overfitting and improves the model's generalization ability under different conditions.

[0017] 3. The model training is carried out by combining the gradient descent optimization algorithm with the learning rate decay strategy, which can not only accelerate the convergence speed, but also improve the final model performance and ensure the optimal maintenance tool identification model.

[0018] 4. By performing hand-eye calibration between the visual system Cam and the robotic arm Rob and calculating the affine transformation matrix Mat0, the identified target position can be accurately converted into action coordinates executable by the robotic arm, thus achieving an effective transformation from visual information to physical operations.

[0019] 5. The entire process involves multiple links such as automated image processing, target recognition, and robot control, which greatly reduces the need for human intervention, improves work efficiency, and reduces the possibility of human errors. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The present invention will be further described below in conjunction with the accompanying drawings and embodiments: Figure 1 It is a flow chart of the present invention; Figure 2 The SqueezeNet network structure diagram of the present invention; Figure 3 This is a structural diagram of the Fire module of the present invention; Figure 4 The present invention is a pre-processed maintenance tool; Figure 5 This is an identification effect diagram of the maintenance tool of the present invention. DETAILED DESCRIPTION

[0021] The present invention selects some water turbine maintenance tools as examples for illustration, including: wrenches, screwdrivers, and sockets. However, the present invention is not limited to the above examples. The water and electricity maintenance tool sorting method based on the SqueezeNet convolutional neural network model includes the following steps: Step S1: collecting images of the maintenance tool in a backlight environment and performing preprocessing operations; The purpose of collecting images in a backlit environment is to clearly image the contour of the maintenance tool without being affected by the reflection of the metal surface. Then, the image difference and mean filtering algorithms are combined to remove the image background and noise interference. Finally, the maximum inter-class variance method can be used to stably segment the target area, and the rest is filled with 0 to generate a binary image of the maintenance tool.

[0022] Step S2: Dataset preprocessing; In order to prevent the model from overfitting due to too few samples, a large number of samples need to be collected to train the convolutional neural network model. However, collecting image data also takes a lot of time and effort. Therefore, the binary image in step S1 is enhanced to increase the number of samples. The above data enhancement mainly adopts the following methods: Morphological algorithms such as dilation and erosion: For each binary image in step S1, dilation and erosion are performed randomly to expand the data set by 2 times; Image rotation: Each binary image in step S1 is rotated by 30°, 60°, 90°, 120°, 150°, and 180° to expand the data set by 6 times.

[0023] After completing the data set expansion, the data set is divided into training set, validation set and test set according to the ratio of 70%, 15% and 15%.

[0024] Step S3: training a maintenance tool recognition model M0 based on a SqueezeNet convolutional neural network; like Figure 2 As shown in the figure, the SqueezeNet convolutional neural network uses convolutional layers conv1 and conv10 as the starting layer and the ending layer, respectively, and is stacked with Fire modules in the middle (fire2~fire9). Figure 3 As shown, each Fire module consists of two parts: squeeze and expand.

[0025] During the training process, the input image is propagated forward through the network structure of the SqueezeNet model to obtain the model prediction result. As shown in formula (1), the prediction result can be calculated using the cross entropy loss function and the true label The loss between loss.

[0026] (1); According to the back propagation algorithm, the gradient of the loss to the i-th layer model can be calculated , .

[0027] As shown in formula (2), the gradient descent optimization algorithm is used to update the model parameters.

[0028] (2); in, is the learning rate, , are the weight matrix and bias vector of the i-th layer respectively.

[0029] Through multiple iterations of the above process, and after each training round, the model M is tested on the validation set. n The evaluation is performed until all training rounds are completed. Then, the maintenance tool identification model M0 with the best prediction result is selected.

[0030] In order to obtain better training results, a learning rate decay strategy was adopted, that is, the initial learning rate was preset to 0.001, the number of training rounds was 200, and the training rounds were divided into 4 stages, and the learning rate decayed by 0.5 times.

[0031] Step S4: The visual system Cam and the robot arm Rob perform hand-eye calibration and calculate the affine transformation matrix Mat0; The purpose of hand-eye calibration is to calculate the affine transformation matrix of the coordinate system from the visual system Cam to the robot arm Rob. The present invention adopts a nine-point calibration method, and the specific implementation method is as follows: The robot arm Rob moves nine points in sequence, and the visual system Cam also obtains the coordinates of the above points in sequence and records them according to the corresponding relationship.

[0032] As shown in Formula 3, the matrix Mat0 can be obtained according to the above coordinate group using the least squares algorithm.

[0033] (3); Step S5: separate the flexible vibration discs from the stacked state maintenance tool; A flexible vibration plate is used to shake the stacked maintenance tools apart, and the scattered single maintenance tools are the tools to be sorted.

[0034] It can be understood that steps S1-S4 are preliminary preparation processes for implementing the method, and the tool sorting task does not need to be repeated.

[0035] Step S6: collecting an image of the maintenance tool in a backlight environment to obtain a pre-processed image I0; The image of the maintenance tool in the backlight environment is collected, and the binary image I0 of the maintenance tool is obtained by the same method as in step S1. The specific implementation effect is as follows: Figure 4 shown.

[0036] Step S7: using the maintenance tool recognition model M0, identifying the maintenance tool T0 in the image I0, and calculating the center coordinate P0; According to the image I0 obtained in step S6, the model M described in step S4 is input o , the specific category of the maintenance tool in image I0 can be calculated.

[0037] At the same time, the threshold segmentation algorithm can be used to extract the target area and calculate its center coordinate P0.

[0038] Specific implementation effects such as Figure 5 shown.

[0039] Step S8: Calculate the target position P1 of the robot arm through the center coordinate P0 of the maintenance tool T0 and the relationship matrix Mat0.

[0040] As shown in formula (4), according to Mat0 in step S5 and coordinate P0 in step S7, the grasping position P1 in the robot base coordinate system can be obtained.

[0041] (4).

[0042] The above embodiments are only preferred technical solutions of the present invention and should not be regarded as limiting the present invention. The protection scope of the present invention shall be the technical solutions recorded in the claims, including equivalent replacement solutions of the technical features in the technical solutions recorded in the claims. That is, equivalent replacement improvements within this scope are also within the protection scope of the present invention.

Claims

1. A water and electricity maintenance tool sorting method based on the SqueezeNet convolutional neural network model, characterized in that The following steps are involved: Step S1: collecting images of the maintenance tool in a backlight environment and performing preprocessing operations; Step S2: preprocessing the data set; Step S3: training a maintenance tool recognition model M0 based on a SqueezeNet convolutional neural network; Step S4: Use the visual system Cam and the robot arm Rob to perform hand-eye calibration and calculate the affine transformation matrix Mat0; Step S5: Separating the stacked maintenance tools using a flexible vibration plate; Step S6: collecting an image of the maintenance tool in a backlight environment to obtain a pre-processed image I0; Step S7: using the maintenance tool recognition model M0, identifying the maintenance tool T0 in the image I0, and calculating its center coordinate P0; Step S8: Calculate the target position P1 of the robot arm using the center coordinate P0 of the maintenance tool T0 and the relationship matrix Mat0.

2. The water and electricity maintenance tool sorting method based on the SqueezeNet convolutional neural network model according to claim 1 is characterized in that: In step S1, the image difference and mean filtering algorithms are first used to remove the image background and noise interference, and then the maximum inter-class variance method is used to extract the target area to generate a binary image.

3. The water and electricity maintenance tool sorting method based on the SqueezeNet convolutional neural network model according to claim 2 is characterized in that: In step S2, the binary image in step S1 is randomly processed using morphological algorithms including image dilation and erosion and geometric transformation algorithms including image rotation to expand the data set; the expanded data set is divided into a training set, a validation set, and a test set according to the ratio of 70%, 15%, and 15%.

4. The method for sorting water and electricity maintenance tools based on the SqueezeNet convolutional neural network model according to claim 1 is characterized in that: In step S3, a learning rate decay strategy is adopted to train the SqueezeNet convolutional neural network-based maintenance tool identification model M0, that is, the entire training process is divided into 4 stages, and the learning rate decays by 0.5 times.

5. The method for sorting water and electricity maintenance tools based on the SqueezeNet convolutional neural network model according to claim 4 is characterized in that: The SqueezeNet convolutional neural network uses convolutional layers conv1 and conv10 as the starting and ending layers, respectively, with Fire modules stacked in the middle; each Fire module consists of two parts: squeeze and expand; During the training process, the input image is propagated forward through the network structure of the SqueezeNet model to obtain the model prediction result. ; As shown in formula (1), the prediction result can be calculated using the cross entropy loss function and the true label The loss between: （1）； According to the back propagation algorithm, find the gradient of the loss to the i-th layer model , ; As shown in formula (2), the gradient descent optimization algorithm is used to update the model parameters: （2）； in, is the learning rate, , are the weight matrix and bias vector of the i-th layer respectively; Through multiple iterations of the above process, and after each training round, the model M is tested on the validation set. n The evaluation is performed until all training rounds are completed; then, the maintenance tool identification model M0 with the best prediction result is selected.

6. The method for sorting water and electricity maintenance tools based on the SqueezeNet convolutional neural network model according to claim 1 is characterized in that: In step S4, the visual system Cam and the robot arm Rob perform hand-eye calibration and calculate the affine transformation matrix Mat0; the method is as follows: The purpose of hand-eye calibration is to calculate the affine transformation matrix of the coordinate system from the visual system Cam to the robot arm Rob; the nine-point calibration method is used, and the implementation method is as follows: The robot arm Rob moves nine points in sequence, and the visual system Cam also moves the coordinates of the above points in sequence and records them according to the corresponding relationship; As shown in formula (3), using the least squares algorithm, the matrix Mat0 can be obtained according to the above coordinate group; （3）； In the formula, , are the X and Y coordinates of the i-th point collected in the robot arm Rob coordinate system; , are the X and Y coordinates of the i-th point collected in the Cam coordinate system of the visual system; where i is an integer, and .

7. The method for sorting water and electricity maintenance tools based on the SqueezeNet convolutional neural network model according to claim 1 is characterized in that: In step S5, the stacked maintenance tools are vibrated apart using a flexible vibration plate, and algorithms including image difference, mean filtering and maximum inter-class variance method are used to obtain a pre-processed image I0.

8. The method for sorting water and electricity maintenance tools based on the SqueezeNet convolutional neural network model according to claim 1 is characterized in that: Step S6: Capture an image of the maintenance tool in a backlight environment to obtain a preprocessed image I0; capture an image of the maintenance tool in a backlight environment, and use the same method as in step S1 to obtain a binary image I0 of the maintenance tool.

9. The method for sorting water and electricity maintenance tools based on the SqueezeNet convolutional neural network model according to claim 8 is characterized in that: Step S7: Use the inspection tool recognition model M0 to recognize the inspection tool T0 in the image I0 and calculate the center coordinate P0; according to the image I0 obtained in step S6, input the model M0 described in step S4 o In the above example, the specific category of the maintenance tool in the image I0 can be calculated; at the same time, the threshold segmentation algorithm can be used to extract the target area and calculate its center coordinate P0.

10. The method for sorting water and electricity maintenance tools based on the SqueezeNet convolutional neural network model according to claim 9 is characterized in that: Step S8: Calculate the target position P1 of the robot arm through the center coordinate P0 of the maintenance tool T0 and the relationship matrix Mat0; As shown in formula (4), according to Mat0 in step S5 and coordinate P0 in step S7, the grasping position P1 in the robot base coordinate system can be obtained; （4）。

Images