Steel bar bending angle detection system and method based on machine vision, storage medium and computer program product

Abstract

The invention discloses a reinforcing steel bar bending angle detection system and method based on machine vision, a storage medium and a computer program product. The method comprises the following steps: segmenting an acquired reinforcing steel bar image based on an image segmentation model of deep learning to obtain a binary image only containing reinforcing steel bars or an image with reinforcing steel bar contour information; and detecting the binary image only containing the reinforcing steel bar or the image with the reinforcing steel bar contour information to obtain the edge of the reinforcing steel bar and fit a straight line. According to the invention, the bending angle of the reinforcing steel bar can be automatically detected based on machine vision. The image segmentation module can accurately separate out a reinforcing steel bar image, eliminate background interference and provide a basis for subsequent accurate detection; the steel bar contour detection module fits a steel bar edge straight line to facilitate quantitative analysis; the included angle calculation module calculates the bending angle based on the fitting straight line, the detection accuracy and automation degree are improved through the modular design, and compared with traditional manual detection, the efficiency and precision are higher.

Description

Technical Field

[0001] The present invention belongs to the technical field of quality inspection of semi-finished steel bars, and particularly relates to a steel bar bending angle detection system, method, storage medium and computer program product based on machine vision. Background Art

[0002] In the current construction industry, numerous platforms for managing the construction process have been built. For example, the steel bar cloud factory platform can achieve refined control of materials, support the control of steel bar progress and cost, but lacks the management of the quality of steel bar processing. At present, there is a lack of effective control means for the quality management of semi-finished steel bars. In the production process, manual marking or making special molds are usually used to control the quality. Moreover, the conventional manual sampling method is often not implemented in place, and the sampling method cannot cover all semi-finished steel bars, resulting in unqualified semi-finished products being delivered out of the factory, thus affecting the on-site construction quality. Summary of the Invention

[0003] To improve the quality of semi-finished steel bar processing, the present invention proposes a steel bar bending angle detection system, method, storage medium and computer program product based on machine vision.

[0004] A steel bar bending angle detection system based on machine vision for achieving one of the purposes of the present invention includes:

[0005] An image segmentation module: used to segment the collected steel bar image based on a deep learning-based image segmentation model to obtain a binary image containing only the steel bar or an image with steel bar contour information;

[0006] A steel bar contour detection module: used to detect the binary image containing only the steel bar or the image with steel bar contour information to obtain the edge of the steel bar and fit a straight line;

[0007] An included angle calculation module: used to calculate the intersection angle of the fitted straight lines to obtain the bending angle of the steel bar.

[0008] Furthermore, the system further includes a model training module for training the deep learning-based image segmentation model, and the training method includes:

[0009] Collect multiple bending images covering various steel bar bending situations during the production process of semi-finished steel bars, and divide the images into a training set, a validation set and a test set;

[0010] Import the training set and validation set images into an interactive segmentation annotation software, and load an annotation model to assist in annotation; use the polygon annotation function of the software to outline the contour of the bent steel bar; after the annotation is completed, the software generates an image file that meets the input format requirements of the first image segmentation model for recording image classification information;

[0011] Import the training set and validation set images into the graphic image annotation tool, and select the polygon annotation function to annotate the profile of the bent steel bar; after the annotation is completed, the tool generates an annotation file that meets the input requirements of the second image segmentation model for saving the steel bar annotation information;

[0012] Sort out the annotated data sets respectively. According to the input requirements of the image segmentation model, place the image files and annotation files in the corresponding folder structure to ensure that the model can correctly read the data during training;

[0013] Augment the images in the annotated data set and then expand the number of images;

[0014] Use the annotated files and the test set to train and test the first image segmentation model and the second image segmentation model to obtain the final image segmentation model.

[0015] The final image segmentation model can combine the prediction results of the two models in a weighted average manner, and assign different weights to them according to their performance on different test subsets, but this is not limited to this fusion method.

[0016] The beneficial effects of the above model training module include: collecting images covering various steel bar bending situations and dividing the data set, generating files that meet the model input requirements through annotation with different software, and then performing data augmentation and model training and testing, so that the model can better learn the characteristics of steel bar bending, improve the generalization ability and adaptability of the model, thereby improving the accuracy of image segmentation and further ensuring the accuracy of steel bar bending angle detection.

[0017] A further technical solution includes that the first image segmentation model is a model based on PaddleSeg, and the method for training it includes:

[0018] Configure the model parameters in the yml file, write the paths of the training set and validation set, and then start the training of the network model; the training uses the SGD stochastic gradient descent algorithm, and set the number of training epochs, the initial learning rate of the network and the size of the training batch_size; after the training is completed, start the script file to call the trained model to predict the test set and obtain the segmentation result of the bent steel bar.

[0019] The second image segmentation model is a model based on YOLOv8-seg, and the method for training it includes:

[0020] Configure model parameters in the yaml file, fill in the paths of the training set and validation set and the detection type, select the yolov8-seg.yaml network structure and start training, set the number of training epochs, the initial learning rate of the network and the training batch_size; after training, obtain the second image segmentation model based on YOLOv8-seg that has been trained, start the script file to predict the test set, and obtain the segmentation results of the bent steel bars.

[0021] Furthermore, the system also includes an image preprocessing module for preprocessing the collected steel bar images to improve the image quality; the preprocessing includes: image grayscale conversion and / or Gaussian filtering and / or bubble-type noise removal; the grayscale conversion includes converting a color image into a grayscale image to reduce the amount of data; the Gaussian filtering is used to smooth the image and remove high-frequency noise in the image; the bubble-type noise removal is used to find and remove possible bubble-type noise in the image to make the contour of the steel bar clearer.

[0022] The technical effects of the above image preprocessing module include: the image preprocessing module grayscales the collected steel bar images to reduce the amount of data and improve the subsequent processing speed; Gaussian filtering removes high-frequency noise and enhances the image stability; bubble-type noise removal makes the steel bar contour clearer. These preprocessing operations effectively improve the image quality, provide more accurate data for subsequent steel bar contour detection and bending angle calculation, and improve the reliability of the detection results.

[0023] Even further, the steel bar contour detection module includes:

[0024] Edge detection module: used to detect the edges of the steel bars in the image to obtain an edge image;

[0025] Line detection module: used to transform the edge image, detect the lines in the image, and calculate the inclination angle of each line;

[0026] Line grouping and screening module: used to group the detected lines according to the inclination angles of the lines and select the optimal lines as the edges of the steel bars.

[0027] The beneficial effects of the above steel bar contour detection module include: the edge detection module extracts the edges of the steel bars, the line detection module detects the lines and calculates the angles, and the line grouping and screening module groups according to the angles and selects the optimal lines. This modular and subdivided design improves the accuracy and efficiency of steel bar contour detection, can more accurately determine the edges of the steel bars, and provides a reliable basis for accurately calculating the bending angles.

[0028] Even further, the method for selecting the optimal lines includes:

[0029] According to the inclination angles of each obtained straight line, the straight lines with an angle difference less than the set angle threshold are grouped into one group;

[0030] Select the top two groups with the most straight lines; select the straight line with the angle closest to the median within the group as the optimal straight line for this group.

[0031] The beneficial effects of the above method for selecting the optimal straight line include: effectively filtering out noise and irrelevant straight lines, selecting more representative straight lines to represent the edges of the steel bars, making the calculation of the bending angle of the steel bars more accurate, and thus improving the detection accuracy of the entire steel bar bending angle detection system.

[0032] Furthermore, the method for calculating the inclination angle of each straight line includes:

[0033] After edge detection, use the probabilistic Hough transform to detect straight lines in the image, and then calculate the inclination angle of each straight line through the endpoint coordinates of the straight line.

[0034] Furthermore, the method for obtaining the edge image includes:

[0035] Apply the Canny edge detection algorithm to extract edge information in the image; the gradient calculation formula of the Canny edge detection algorithm includes:

[0036]

[0037] where G x and G y are the gradients of the image in the horizontal and vertical directions respectively;

[0038] Calculate the gradient magnitude and gradient direction of each pixel in the image;

[0039] Detect significant edges through the double-threshold method. Pixels above the upper threshold are considered edges, and pixels below the lower threshold are ignored; finally, an edge image is generated, where the area with the pixel value of the first extreme value represents the detected edge, and the pixel values of other areas are the second extreme value.

[0040] A steel bar bending angle detection system based on machine vision for achieving the second object of the present invention includes:

[0041] Segment the collected steel bar image using a deep learning-based image segmentation model to obtain a binary image containing only the steel bars or an image with steel bar contour information;

[0042] Detect the binary image containing only the steel bars or the image with steel bar contour information to obtain the edges of the steel bars and fit straight lines;

[0043] Calculate the intersection angle of the fitted straight lines to obtain the bending angle of the steel bars.

[0044] A non-transitory computer-readable storage medium for achieving the third object of the present invention, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the above-mentioned method for detecting the bending angle of steel bars based on machine vision are implemented.

[0045] A computer program product for achieving the fourth object of the present invention, including a computer program / instructions, and when the computer program / instructions are executed by a processor, the steps of the above-mentioned method for detecting the bending angle of steel bars based on machine vision are implemented.

[0046] The beneficial effects of the present invention include:

[0047] Through the collaborative work of the image segmentation module, the steel bar contour detection module, and the included angle calculation module, the present invention can automatically detect the bending angle of steel bars based on machine vision. Among them, the image segmentation module is based on a deep learning image segmentation model, which can accurately separate the steel bar image, eliminate background interference, and provide a basis for subsequent accurate detection; the steel bar contour detection module fits the straight line of the steel bar edge for quantitative analysis; the included angle calculation module calculates the bending angle based on the fitted straight line. This modular design improves the accuracy and automation of detection, and has higher efficiency and accuracy compared with traditional manual detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 is a schematic flowchart of an embodiment of the method of the present invention;

[0049] Figure 2 is a structural diagram of a dataset of bent steel bars. DETAILED DESCRIPTION OF THE INVENTION

[0050] The following detailed description is used to explain the technical solution of the claims of the present invention so that those skilled in the art can understand the claims. The protection scope of the present invention is not limited to the following specific implementation structures. Those made by those skilled in the art that include the technical solution of the claims of the present invention and are different from the following detailed description are also within the protection scope of the present invention.

[0051] An embodiment of the present invention provides a method for detecting the bending angle of steel bars based on machine vision, Figure 1 as shown in the following steps:

[0052] S1. Data collection and dataset annotation

[0053] Data collection: Use a high-definition camera to obtain the images of the bent steel bars during the production process of semi-finished steel bars. The images are required to be in a lossless compression format. A total of more than 1000 images of various bent steel bars are obtained and divided into a training set, a validation set, and a test set according to the ratio of 7:2:1. Among them, 70% of the images are used as the training set to learn the bending characteristics of the steel bars by the model; 20% of the images are used as the validation set to evaluate the model performance and adjust the hyperparameters during the training process; the remaining 10% of the images are used as the test set to finally evaluate the accuracy of the trained model.

[0054] Data annotation:

[0055] I. Annotation operation based on the PaddleSeg model: Import the divided training set and validation set images into the EISeg interactive segmentation annotation software. At the same time, load the high-precision annotation model hrnet18s_ocr48_cocolvis into the EISeg software to assist the annotation work. Use the polygon annotation function of the EISeg software to carefully outline the contour of the bent steel bars in each image. Note that when annotating, if there are other steel bars in the image background, use image processing software for occlusion processing to ensure that there are no other steel bar targets in the image except the bent steel bars. At the same time, because the color of the steel bars is similar to that of the bending machine, special attention should be paid to distinguishing the contour of the steel bars. After the annotation is completed, the software will automatically generate label files that meet the input format requirements of the PaddleSeg model. These label files record the image classification information in the form of binary pixel values (for example, pixel value 0 represents the background, pixel value 1 represents the steel bars, etc., and the specific categories are determined according to the actual annotation situation).

[0056] II. Annotation operation based on the YOLOv8-seg model

[0057] Import the divided training set and validation set into the graphic image annotation tool labelme, and use the polygon annotation function of this annotation tool to annotate the contour of the bent steel bars. Similarly, if there are other steel bars in the image background, use image processing software for occlusion processing to avoid interference. Because the color of the steel bars is similar to that of the bending machine, special attention should be paid to distinguishing the contour of the steel bars during annotation to improve the accuracy of the trained segmentation model. After the annotation is completed, labelme will generate a JSON format annotation file that meets the input format requirements of the YOLOv8-seg model, recording the annotation information of the steel bars.

[0058] Data Arrangement: Arrange the datasets labeled based on the PaddleSeg model and the YOLOv8-seg model respectively. Place the image files and their corresponding annotation files in the appropriate folder structure to ensure that the data can be correctly read during model training. For the PaddleSeg model, the dataset structure should meet its input requirements. Generally, the image and label files are stored in different folders respectively, and the file names have a corresponding relationship. For the YOLOv8-seg model, organize the images and JSON annotation files according to its specified dataset structure. For example, create images and labels folders under the main folder to store the corresponding files. The structure of the bent steel bar dataset is as Figure 2 shown.

[0059] S2. Train the segmentation model

[0060] Data Augmentation: First, randomly apply data augmentation methods such as rotation, adding noise, color jitter, random lighting, and simulating raindrops to the images in the dataset, expanding the number of dataset images from the original 1000 to 2400.

[0061] Train the PaddleSeg model: Configure the model-related parameters in the yml file, write the paths of the training set and the validation set, and start training the PaddleSeg model on the GPU of GeForce GTX 2080Ti in the Ubuntu20.04 system. During the training process, use the SGD (Stochastic Gradient Descent) algorithm. In the experiment, set the number of training epochs to 500, the initial learning rate of the network to 0.001, and the training batch_size to 4. After training, obtain the model model.pdparams, and start the predict.py file to use the PaddleSeg model to predict the test set and obtain the segmentation results of the bent steel bars.

[0062] Train the YOLOv8-seg model: Configure the skinning.yaml file to fill in the paths of the training set, the validation set, and the detection type, select the yolov8-seg.yaml network structure, and start training the YOLOv8-seg model on the GPU of GeForceGTX 2080Ti in the Ubuntu20.04 system. In the experiment, set the number of training epochs to 800, the initial learning rate of the network to 0.001, and the training batch_size to 4. Finally, after training, obtain the model best.pt, and start the predict.py file to use the best.py model to predict the test set and obtain the segmentation results of the bent steel bars.

[0063] Test set result evaluation: After using the model model.pdparams and the model best.pt to make predictions on the test set, comprehensively analyze the prediction results. Calculate various evaluation metrics of the model on the test set, such as accuracy, recall, F1 value, etc. For the task of detecting the bending angle of steel bars, the accuracy reflects the proportion of the model correctly identifying the bending angle of steel bars, the recall reflects the ability of the model to detect all actual bent steel bars, and the F1 value comprehensively considers the accuracy and recall, which can more comprehensively evaluate the performance of the model.

[0064] Model adjustment: If the model shows a low accuracy or an unsatisfactory recall on the test set, different adjustment strategies are adopted according to specific problems. If it is an overfitting problem, that is, the model performs well on the training set but poorly on the test set, try to reduce the model complexity, such as pruning the number of network layers, reducing the number of convolutional kernels, etc., and at the same time increase the regularization term, such as L1 or L2 regularization, to prevent the model from overfitting to the noise in the training data. If there is an underfitting problem, that is, the model performs poorly on both the training set and the test set, then it is necessary to increase the model complexity, such as increasing the number of network layers, using a more complex convolutional neural network structure, or adjusting hyperparameters, such as appropriately increasing the learning rate, increasing the number of training epochs, etc., to enable the model to better learn the bending characteristics of steel bars.

[0065] Cross-validation and model fusion: To further improve the stability and generalization ability of the model, the cross-validation method is adopted. Divide the test set into multiple subsets, perform model training and evaluation multiple times, each time using a different subset as the validation set, and select the optimal model parameters by integrating the results of multiple times. Obtain the final first image segmentation model based on PaddleSeg and the second image segmentation model based on YOLOv8-seg. At the same time, consider fusing the models, such as combining the prediction results of the two models in a weighted average manner, and assigning different weights to them according to their performance on different test subsets to give full play to the advantages of the two models and improve the performance of the final model.

[0066] S3. Image preprocessing

[0067] Save the segmentation results of the bent steel bars obtained by image segmentation to the pseudo_color_prediction file, and perform image preprocessing operations on the images in this file.

[0068] The first step is to load the image as a grayscale image, and the grayscale image is denoted as I gray (x,y):

[0069] I gray (x,y) = 0.299·I r (x,y) + 0.587·I g (x,y) + 0.114·Ib (x, y)

[0070] Among them, I r (x, y), I g (x, y), I b (x, y) represents the red, green, and blue channels of the image at the position (x, y).

[0071] Second, use a Gaussian filter to smooth the image. The parameter is a 5x5 convolution kernel to remove high-frequency noise in the image and improve the accuracy of subsequent edge detection. The Gaussian smoothing process is expressed in the following form:

[0072]

[0073] I s (x, y) represents the pixel value of the image at the coordinate (x, y) after Gaussian smoothing; is a two-dimensional Gaussian function, which is the core of the Gaussian filter; σ (sigma) is the standard deviation of the Gaussian distribution, which controls the smoothing degree of the Gaussian filter. The larger the σ value, the more obvious the smoothing effect of the filter.

[0074] However, the Gaussian filter can only process relatively small noise points in the image, and has a poor processing effect on noise points with a large area and similar to bubbles in the image;

[0075] Third, to solve the bubble-type noise points in the image, use the cv2.findContours function in the cv2 library to find the contours of the noise points in the image;

[0076] Fourth, set the minimum contour area threshold to 100. Contours with an area less than 100 are all identified as bubble-type noise point contours;

[0077] Fifth, traverse all detected contours, calculate the area of each contour. For contours with an area less than 100, if their average pixel value is higher than 128, then modify the pixels inside the contour to 0 (modified to a black area), otherwise modify them to 255 (modified to a white area).

[0078] The main function of the above steps is to denoise the segmented result image of the bent steel bars and fill in the bubble-type noise points existing in individual images.

[0079] S4. Line Detection and Fitting Method

[0080] After the above image preprocessing, a clear grayscale image with a black background and white steel bars is generated. On this basis, we subsequently fit the white areas representing the steel bars in the image into several straight lines; in the field of image processing and computer vision, straight line detection is a basic but important step. Traditional methods include edge detection and Hough transform, but it is often difficult to obtain stable and accurate results when used alone. The embodiment of the present invention combines the straight line detection method of Canny edge detection and probabilistic Hough transform to effectively improve the accuracy and robustness of straight line detection.

[0081] S4.1. Detecting the steel bar contour in the image

[0082] After preprocessing the image, the Canny edge detection algorithm is applied to extract the edge information in the image. Canny edge detection uses the gradient information of the image to identify the area with obvious grayscale changes in the image, that is, the edge part. The core of Canny edge detection is to calculate the gradient amplitude and direction of the image, and then use the double threshold technology to identify the true edge.

[0083] 1. Calculate the gradient:

[0084]

[0085] Among them, G x and G y are the gradients of the image in the horizontal and vertical directions respectively. Through the convolution operation, the grayscale image I s With Sobel operator (G x and G y The corresponding convolution kernel) is multiplied to calculate the horizontal and vertical edge strength of the image. The Sobel operator has weighted coefficients in the calculation process. Larger coefficients (such as -2 and 2) are used to emphasize the differences between adjacent pixels, thereby highlighting the edges.

[0086] Gradient Magnitude:

[0087]

[0088] This formula calculates the gradient magnitude of each pixel in the image, which represents the strength of the edge at that pixel. By taking the square root of the sum of squares, the changes in the horizontal and vertical directions are taken into account. The larger the value, the more dramatic the brightness change at the pixel location, that is, the more likely it is to be at the edge.

[0089] Gradient direction:

[0090]

[0091] This formula calculates the gradient direction of each pixel in the image, indicating the direction of the edge. θ(x,y) is the change in the vertical direction G y and the horizontal change Gx The calculated angle. The result is usually between -π and π, representing the direction of the edge.

[0092] In the embodiments of the present invention, the low threshold is set to 50, the high threshold is set to 150, the Sobel operator of 3x3 is used to calculate the image gradient, and the significant edges are detected by the double threshold method. The pixels above the upper threshold are considered as edges, and the pixels below the lower threshold are ignored. This step generates an edge image, where the area with pixel value 255 represents the detected edges, and the pixel values of other areas are 0. Through Canny edge detection, the edge information in the image can be effectively extracted, providing a basis for subsequent line detection.

[0093] 2. Calculate the angle of the detected contour line

[0094] After edge detection, the probabilistic Hough transform (HoughLinesP) is used to detect the lines in the image. Then, through the endpoint coordinates of the lines, the inclination angle of each line is calculated. The inclination angle is obtained through the angle calculation formula between two point coordinates.

[0095] Assume that the two endpoints of the line are (x1, y1) and (x2, y2), then the inclination angle θ of the line can be calculated by the following formula:

[0096]

[0097] where θ represents the inclination angle of the line, and (x1, y1) and (x2, y2) represent the starting point and ending point coordinates of the line respectively.

[0098] The calculation of the inclination angle is an important basis for grouping and analyzing the lines. By obtaining the angle information of the lines, the geometric forms in the image can be effectively analyzed, and it provides support for line grouping and optimization in subsequent steps.

[0099] III. Group the lines according to the angle

[0100] According to the angle of each obtained line, the lines with similar angles are grouped into one group. By setting an angle threshold Δθ (such as 3°), the lines with a difference less than this threshold are grouped into the same group. For two lines, if their angle difference satisfies the following condition:

[0101] |θ i -θ j |≤Δθ

[0102] Then they are grouped into the same group. Here, θ i and θ jRepresent the angles of two lines respectively. Grouping lines with similar angles can effectively identify the main directions of the lines in the image and filter out those noisy or irrelevant lines with large angle differences. This process simplifies the geometric structure in the image and makes the analysis more focused on the main forms.

[0103] IV. Select the optimal lines

[0104] Select the top two groups that contain the most lines. Then, from these two groups, select the line with the angle closest to the median within the group as the representative line of the group.

[0105] For each group, calculate the median of the angles according to the following formula:

[0106] θ median = median(θ1, θ2, …, θ n )

[0107] Then select the line with the angle closest to the median:

[0108] optimal_line = min(|θ i - θ median |)

[0109] where θ median represents the median of the angles within the group, and θ i represents the angle of each line. By selecting the line with the angle closest to the median as the optimal line of the group. This step ensures that the selected line can accurately represent the geometric form in the image and simplifies the complex image structure.

[0110] S5. Angle calculation

[0111] S5.1. Extend the lines

[0112] For the selected optimal lines, extend them within the boundaries of the image. The extended lines ensure full display in the image and are used for further angle measurement.

[0113] For the endpoints (x1, y1) and (x2, y2) of the line, the coordinates of the extended endpoints are calculated as follows:

[0114] x’1 = max(0, 2x1 - x2), y’1 = max(0, 2y1 - y2)

[0115] x’2 = max(W, 2x2 - x1), y’2 = min(H, 2y2 - y1)

[0116] Among them, W and H are the width and height of the image respectively. The straight lines are extended to ensure complete display within the image boundaries, making the cross - angle measurement in subsequent steps more accurate. This step ensures the integrity of the straight lines and avoids analysis errors caused by overly short straight lines.

[0117] S5.2. Calculate the cross - angles between the straight lines and visualize them

[0118] For the extended optimal straight lines, calculate the cross - angles between them. The cross - angle represents the relative direction of two straight lines and is determined by the difference between the inclination angles of the two straight lines. Finally, display the calculated cross - angle information on the image.

[0119] Let the inclination angles of the two straight lines be θ1 and θ2 respectively, then their cross - angle is calculated as follows:

[0120]

[0121] The intersection point coordinates are calculated by the following formula:

[0122]

[0123] where represents the cross - angle of the two straight lines, and intersection_point represents the coordinates of the intersection point.

[0124] Through the above steps, the bending angle information of the steel bars is finally obtained from the straight lines fitted to the steel bar contours. Based on the bending angle of the steel bars, it is judged whether this bending operation is successful and whether a secondary bending operation is required.

[0125] It should be understood that the magnitudes of the sequence numbers of the steps in the above embodiments do not imply the order of execution. The order of execution of each process should be determined by its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of the present invention.

[0126] The embodiments of the present invention also provide an edge deployment method for the above - mentioned steel bar bending angle detection system based on machine vision, including:

[0127] Use NVIDIA's embedded development board for environment setup and algorithm deployment. The deployment process includes:

[0128] First, burn the operating system Ubuntu 18.04 into the storage medium of the development board - TF card, configure the required environment, and install various support packages, such as dependencies and libraries like CUDA, pip3, jtop, opencv, pytorch, torchvision, etc. The above storage medium can also be a magnetic disk drive, a traditional hard disk drive (HDD), a rotational hard disk, a magnetic recording storage, a magnetic storage medium, a solid state drive (SSD), a semiconductor storage device, a flash memory hard disk, a non-volatile storage medium, a portable flash drive, a USB flash memory, a portable storage device, etc.

[0129] After the system burning and environment setup are completed, the current TF card serves as the main storage medium in the embedded system, used to store software such as the operating system, application programs, etc. Deploy the optimized model in the operating system to directly perform real-time inference on the images captured by the camera, extract the edges of the bent steel bars, and accurately calculate their bending angles.

[0130] Use the publish-subscribe mode to write the inference results into the steel cage construction process discrimination algorithm in real time for identifying the steel cage process.

[0131] Install the SIM7600G-H 4G module on the development board, develop the NDIS dialing function, and use MQTT publish-subscribe to publish the detection results of the steel bar bending angles.

[0132] The embodiment of the present invention also provides a steel bar bending angle detection system based on machine vision, including:

[0133] An image segmentation module: used to segment the captured steel bar images based on a deep learning-based image segmentation model to obtain a binary image containing only the steel bars or an image with steel bar contour information;

[0134] Steel bar contour detection module: used to detect the binary image containing only steel bars or the image with steel bar contour information, obtain the edges of the steel bars and fit them into straight lines;

[0135] Angle calculation module: used to calculate the intersection angle of the fitted straight lines, so as to obtain the bending angle of the steel bars.

[0136] In some embodiments, the system further includes a model training module for training the deep learning-based image segmentation model. The training method includes:

[0137] Collect multiple bending images covering various steel bar bending situations during the production process of semi-finished steel bars, and divide the images into a training set, a validation set and a test set;

[0138] Import the training set and validation set images into the interactive segmentation annotation software, and load the annotation model for auxiliary annotation; use the polygon annotation function of the software to outline the contour of the bent steel bars; after the annotation is completed, the software generates an image file that meets the input format requirements of the first image segmentation model for recording image classification information;

[0139] Import the training set and validation set images into the graphic image annotation tool, select the polygon annotation function to annotate the contour of the bent steel bars; after the annotation is completed, the tool generates an annotation file that meets the input requirements of the second image segmentation model for saving steel bar annotation information;

[0140] Sort out the annotated data sets respectively, and place the image files and annotation files in the corresponding folder structures according to the input requirements of the image segmentation model to ensure that the data can be correctly read during model training;

[0141] Augment the number of images after data augmentation of the images in the annotated data sets;

[0142] Use the annotated files and the test set to train and test the first image segmentation model and the second image segmentation model respectively to obtain the final image segmentation model.

[0143] In some embodiments, the first image segmentation model is a model based on PaddleSeg. The method for training it includes:

[0144] Configure the model parameters in the yml file, write the paths of the training set and the validation set, and then start the training of the network model; the training uses the SGD stochastic gradient descent algorithm, and set the number of training epochs, the initial learning rate of the network and the size of the training batch_size; after the training is completed, start the script file to call the trained model to predict the test set and obtain the segmentation result of the bent steel bars.

[0145] The second image segmentation model is a model based on YOLOv8-seg, and the method for training it includes:

[0146] Configure the model parameters in the yaml file, fill in the paths of the training set, validation set and detection type, select the yolov8-seg.yaml network structure and then start training, set the number of training epochs, the initial learning rate of the network and the size of the training batch_size; after training, obtain the trained second image segmentation model based on YOLOv8-seg, start the script file to predict the test set, and obtain the segmentation result of the bent steel bars.

[0147] In some embodiments, it further includes an image preprocessing module for preprocessing the collected steel bar images to improve the image quality; the preprocessing includes: image grayscale conversion and / or Gaussian filtering and / or bubble-type noise removal; the grayscale conversion includes converting a color image into a grayscale image to reduce the amount of data; the Gaussian filtering is used to smooth the image and remove the high-frequency noise in the image; the bubble-type noise removal is used to find and remove the possible bubble-type noise in the image to make the contour of the steel bar clearer.

[0148] In some embodiments, the steel bar contour detection module includes:

[0149] Edge detection module: used to detect the edges of the steel bars in the image to obtain an edge image;

[0150] Line detection module: used to transform the edge image, detect the lines in the image, and calculate the inclination angle of each line;

[0151] Line grouping and screening module: used to group the detected lines according to the inclination angles of the lines, and select the optimal lines as the edges of the steel bars.

[0152] In some embodiments, the method for selecting the optimal lines includes:

[0153] According to the inclination angle of each obtained line, group the lines with an angle difference less than the set angle threshold;

[0154] Select the top two groups with the most lines; select the line with the angle closest to the median within the group as the optimal line of the group.

[0155] In some embodiments, the method for calculating the inclination angle of each line includes:

[0156] After edge detection, use the probabilistic Hough transform to detect the lines in the image, and then calculate the inclination angle of each line through the endpoint coordinates of the lines.

[0157] In some embodiments, the method for obtaining an edge image includes:

[0158] Applying the Canny edge detection algorithm to extract edge information in the image; the gradient calculation formula of the Canny edge detection algorithm includes:

[0159]

[0160] where G x and G y are the gradients of the image in the horizontal and vertical directions respectively;

[0161] Calculating the gradient magnitude and gradient direction of each pixel in the image;

[0162] Detecting significant edges by the double-threshold method. Pixels above the upper threshold are considered edges, and pixels below the lower threshold are ignored; finally, an edge image is generated, where the region with the pixel value being the first extreme value represents the detected edge, and the pixel values in other regions are the second extreme value.

[0163] An embodiment of the present invention further provides a steel bar bending angle detection system based on machine vision, including:

[0164] Segmenting the collected steel bar image by an image segmentation model based on deep learning to obtain a binary image containing only the steel bar or an image with steel bar contour information;

[0165] Detecting the binary image containing only the steel bar or the image with steel bar contour information to obtain the edge of the steel bar and fitting a straight line;

[0166] Calculating the intersection angle of the fitted straight line to obtain the bending angle of the steel bar.

[0167] An embodiment of the present invention further provides a non-transitory computer-readable storage medium. This computer-readable storage medium stores a computer program, and this computer program includes program instructions. When the program instructions are executed by a processor, each step of the method described in the present invention is implemented, which will not be elaborated here.

[0168] The computer-readable storage medium can be the internal storage unit of the data transmission device or computer device provided in any of the foregoing embodiments, such as the hard disk or memory of the computer device. This computer-readable storage medium can also be an external storage device of the computer device, such as a plug-in hard disk equipped on the computer device, a smart media card (SMC), a secure digital (SD) card, a flash card, etc.

[0169] Further, the computer-readable storage medium may also include both the internal storage unit of the computer device and external storage devices. The computer-readable storage medium is used to store the computer program and other programs and data required by the computer device. The computer-readable storage medium may also be used to temporarily store data to be output or already output.

[0170] Those skilled in the art should understand that the embodiments of the present invention may be provided as a method, a system, or a computer program product. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk memories, CD-ROMs, optical memories, etc.) containing computer-usable program code.

[0171] The present invention is described with reference to the flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of flows and / or blocks in the flowcharts and / or block diagrams, can be realized by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing devices generate means for realizing the functions specified in Figure 1 one or more of the flows Figure 1 or blocks.

[0172] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, such that the instructions stored in the computer-readable memory generate a manufactured article including instruction means that realizes the functions specified in Figure 1 one or more of the flows Figure 1 or blocks.

[0173] These computer program instructions can also be loaded onto a computer or other programmable data processing device, such that a series of operation steps are executed on the computer or other programmable device to generate a computer-implemented process, and thus the instructions executed on the computer or other programmable device provide steps for realizing the functions specified in Figure 1 one or more of the flows Figure 1 or blocks.

[0174] An embodiment of the present invention further provides a computer program product, including a computer program / instructions, which implement the steps of the method for detecting the bending angle of steel bars based on machine vision when executed by a processor.

[0175] The content not described in detail in this specification belongs to the prior art well-known to those skilled in the art.

Claims

1. A steel bar bending angle detection system based on machine vision, characterized in that, Including: An image segmentation module: used to segment the collected steel bar images based on a deep learning-based image segmentation model to obtain a binary image containing only steel bars or an image with steel bar contour information; A steel bar contour detection module: used to detect the binary image containing only steel bars or the image with steel bar contour information to obtain the edges of the steel bars and fit them into straight lines; An included angle calculation module: used to calculate the intersection angle of the fitted straight lines, thereby obtaining the bending angle of the steel bars.

2. The steel bar bending angle detection system based on machine vision according to claim 1, wherein, It also includes a model training module for training the deep learning-based image segmentation model. The training method includes: Collecting multiple bending images covering various steel bar bending situations during the production process of semi-finished steel bars, and dividing the images into a training set, a validation set, and a test set; Importing the training set and validation set images into an interactive segmentation annotation software, and loading an annotation model to assist in annotation; using the polygon annotation function of the software to outline the contours of the bent steel bars; after the annotation is completed, the software generates an image file that meets the input format requirements of the first image segmentation model for recording image classification information; Importing the training set and validation set images into a graphic image annotation tool, selecting the polygon annotation function to annotate the contours of the bent steel bars; after the annotation is completed, the tool generates an annotation file that meets the input requirements of the second image segmentation model for saving steel bar annotation information; According to the input requirements of the image segmentation model, placing the image file and the annotation file in the corresponding folder structure to ensure that the data can be correctly read during model training; Using the annotated files and the test set to train and test the first image segmentation model and the second image segmentation model to obtain the final image segmentation model.

3. The steel bar bending angle detection system based on machine vision according to claim 2, characterized in that The first image segmentation model is a model based on PaddleSeg. The method for training it includes: Configuring model parameters; writing the paths of the training set and the validation set and then starting the training of the network model; the training uses the SGD stochastic gradient descent algorithm, setting the number of training epochs, the initial learning rate of the network, and the size of the training batch_size; after the training is completed, starting a script file to call the trained model to predict the test set and obtain the segmentation result of the bent steel bars.

4. The machine vision-based steel bar bending angle detection system according to claim 2 or 3, characterized in that The second image segmentation model is a model based on YOLOv8-seg. The method for training it includes: Configuring model parameters; filling in the paths of the training set, the validation set, and the detection type, selecting the selected network structure and then starting the training, setting the number of training epochs, the initial learning rate of the network, and the size of the training batch_size; after the training is completed, obtaining the trained second image segmentation model based on YOLOv8-seg, starting a script file to predict the test set and obtain the segmentation result of the bent steel bars.

5. The steel bar bending angle detection system based on machine vision according to claim 1, characterized in that It further includes an image preprocessing module for preprocessing the collected steel bar images to improve the image quality; the preprocessing includes: image grayscale conversion and / or Gaussian filtering and / or bubble-type noise removal; the grayscale conversion includes converting a color image into a grayscale image to reduce the data volume; the Gaussian filtering is used to smooth the image and remove high-frequency noise in the image; the bubble-type noise removal is used to find and remove possible bubble-type noise in the image to make the contour of the steel bar clearer.

6. The steel bar bending angle detection system based on machine vision according to claim 1, wherein The steel bar contour detection module includes: An edge detection module: for detecting the edge of the steel bar in the image to obtain an edge image; A straight line detection module: for transforming the edge image, detecting straight lines in the image, and calculating the inclination angle of each straight line; A straight line grouping and screening module: for grouping the detected straight lines according to the inclination angles of the straight lines and selecting the optimal straight lines as the edges of the steel bars.

7. The steel bar bending angle detection system based on machine vision according to claim 6, characterized in that The method for selecting the optimal straight lines includes: According to the inclination angles of each obtained straight line, dividing the straight lines with an angle difference less than a set angle threshold into a group; Selecting the top two groups with the most straight lines; selecting the straight line with an angle closest to the median within the group as the optimal straight line of the group.

8. A method for detecting the bending angle of steel bars by machine vision of the system according to claim 1, characterized in that, It includes: Segmenting the collected steel bar images by an image segmentation model based on deep learning to obtain a binary image containing only steel bars or an image with steel bar contour information; Detecting the binary image containing only steel bars or the image with steel bar contour information to obtain the edge of the steel bar and fitting a straight line; Calculating the intersection angle of the fitted straight line to obtain the bending angle of the steel bar.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method for detecting the bending angle of a steel bar based on machine vision as claimed in claim 8.

10. A computer program product, comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by a processor, it implements the steps of the method for detecting the bending angle of a steel bar based on machine vision as claimed in claim 8.

Images