A steel bar binding control system and a binding position recognition method

Abstract

The present invention provides a steel bar binding control system, comprising: an image acquisition device for acquiring a depth image and an RGB image of an area where steel bars to be bound are located; a central processing unit, comprising an image preprocessing module, for replacing background information and underlying steel bar information in the RGB image with monochrome according to spatial depth data of the depth image, and obtaining a preprocessed RGB image; a recognition module, for inputting the preprocessed RGB image into a pre-trained recognition model of a position to be bound, outputting a bounding box of an intersection to be bounded, processing each bounding box based on a method of local feature points, obtaining edge feature points of each steel bar in the bounding box, fitting the straight edge of the steel bar according to the edge feature points, and taking the intersection of the straight edge of the steel bar as the position to be bounded; and an actuator driver, for controlling an actuator to bind the steel bars according to the position to be bounded. Accurately identify the position to be bounded, and control the actuator to complete high-quality binding.

Description

Technical Field

[0001] The present invention relates to the technical field of steel bar binding control, and in particular to a steel bar binding control system and a binding positioning identification method. Background Art

[0002] In the construction industry, steel bars are equivalent to human skeletons, playing the role of supporting and connecting the building body. A large number of steel bars need to be fixed in sequence according to the engineering drawings before pouring concrete, so that the steel skeleton can be reliably fixed. With the development of building industrialization and prefabricated buildings, the traditional method of processing steel bars for PC components is mainly to manually mark and tie them on the ground according to the drawings, which is inefficient. Different types of composite slabs need to be repeatedly re-marked, with a high error rate, and high professional requirements for operators. In addition, the next process requires the steel bars to be inspected and reviewed, which is very easy to produce waste and defective products.

[0003] Therefore, there is an urgent need for a steel bar binding control system to realize mechanical control, identification and positioning, and then complete the steel bar binding action. Summary of the invention

[0004] 1. Technical issues to be resolved

[0005] In view of the problems existing in the above technologies, the present invention solves them at least to a certain extent. To this end, the first object of the present invention is to propose a steel bar binding control system, which is conducive to the accurate identification of the position to be bound, and then controls the actuator to complete the high-quality binding operation.

[0006] The second objective of the present invention is to provide a binding positioning and identification method.

[0007] (II) Technical solution

[0008] In order to achieve the above object, the present invention provides a steel bar binding control system, comprising:

[0009] An image acquisition device is used to obtain a depth image and an RGB image of the area where the steel bars to be tied are located;

[0010] A central processing unit, including an image preprocessing module and a to-be-bound-position recognition module;

[0011] An image preprocessing module is used to remove background information and underlying steel bar information in the RGB image according to the spatial depth data of the depth image, and fill the positions where the background information and underlying steel bar information are located in the RGB image with monochrome to obtain a preprocessed RGB image;

[0012] The position to be tied recognition module is used to input the preprocessed RGB image into a pre-trained position to be tied recognition model, output the bounding box of the intersection points to be tied, process each bounding box based on the local feature point method, obtain the edge feature points of each steel bar in the bounding box, fit the straight edge of the steel bar according to the edge feature points, and take the intersection points of the straight edges of the steel bars in each bounding box as the positions to be tied;

[0013] The actuator driver is used to control the actuator to tie the steel bars according to the positions to be tied.

[0014] Optionally, the image acquisition device is an RGB-D camera.

[0015] Optionally, the position to be tied recognition module is used to input the preprocessed RGB image into a pre-trained Ghost-YOLOX network model, and output the bounding box of the intersection points to be tied; the construction of the Ghost-YOLOX network model includes: replacing the convolution in the CBL module of the YOLOX network with a Ghost module.

[0016] Optionally, the construction of the Ghost-YOLOX network model further includes: introducing an SE module based on the attention channel mechanism into the backbone feature extraction network of the YOLOX network, inputting the feature layers extracted by the backbone feature extraction network into the SE module, and performing feature fusion according to the output of the SE module.

[0017] Optionally, the position to be tied recognition module processes each bounding box based on the local feature point method to obtain the edge feature points of each steel bar in the bounding box, including: the position to be tied recognition module sets multiple horizontal lines and multiple vertical lines in the bounding box area, the multiple horizontal lines are used as the first group of Line ROI, and the multiple vertical lines are used as the second group of Line ROI; extract the corresponding image pixel values on each line in the Line ROI, and calculate the position with the largest pixel value change on each line as the edge feature point of the steel bar.

[0018] Optionally, the position to be tied recognition module is further used to use the RANSAC algorithm to remove the outliers in the edge feature points of the steel bars through iterative optimization; the outliers are the edge feature points at the noise positions; correspondingly, the position to be tied recognition module fits the straight edge of the steel bar according to the remaining edge feature points, and takes the intersection points of the straight edges of the steel bars in each bounding box as the positions to be tied.

[0019] In a second aspect, the present invention provides a tying position recognition method, including the following steps:

[0020] S1. Obtain the depth image and RGB image of the area where the steel bars to be tied are located;

[0021] S2. According to the spatial depth data of the depth image, eliminate the background information and the underlying steel bar information in the RGB image, and fill the positions where the background information and the underlying steel bar information are located in the RGB image with a single color to obtain a preprocessed RGB image;

[0022] S3. Input the preprocessed RGB image into a pre-trained model for identifying the positions to be tied. Output the bounding boxes of the intersections to be tied. Process each bounding box based on the method of local feature points to obtain the edge feature points of each steel bar in the bounding box. Fit the straight edges of the steel bars according to the edge feature points, and take the intersection points of the straight edges of the steel bars in each bounding box as the positions to be tied.

[0023] As an improvement of the method of the present invention, the model for identifying the positions to be tied is the Ghost-YOLOX network model; the construction of the Ghost-YOLOX network model includes: replacing the convolution in the CBL module of the YOLOX network with a Ghost module.

[0024] As an improvement of the method of the present invention, the construction of the Ghost-YOLOX network model further includes: introducing an SE module based on the attention channel mechanism into the backbone feature extraction network Backbone of the YOLOX network. The feature layers extracted by the backbone feature extraction network are input into the SE module, and feature fusion is performed according to the output of the SE module.

[0025] As an improvement of the method of the present invention, processing each bounding box based on the method of local feature points to obtain the edge feature points of each steel bar in the bounding box includes: setting multiple horizontal lines and multiple vertical lines in the bounding box area. The multiple horizontal lines are used as the first group of Line ROI, and the multiple vertical lines are used as the second group of Line ROI; extracting the corresponding image pixel values on each line in the Line ROI, and calculating the position with the largest change in pixel values on each line as the edge feature points of the steel bar.

[0026] (III) Beneficial effects

[0027] The beneficial effects of the present invention are:

[0028] The steel bar tying control system provided by the present invention, by acquiring the depth image and the RGB image of the area where the steel bars to be tied are located, and preprocessing the RGB image according to the depth image, increases the contrast between the top-layer steel bars to be recognized and the background in the RGB image, which is beneficial to the accurate recognition of the positions to be tied; and first extracts the rough positions of each untied intersection through the model for identifying the positions to be tied, and then identifies the untied intersections based on the method of local feature points in the rough positions of each untied intersection, so as to obtain the accurate coordinates of the positions to be tied. Description of the drawings

[0029] The present invention is described with the aid of the following drawings:

[0030] Figure 1 It is a schematic structural diagram of a steel bar binding control system according to a specific embodiment of the present invention;

[0031] Figure 2 It is a schematic structural diagram of the Ghost module according to a specific embodiment of the present invention;

[0032] Figure 3 It is a schematic diagram of two residual structures in the GBL module according to a specific embodiment of the present invention; wherein, Ghostmodule represents the Ghost module, BN represents batch normalization, Silu represents the Silu activation function, Add represents summation, and DWconv represents depthwise separable convolution;

[0033] Figure 4 It is a schematic diagram of the Ghost-YOLOX network structure according to a specific embodiment of the present invention; wherein, Input represents the input, Focus represents the picture slicing operation, SPP represents spatial pyramid pooling, Upsample represents picture upsampling, and Concat represents feature fusion;

[0034] Figure 5 It is a schematic structural diagram of the SE module according to a specific embodiment of the present invention;

[0035] Figure 6 It is a schematic diagram of the recognition result of the steel bar binding position in a bounding box area according to a specific embodiment of the present invention.

[0036]

Description of the reference numerals

[0037] 1: Image acquisition device;

[0038] 2: Central processing unit; 21: Image preprocessing module; 22: Unbound position recognition module; 23: Communication module;

[0039] 3: Actuator driver. Specific embodiments

[0040] In order to better explain the present invention for easy understanding, the present invention will be described in detail below with reference to the drawings through specific embodiments.

[0041] As Figure 1 shown, the present invention provides a steel bar binding control system. The steel bar binding control system includes an image acquisition device 1, a central processing unit 2, and an actuator driver 3.

[0042] Image acquisition device 1 is used to obtain the depth image and RGB image of the area where the steel bars are to be tied. In this way, the RGB image is obtained because the YOLOX network can only process RGB images but not depth images; the depth image is obtained in order to pre-process the RGB image, because there is no spatial depth data in the RGB image.

[0043] Preferably, the image acquisition device 1 is an RGB-D camera, which can simultaneously acquire a depth image and an RGB image of the area where the steel bars to be tied are located, thereby facilitating subsequent image processing.

[0044] The central processing unit 2 includes an image preprocessing module 21 and a to-be-bound position identification module 22 .

[0045] The image preprocessing module 21 is used to remove the background information and the bottom steel bar information in the RGB image according to the spatial depth data of the depth image, and fill the positions where the background information and the bottom steel bar information are located in the RGB image with monochrome to obtain a preprocessed RGB image. Among them, the background information refers to other information in the RGB image except the steel bar information, and the bottom steel bar information refers to other steel bar information in the RGB image except the top steel bar to be identified. In this way, the contrast between the top steel bar to be identified in the RGB image and the background is increased, which is conducive to the accurate identification of the position to be tied.

[0046] Specifically, as an example, the image preprocessing module 21 is used to remove the background information and underlying steel bar information in the RGB image according to the spatial depth data of the depth image, and fill the positions where the background information and underlying steel bar information are located in the RGB image with black to obtain a preprocessed RGB image.

[0047] The module 22 for identifying the position to be tied is used to input the pre-processed RGB image into the pre-trained model for identifying the position to be tied, output the bounding box of the intersection to be tied, process each bounding box based on the method of local feature points, obtain the edge feature points of each steel bar in the bounding box, fit the straight edge of the steel bar according to the edge feature points, and take the intersection of the straight edge of the steel bar in each bounding box as the position to be tied. In this way, the rough position of each untied intersection is first extracted through the model for identifying the position to be tied, and then the untied intersection is identified in the rough position of each untied intersection based on the method of local feature points, so that the accurate coordinates of the position to be tied can be obtained.

[0048] Preferably, the module 22 for identifying the position to be bound is used to input the preprocessed RGB image into the pre-trained Ghost-YOLOX network model, and output the bounding box of the intersection to be bound. The construction of the Ghost-YOLOX network model includes: replacing the convolution in the CBL module of the YOLOX network with the Ghost module.

[0049] Among them, the YOLOX network can be divided into three parts: the backbone feature extraction network Backbone, the context feature fusion Neck, and the result prediction Prediction. In the backbone feature extraction network, first, the Focus operation (image slicing operation) is used to obtain a two-fold downsampled feature map without information loss, and at the same time, the number of channels is expanded to four times the original. Then, the CBL module and the CSP module are used for feature extraction. In the context feature fusion part, according to the three feature layers extracted by the backbone feature extraction network, a feature pyramid is constructed and effective feature fusion is carried out. In the result prediction part, first, a 1×1 convolution is used to decouple the classification and regression predictions, and then the category to which the target belongs and the target position are predicted respectively. The Ghost-YOLOX network obtained by improving the YOLOX network in the present invention replaces the convolution in the CBL module of the YOLOX network with a Ghost module, so that the CBL module becomes a GBL module, as Figure 4 shown. In this way, on the one hand, the YOLOX network is selected as the benchmark model of the present invention, and the detection accuracy and detection speed of the binding position are relatively high. On the other hand, by replacing the convolution in the CBL module of the YOLOX network with a Ghost module, the execution efficiency of the model and the accuracy of feature extraction are further improved.

[0050] Among them, the structure of the Ghost module is as Figure 2 shown. For the input , where is a real number matrix, is the number of channels of the input data, is the height of the input data, is the width of the input data. First, a convolution operation is used to map the input to the eigenfeature map , where is the number of channels of the feature map, is the height of the feature layer, is the width of the feature layer. Then, the depthwise separable convolution is used for the eigenfeature map to obtain the Ghost feature map, and the eigenfeature map ( Figure 1 the result of the convolution operation on the input in is the eigenfeature map) and the Ghost feature map (the result of using the depthwise separable convolution on the eigenfeature map is the Ghost feature map) are concatenated in the channel dimension to obtain the final output . Using fewer conventional convolutions to generate the eigenfeature map and using the depthwise separable convolution to generate the Ghost feature map greatly reduces the number of parameters of the model and improves the inference speed of the model. The two residual structures constructed by combining the Ghost module with the batch normalization operation BN, the Silu activation function, and the depthwise separable convolution DWConv are as Figure 3As shown, the GBL module is formed by serially stacking two residual structures as shown in Figure 3 . The set residual structure is used to extract features from the picture, and the features extracted are more accurate than those extracted by the ordinary convolutional structure.

[0051] Preferably, the construction of the Ghost-YOLOX network model further includes: introducing an SE module based on the attention channel mechanism into the backbone feature extraction network of the YOLOX network. The feature layer extracted by the backbone feature extraction network is input into the SE module, and feature fusion is performed according to the output of the SE module, as shown in Figure 4 . In this way, the SE module enables different channels of the feature layer extracted by the backbone feature extraction network to use the information of the global receptive field.

[0052] The structure of the SE module is as shown in Figure 5 . First, the input is mapped to the feature map through the transformation, and then the feature map is compressed in its spatial dimension ( ) through the operation to generate a channel descriptor, which can generate a global distribution feature representation of the per-channel feature response. After that, the global feature representation is output as a set of adjusted weights for each channel through the operation; finally, according to this weight set, the feature map is adjusted to the output through the

[0053] operation. In this way, the detection accuracy of the Ghost-YOLOX network model is further improved. Preferably, the binding position recognition module 22 processes each bounding box based on the local feature point method to obtain the edge feature points of each steel bar in the bounding box, including: the binding position recognition module 22 sets multiple horizontal lines and multiple vertical lines in the bounding box area. The multiple horizontal lines are used as the first group of Line ROI, and the multiple vertical lines are used as the second group of Line ROI; the corresponding image pixel values on each line in the Line ROI are extracted, and the position with the largest change in pixel values on each line is calculated as the edge feature point of the steel bar.

[0054] Since each region of interest (i.e., the bounding box) extracted using the Ghost-YOLOX network contains only one horizontal steel bar, one vertical steel bar, and a small amount of interference noise. Therefore, a set of horizontal Line ROI and a set of vertical Line ROI can be respectively set in each rectangular region of interest (as shown in Figure 6For the shown region of interest, multiple lines are set. All horizontal lines can be regarded as a group of horizontal Line ROI, and all vertical lines can be regarded as a group of vertical Line ROI. Then, the edge feature points of longitudinal steel bars and transverse steel bars can be extracted respectively. Furthermore, based on the edge feature points, the straight edges of the steel bars can be accurately fitted. By calculating the intersection points of the edges of transverse steel bars and longitudinal steel bars, the accurate coordinates of the positions to be tied can be obtained.

[0055] As an example, the bounding box is described by the horizontal and vertical coordinates of its top-left and bottom-right vertices relative to the entire image.

[0056] Further preferably, the position-to-be-tied recognition module 22 extracts the corresponding image pixel values on each line in the Line ROI, and uses the first-order difference algorithm to calculate the position with the largest pixel value change on each line as the edge feature points of the steel bars. Specifically, in this embodiment, as Figure 6 shown, a group of Line ROI contains 15 lines, that is, a group of Line ROI can extract 15 independent edge feature points.

[0057] Since the Line ROI may extract noise positions, the edge feature points at the noise positions can be regarded as outliers. Therefore, further preferably, the position-to-be-tied recognition module 22 is also used to use the RANSAC algorithm to remove the outliers in the edge feature points of the steel bars through an iterative optimization method; correspondingly, the position-to-be-tied recognition module 22 fits the straight edges of the steel bars according to the remaining edge feature points, and takes the intersection points of the straight edges of the steel bars in each bounding box as the positions to be tied. Further, the number of removed outliers is 30% - 50% of the total number of edge feature points. Specifically, in this embodiment, the number of removed outliers is 40% of the total number of edge feature points.

[0058] The actuator driver 3 is used to control the actuator to tie the steel bars according to the positions to be tied.

[0059] Specifically, in this embodiment, the actuator driver 3 is a tying servo motor driver.

[0060] Preferably, the central processor 2 further includes a communication module 23, and the communication module 23 is used to convert the position-to-be-tied information into a protocol format supported by the servo motor driver. Specifically, in this embodiment, the communication module 23 is used to convert the position-to-be-tied information into the modbus protocol format supported by the servo motor driver, and the central processor 2 sends it to the servo motor driver through the serial port for rs485 communication, so as to control the actuator to tie the steel bars.

[0061] The present invention also provides a tying positioning recognition method, including the following steps:

[0062] Step S1: Obtain the depth image and RGB image of the area where the steel bars to be tied are located, and perform alignment processing on the depth image and the RGB image.

[0063] Step S2: According to the spatial depth data of the depth image, eliminate the background information and the information of the bottom steel bars in the RGB image, and fill the positions where the background information and the bottom steel bar information are located in the RGB image with a single color to obtain a preprocessed RGB image.

[0064] Step S3: Input the preprocessed RGB image into a pre-trained recognition model for the tying position, output the bounding boxes of the tying intersections, process each bounding box based on the method of local feature points to obtain the edge feature points of each steel bar in the bounding box, fit the straight edges of the steel bars according to the edge feature points, and take the intersection points of the straight edges of the steel bars in each bounding box as the tying positions.

[0065] Before step S1, it further includes: using an industrial camera to collect pictures of the steel bars on site that are tied and not tied, and using the LableImg picture annotation tool to annotate the tied or to-be-tied positions in the pictures. The annotation labels are two categories: tied and not tied, which are used as training data for training the recognition model for the tying position.

[0066] It should be noted that for the image processing process in the tying position recognition method provided by the present invention, reference can be made to the detailed description of the steel bar tying control system provided in the above embodiment, and details will not be elaborated here.

[0067] It should be understood that the description of the specific embodiments of the present invention above is only for explaining the technical route and characteristics of the present invention. The purpose is to enable those skilled in the art to understand the content of the present invention and implement it accordingly. However, the present invention is not limited to the above specific embodiments. Any changes or modifications made within the scope of the claims of the present invention should be covered by the protection scope of the present invention.

Claims

1. A steel bar tying control system, It is characterized in that include: An image acquisition device is used to obtain a depth image and an RGB image of the area where the steel bars to be tied are located; A central processing unit, including an image preprocessing module and a to-be-bound-position recognition module; An image preprocessing module is used to remove background information and underlying steel bar information in the RGB image according to the spatial depth data of the depth image, and fill the positions where the background information and underlying steel bar information are located in the RGB image with monochrome to obtain a preprocessed RGB image; The module for identifying the position to be tied is used to input the pre-processed RGB image into the pre-trained model for identifying the position to be tied, output the bounding box of the intersection to be tied, process each bounding box based on the method of local feature points, obtain the edge feature points of each steel bar in the bounding box, fit the straight edge of the steel bar according to the edge feature points, and take the intersection of the straight edge of the steel bar in each bounding box as the position to be tied; An actuator driver, used for controlling the actuator to tie the steel bars according to the positions to be tied; The module for identifying the positions to be tied is used to input the pre-processed RGB image into the pre-trained Ghost-YOLOX network model and output the bounding box of the intersections to be tied; The construction of the Ghost-YOLOX network model includes: replacing the convolution in the CBL module of the YOLOX network with the Ghost module, introducing the SE module based on the attention channel mechanism into the backbone feature extraction network of the YOLOX network, inputting the feature layer extracted by the backbone feature extraction network into the SE module, and performing feature fusion according to the output of the SE module.

2. The steel bar tying control system according to claim 1, It is characterized in that The image acquisition device is an RGB-D camera.

3. The steel bar tying control system according to claim 1, It is characterized in that The module for identifying the position to be tied processes each bounding box based on the method of local feature points to obtain the edge feature points of each steel bar in the bounding box, including: The to-be-bounded position recognition module sets a plurality of horizontal lines and a plurality of vertical lines in the bounding box area, wherein the plurality of horizontal lines serve as a first group of Line ROIs, and the plurality of vertical lines serve as a second group of Line ROIs; The image pixel values ​​corresponding to each straight line in the Line ROI are extracted, and the position with the largest pixel value change on each straight line is calculated as the edge feature point of the steel bar.

4. The steel bar tying control system according to claim 3, It is characterized in that The module for identifying the position to be tied is also used to remove outliers from the edge feature points of the steel bars by using the RANSAC algorithm through an iterative optimization method; the outliers are edge feature points at noise positions; Correspondingly, the module for identifying the position to be tied fits the straight edges of the steel bars according to the remaining edge feature points, and takes the intersection of the straight edges of the steel bars in each bounding box as the position to be tied.

5. A binding positioning and identification method, It is characterized in that The following steps are involved: S1, obtaining a depth image and an RGB image of the area where the steel bars to be tied are located; S2. According to the spatial depth data of the depth image, eliminate the background information and the underlying steel bar information in the RGB image, and fill the positions where the background information and the underlying steel bar information are located in the RGB image with a single color to obtain a preprocessed RGB image; S3. Input the preprocessed RGB image into a pre-trained model for identifying the positions to be tied. Output the bounding boxes of the intersections to be tied. Process each bounding box based on the method of local feature points to obtain the edge feature points of each steel bar in the bounding box. Fit the straight edges of the steel bars according to the edge feature points, and take the intersection points of the straight edges of the steel bars in each bounding box as the positions to be tied; The model for identifying the positions to be tied is the Ghost-YOLOX network model; The construction of the Ghost-YOLOX network model includes: replacing the convolution in the CBL module of the YOLOX network with a Ghost module; introducing an SE module based on the attention channel mechanism into the backbone feature extraction network Backbone of the YOLOX network. The feature layers extracted by the backbone feature extraction network are input into the SE module, and feature fusion is performed according to the output of the SE module.

6. The tying position identification method according to claim 5, wherein, processing each bounding box based on the method of local feature points to obtain the edge feature points of each steel bar in the bounding box includes: setting a plurality of horizontal lines and a plurality of vertical lines in the bounding box area. The plurality of horizontal lines are used as the first group of Line ROI, and the plurality of vertical lines are used as the second group of Line ROI; extracting the corresponding image pixel values on each straight line in the Line ROI, and calculating the position with the largest change in pixel values on each straight line as the edge feature points of the steel bar.

Images