Deep learning training method and data set optimization method in strapping robot vision system

Abstract

The present invention discloses a deep learning training method and a data set optimization method in the strapping robot vision system described in the present invention, including detailed sampling of steel bar strapping operations on industrial sites; completion of data set creation and annotation; image processing and data set construction; targeted deep learning model training, scene adaptability optimization and model verification. The present invention significantly improves the recognition accuracy and versatility of the model through more accurate and balanced data set creation and optimization. And through dynamic data set updates and confidence-based feedback adjustments, it can continuously adapt to different working environments, effectively respond to changes in light and external conditions, and ensure the stability and reliability of the system. The image enhancement and data balancing strategies effectively increase the number of samples of rare categories, thereby improving the performance of the entire visual system.

Description

Technical Field

[0001] The present invention belongs to the technical field of steel bar bundling, and in particular relates to a deep learning training method and a data set optimization method in a bundling robot vision system. Background Art

[0002] With the rise of intelligent construction, automated workshops have become the mainstream in the construction industry. Among them, visual recognition plays an indispensable role in automated workshops and has been widely used in various industrial workshop assembly lines. Although this technology is widely used, it faces some key challenges. The main problem is that the current visual inspection method relies on data-driven deep learning, but lacks a common public dataset for specific projects. It is difficult to obtain a very large amount of data in a short time for self-built datasets, and too little data will lead to low accuracy and versatility of the trained model. Especially when the light and external environment in the industrial workshop affect the recognition area, the recognition accuracy will be further reduced. To address this problem, engineers need to re-modify the training parameters for different scenarios and perform multiple training optimizations. This not only increases the workload of developers and leads to low work efficiency, but also the retrained model may still have the same problems, which is not conducive to the generalization and standardization of the visual system. Therefore, the existing technology has significant deficiencies in dataset management and model adaptability. Summary of the invention

[0003] Purpose of the invention: In order to overcome the above shortcomings, the purpose of the present invention is to provide a deep learning training method and a data set optimization method in a strapping robot vision system, which significantly improves the recognition accuracy and versatility of the model through more accurate and balanced data set creation and optimization. And through dynamic data set updates and confidence-based feedback adjustments, it can continuously adapt to different working environments, effectively respond to changes in light and external conditions, and ensure the stability and reliability of the system.

[0004] Technical solution: In order to achieve the above-mentioned purpose, the present invention provides a deep learning training method and a data set optimization method in a strapping robot vision system, comprising:

[0005] 1): Carefully sample the steel bar bundling operations at industrial sites and complete the creation and annotation of data sets. That is, the staff uses industrial array cameras to collect all-round image data of the steel bar bundling operations at different time periods (including morning, noon, and evening) and under different lighting conditions (sunny, cloudy, and with lighting equipment). The collected image data is then classified and labeled and the category distribution is analyzed. The image annotation tool is used to accurately locate and mark the steel bar bundling points in each image, and their category information is recorded. The number of bundling points of different categories is counted through statistical analysis software, and a category distribution map is generated to identify the categories of bundling points with insufficient sample size in the data set.

[0006] 2): Image processing and data set construction, that is, according to the category distribution of the bundling points in the current data set, mosaic image enhancement optimization is used for the bundling point categories with insufficient sample size to obtain a preliminary dataset of steel bar bundling points, and the dataset is enhanced and trained;

[0007] 3): Targeted deep learning model training: Use the processed data set to train the deep learning model and verify the accuracy of the model under different working conditions;

[0008] 4): Scenario adaptability optimization and model verification. After the model training is completed, the staff will conduct rigorous testing on the model in a real rebar bundling working environment, including environmental adaptability testing, performance stability testing, and final accuracy evaluation.

[0009] The deep learning training method and data set optimization method in the strapping robot vision system described in the present invention, the specific process of data set creation and labeling in 1) is as follows: Create a data set monitor, classify and count the three types of labels in different production line scenarios, collect image data of steel bar strapping points through industrial cameras, and perform 4 ~ The 8cm rectangular calibration frame is annotated. Through the image data features of the binding points, it can be distinguished into bound binding points, unbound binding points and the intersections of the steel bars and the side molds. The intersections of the steel bars that have been bound with steel wires are distinguished as bound binding points, and the intersections of the steel bars that have not been bound with steel wires are distinguished as unbound binding points. The intersections of the steel bars and the side molds are identified as the intersections of the steel bars and the side molds, thereby generating the original data set and its feature labels. According to the label category distribution in the monitor,

[0010] The deep learning training method and data set optimization method in the strapping robot vision system described in the present invention, the specific process of adding images in the 2) image processing and data set construction is as follows:

[0011] 201): To analyze the imbalance of data set categories, the technicians first used data analysis tools to perform detailed statistics on the image classification in the data set and identify the categories of strapping points with insufficient sample size in the collected data set; these categories are usually due to low frequency of occurrence on site or difficult to collect under certain conditions;

[0012] 202): For the categories with insufficient sample size identified in step 201), mosaic is used to enhance the images. Four images are read each time, and then the four images are flipped, translated, scaled, cropped, and the color gamut is changed. The four images are then arranged in different directions, which can greatly enhance the diversity and complexity of the training data set, thereby improving the robustness and generalization ability of the deep learning model; the four images are then randomly cropped according to their respective positions, and finally the images and frames are combined to obtain a more balanced and diverse data set; the purpose of the enhancement operation is to artificially create more diverse data to balance the representativeness of each category in the data set and enhance the generalization ability of the model;

[0013] 203): Enhanced dataset training: The balanced and diversified dataset obtained in the previous step is trained with a deep learning model. During the training process, the performance of the model in each category will be closely monitored. The category labels with a smaller number will be segmented out in the image, processed with 40% transparency, and randomly added to other random datasets for mixed training. This method is used to balance the number of category labels in the dataset. If necessary, the parameters of the model need to be adjusted.

[0014] In the deep learning training method and data set optimization method in the strapping robot vision system described in the present invention, the mosaic operation execution standard 202) is determined by the following formula:

[0015]

[0016] in: is the preset image enhancement probability, is the number of samples that need to be enhanced, is the total number of samples in the dataset, is the class imbalance, To ultimately enhance the probability.

[0017] The deep learning training method and data set optimization method in the strapping robot vision system described in the present invention, the specific process of enhancing the data set training in 3) is as follows: 2031): The enhanced image is normalized, and the pixel value range is modified to between 0 and 1. The normalization formula is as follows:

[0018]

[0019] in: is the original pixel value, The minimum pixel value of the image is usually 0; The maximum pixel value of the image, usually 255; is the normalized value;

[0020] Through the above processing, all pixel values ​​are scaled to the range of 0 to 1, which helps to improve the numerical stability and convergence speed of model training;

[0021] 2032): Resize the image to 640×640, and input the image into the YOLO deep learning model with a channel number of 64×640×640×3, where 64 is the batch size, 640×640 is the height × width of the image, and 3 is the number of color RGB (red, green, blue) channels;

[0022] 2033): Set the training parameters and divide the data set into training set, validation set and test set in the proportion of 70%, 15% and 15%. The loss function in training adopts the cross entropy loss function for iterative training;

[0023] 2034): Use accuracy to visualize whether the training results are good enough. The accuracy formula is as follows:

[0024]

[0025] Among them, A is the recognition accuracy;

[0026] TP (True Positives) is the number of true classes: the model correctly predicts the positive class as positive;

[0027] TN (True Negatives) is the number of true negatives: the model correctly predicts the negative class as negative;

[0028] FP (False Positives) is the number of false positives: the model incorrectly predicts the negative class as positive;

[0029] FN (False Negatives) is the number of false negatives: the model incorrectly predicts the positive class as negative;

[0030] 2035): Finally, the learning rate is set to 0.001. In each training batch, the model calculates the loss through forward propagation, and then updates the weights through backpropagation. The loss and accuracy during training and validation are recorded and monitored.

[0031] The deep learning training method and data set optimization method in the strapping robot vision system described in the present invention, the specific process of scene adaptability optimization and model verification in 4) is as follows: 401): real-time monitoring of model performance, through a real-time performance monitoring system, for continuously tracking and evaluating the performance of the model in a new environment, including real-time monitoring of key indicators of model recognition accuracy, response time and stability, and analyzing the trend and potential problems of model performance through regularly generated performance reports;

[0032] 402): Analyze the differences between the new environment data and the original training set. When the model is deployed in a new working environment, the staff first conducts a series of target recognition tests and collects the recognition data of the model in the new environment. By comparing it with the original training set, the difference in category distribution between the two is analyzed to determine whether the data set needs to be updated. This analysis will take into account the new strapping point styles, background changes, and lighting conditions that appear in the new environment;

[0033] 403): Dynamically update the data set. According to the analysis results in step 402), if the data set needs to be updated, the staff will collect image data in the new environment and include it in the training set. At the same time, in order to maintain the timeliness and accuracy of the data set, those samples that are no longer representative in the new environment will be deleted. This process includes manually reviewing the quality of the newly added samples to ensure that the quality of the newly added data meets the training requirements.

[0034] 404): Optimize the model and repeat training and verification. Using the updated data set, the staff will restart the model training and verification process. During this process, the model parameters, learning rate, regularization term and training cycle will be carefully adjusted to adapt to the update of the data set. At the same time, through multiple rounds of training and cross-validation, it is ensured that the model maintains or exceeds the original recognition accuracy in the new environment.

[0035] The deep learning training method and data set optimization method in the strapping robot vision system described in the present invention, the specific process of dynamically updating the data set in 403) is as follows:

[0036] 4031): According to the monitor created previously, the distribution of category labels is analyzed in real time to obtain the number of category labels in the data set;

[0037] 4032): When the visual system enters a new industrial environment, the recognition results are analyzed and the difference between the recognized label category distribution and the label category distribution of the training set is calculated as follows:

[0038]

[0039] in, is the difference in the distribution of class labels, is the total number of label categories, is the number of samples of a certain category in the new environment, is the total number of samples in the new environment, is the number of samples of this category in the original data set, is the total number of samples in the original data set, set the threshold α, if QUOTE When the preset threshold α is exceeded, the system will trigger an alarm and upload new environmental data, store and package the images taken in the new scene and upload them to the local training server, delete the labels in the old dataset that are too different from the new dataset categories, and add the new dataset for retraining.

[0040] The deep learning training method and data set optimization method in the strapping robot vision system described in the present invention, the label recognition method is a confidence-based label recognition method, specifically: when the visual model recognizes an object, it will be accompanied by a confidence standard to visualize the correct probability of recognition. When the recognition confidence of the visual model for a certain category label is lower than the threshold, the system will automatically trigger the model's retraining process. During this process, those categories with low confidence will be preferentially selected for key training to improve the overall recognition accuracy and generalization ability of the model.

[0041] The specific calculation process of the model of the present invention for calculating the loss through forward propagation is as follows:

[0042] After calculating the loss through the BCE (Binary Cross-Entropy) loss function, calculate the gradient descent trend of the loss function obtained in each training, and perform weight analysis through back propagation. The BCE loss function formula is as follows:

[0043] LOSS=−(ylog(p(x)+(1−y)log(1−p(x))

[0044] Where p(x) is the model output and y is the true label.

[0045] It can be seen from the above technical solution that the present invention has the following beneficial effects:

[0046] 1. The deep learning training method and data set optimization method in the strapping robot vision system described in the present invention significantly improve the recognition accuracy and versatility of the model through more accurate and balanced data set creation and optimization. And through dynamic data set updates and confidence-based feedback adjustments, it can continuously adapt to different working environments, effectively respond to changes in light and external conditions, and ensure the stability and reliability of the system. The accuracy will not decrease due to changes in external conditions.

[0047] 2. The image enhancement and data balancing strategies in the present invention effectively increase the number of samples of rare categories, thereby improving the performance of the entire visual system. Arranging multiple images in different directions can greatly enhance the diversity and complexity of the training data set, thereby improving the robustness and generalization ability of the deep learning model. Image enhancement processing can artificially create more diverse data to balance the representativeness of each category in the data set and enhance the generalization ability of the model.

[0048] 3. The present invention ensures that the steel bar tying robot vision system can maintain high accuracy in identifying tying points in various complex on-site working conditions, significantly improving the practicality and work efficiency of the system. The degree of specificity of this method enables technicians in the same technical field to reproduce the process or method of the present invention based on the described content, which is conducive to the universalization of training programs.

[0049] 4. The present invention has significant advantages in improving the accuracy, stability and adaptability of visual recognition in the automated construction industry, and is conducive to the realization of visual recognition training for steel bar bundling.

[0050] 5. In the label recognition process of the present invention, when the visual model recognizes an object, it will be accompanied by a confidence standard to visualize the probability of correct recognition. When the recognition confidence of the visual model for a certain category label is lower than the threshold, the system will automatically trigger the model's retraining process. During this process, those categories with low confidence will be given priority for intensive training to improve the overall recognition accuracy and generalization ability of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 It is a structural schematic diagram of a deep learning training method and a data set optimization method in a strapping robot vision system according to the present invention;

[0052] Figure 2 A flowchart of image processing and data set construction in the present invention;

[0053] Figure 3 It is a flow chart of the scene adaptability optimization and model in the present invention;

[0054] Figure 4 It is a structural schematic diagram of the multi-arm collaborative lashing robot in the present invention;

[0055] Figure 5 It is a rear view of the multi-arm lashing cooperative lashing robot of the present invention;

[0056] Figure 6 It is a structural schematic diagram of the lashing robot in the present invention;

[0057] Figure 7It is a partial schematic diagram of the installation of the lashing mechanism and the supporting beam in the present invention;

[0058] Figure 8 It is a partial schematic diagram of the buffer structure in the present invention;

[0059] Fig. 9 This is a schematic diagram of the installation of the longitudinal and horizontal moving modules in the present invention;

[0060] Fig.10 It is a partial schematic diagram of the connection between the supporting cross beam and the longitudinal supporting beam in the present invention;

[0061] Fig.11 It is a side view of the continuous wire feeding mechanism of the present invention;

[0062] Fig.12 It is a structural schematic diagram of the continuous wire feeding mechanism in the present invention;

[0063] Fig.13 It is a partial schematic diagram of the guide wire assembly in the present invention;

[0064] Fig.14 It is a structural schematic diagram of another viewing angle in the present invention;

[0065] Fig.15 Schematic diagram of the internal structure of the wire guide mechanism in the present invention;

[0066] Fig.16 It is a layout diagram of the strapping machine in the present invention;

[0067] Fig.17 is a flow chart of the motion planning method of the present invention;

[0068] Fig.18 This is a flow chart of the strapping machine motion control in the present invention. DETAILED DESCRIPTION

[0069] The present invention is further explained below in conjunction with the accompanying drawings and specific embodiments. Example

[0070] like Figure 1 to 1 A deep learning training method and a data set optimization method in a strapping robot vision system, characterized by comprising:

[0071] 1): Carefully sample the steel bar bundling operations at industrial sites and complete the creation and annotation of data sets. That is, the staff uses industrial array cameras to collect all-round image data of the steel bar bundling operations at different time periods (including morning, noon, and evening) and under different lighting conditions (sunny, cloudy, and with lighting equipment). The collected image data is then classified and labeled and the category distribution is analyzed. The image annotation tool is used to accurately locate and mark the steel bar bundling points in each image, and their category information is recorded. The number of bundling points of different categories is counted through statistical analysis software, and a category distribution map is generated to identify the categories of bundling points with insufficient sample size in the data set.

[0072] 2): Image processing and data set construction, that is, according to the category distribution of the bundling points in the current data set, mosaic image enhancement optimization is used for the bundling point categories with insufficient sample size to obtain a preliminary dataset of steel bar bundling points, and the dataset is enhanced and trained;

[0073] 3): Targeted deep learning model training: the deep learning algorithm trains the deep learning model on the processed data set and verifies the accuracy of the model under different working conditions;

[0074] 4): Scenario adaptability optimization and model verification. After the model training is completed, the staff will conduct rigorous testing on the model in a real rebar bundling working environment, including environmental adaptability testing, performance stability testing, and final accuracy evaluation.

[0075] It should be noted that in 1), industrial area array cameras are preferred. The staff will use 4 MV-CS200-10GM model 2000w pixel CMOS Gigabit Ethernet industrial area array cameras. It can be imagined that other cameras that can meet the shooting needs can also be selected according to actual production, as long as they can meet the construction needs.

[0076] It should be noted that the deep learning algorithms in 3) may include but are not limited to yolov5, yolov8 series target detection algorithms, and RCNN series target detection algorithms.

[0077] The deep learning training method and data set optimization method in the strapping robot vision system described in this embodiment, the specific process of data set creation and labeling in 1) is as follows: Create a data set monitor, classify and count the three types of labels in different production line scenarios, collect image data of steel bar strapping points through industrial cameras, and perform 4 ~8cm rectangular calibration frame annotation, through the image data features of the bundling point, it can be distinguished into tied bundling points, untied bundling points and intersections between steel bars and side molds. The intersections of steel bars that have been tied with steel wires are distinguished as tied bundling points, and the intersections of steel bars that have not been tied with steel wires are distinguished as untied bundling points. The intersections of steel bars and side molds are identified as intersections between steel bars and side molds, thereby generating the original data set and its feature labels. According to the label category distribution in the monitor.

[0078] In the deep learning training method and data set optimization method of the strapping robot vision system described in this embodiment, the specific process of adding images in the 2) image processing and data set construction is as follows:

[0079] 201): To analyze the imbalance of data set categories, the technicians first used data analysis tools to perform detailed statistics on the image classification in the data set and identify the categories of strapping points with insufficient sample size in the collected data set; these categories are usually due to low frequency of occurrence on site or difficult to collect under certain conditions;

[0080] 202): For the categories with insufficient sample size identified in step 201), mosaic is used to enhance the image. Four images are read each time, and then the four images are flipped, translated, scaled, cropped and the color gamut is changed. The four images are then arranged in different directions, which can greatly enhance the diversity and complexity of the training data set, thereby improving the robustness and generalization ability of the deep learning model; the four images are then randomly cropped according to their respective positions, and finally the images and frames are combined to obtain a more balanced and diverse data set; the purpose of the enhancement operation is to artificially create more diverse data to balance the representativeness of each category in the data set and enhance the generalization ability of the model; 203): Enhanced data set training, the balanced and diverse data set obtained in the previous step is trained with a deep learning model. During the training process, the performance of the model in each category will be closely monitored, and those category labels with a smaller number will be segmented out of the image, processed with 40% transparency, and randomly added to other random data sets for mixed training. This method is used to balance the number of category labels in the data set. If necessary, the parameters of the model need to be adjusted. Arranging in different directions in step 202) can greatly enhance the diversity and complexity of the training data set, thereby improving the robustness and generalization ability of the deep learning model. If the accuracy of the data set is reduced after enhancement, the learning rate can be reduced, a more complex loss function can be used for calculation, or the width, depth and maximum number of channels of the network can be increased to ensure that the model can fully learn the features of the enhanced data, thereby improving the recognition accuracy of the steel bar bundling points.

[0081] In the deep learning training method and data set optimization method in the strapping robot vision system described in this embodiment, the mosaic operation execution standard 202) is determined by the following formula:

[0082]

[0083] in: is the preset image enhancement probability, is the number of samples that need to be enhanced, is the total number of samples in the dataset, is the class imbalance, To ultimately enhance the probability.

[0084] The deep learning training method and data set optimization method in the strapping robot vision system described in this embodiment, the specific process of enhancing the data set training in 3) is as follows: 2031): The enhanced image is normalized, and the pixel value range is modified to between 0 and 1. The normalization formula is as follows:

[0085]

[0086] in: is the original pixel value, The minimum pixel value of the image is usually 0; The maximum pixel value of the image, usually 255; is the normalized value; through the above processing method, all pixel values ​​are scaled to the range of 0 to 1, which helps to improve the numerical stability and convergence speed of model training; 2032): adjust the image size to 640×640, and input the image into the yolo deep learning model according to the number of channels of 64×640×640×3, where 64 is the batch size, 640×640 is the height × width of the image, and 3 is the number of color RGB (red, green, blue) channels; 2033): set the training parameters, divide the data set into training set, validation set and test set, and allocate them in the ratio of 70%, 15%, and 15%. The loss function in training uses the cross entropy loss function for iterative training; 2034): use the accuracy to visualize whether the training results are good enough. The accuracy formula is as follows:

[0087]

[0088] Among them, A is the recognition accuracy;

[0089] TP (True Positives) is the number of true classes: the model correctly predicts the positive class as positive;

[0090] TN (True Negatives) is the number of true negatives: the model correctly predicts the negative class as negative;

[0091] FP (False Positives) is the number of false positives: the model incorrectly predicts the negative class as positive;

[0092] FN (False Negatives) is the number of false negatives: the model incorrectly predicts the positive class as negative;

[0093] 2035): Finally, the learning rate is set to 0.001. In each training batch, the model calculates the loss through forward propagation, and then updates the weights through backpropagation. The loss and accuracy during training and validation are recorded and monitored.

[0094] The deep learning training method and data set optimization method in the strapping robot vision system described in this embodiment, the specific process of scene adaptability optimization and model verification in 4) is as follows: 401): real-time monitoring of model performance, through the real-time performance monitoring system, for continuous tracking and evaluation of the performance of the model in the new environment, including real-time monitoring of key indicators such as model recognition accuracy, response time and stability, and analyzing the trends and potential problems of model performance through regularly generated performance reports; 402): analysis of the difference between the new environment data and the original training set. When the model is deployed to a new working environment, the staff first conducts a series of target recognition tests and collects the recognition data of the model in the new environment. By comparing it with the original training set, the difference in category distribution between the two is analyzed to determine whether the data set needs to be updated; this analysis work will take into account the new environment. New strapping point styles, background changes, and lighting conditions that have emerged; 403): Dynamically update the dataset. According to the analysis results in step 402), if the dataset is identified to be updated, the staff will collect image data in the new environment and include it in the training set; at the same time, in order to maintain the timeliness and accuracy of the dataset, samples that are no longer representative in the new environment will be deleted. This process may include manual review of the quality of the newly added samples to ensure that the quality of the newly added data meets the training requirements; 404): Optimize the model and repeat training and verification. Using the updated dataset, the staff will restart the model training and verification process. During this process, the model parameters, such as the learning rate, regularization term, and training cycle, will be carefully adjusted to adapt to the update of the dataset. At the same time, through multiple rounds of training and cross-validation, ensure that the model maintains or exceeds the original recognition accuracy in the new environment.

[0095] In the strapping robot vision system described in this embodiment, the deep learning training method and the data set optimization method, the specific process of dynamically updating the data set in 403) is as follows: 4031): According to the previously created monitor, the distribution of the category labels is analyzed in real time to obtain the number of category labels of the data set; 4032): When the vision system enters a new industrial environment, the recognized results are analyzed, and the difference between the recognized label category distribution and the label category distribution of the training set is calculated as follows:

[0096]

[0097] in, is the difference in the distribution of class labels, is the total number of label categories, is the number of samples of a certain category in the new environment, is the total number of samples in the new environment, is the number of samples of this category in the original data set, is the total number of samples in the original data set, set the threshold α, if QUOTE When the preset threshold α is exceeded, the system will trigger an alarm and upload new environmental data, store and package the images taken in the new scene and upload them to the local training server, delete the labels in the old dataset that are too different from the new dataset categories, and add the new dataset for retraining.

[0098] The deep learning training method and data set optimization method in the strapping robot vision system described in this embodiment, when the recognition confidence of a certain category label is lower than the threshold, the system will automatically trigger the model retraining process. During this process, those categories with low confidence will be preferentially selected for key training, so as to improve the overall recognition accuracy and generalization ability of the model.

[0099] The specific calculation process of the model through forward propagation to calculate the loss is as follows:

[0100] After calculating the loss through the BCE (Binary Cross-Entropy) loss function, calculate the gradient descent trend of the loss function obtained in each training, and perform weight analysis through back propagation. The BCE loss function formula is as follows:

[0101] LOSS=−(ylog(p(x)+(1−y)log(1−p(x))

[0102] Where p(x) is the model output and y is the true label.

[0103] Example 2

[0104] The deep learning training method and the data set priority method in the visual system of the strapping robot described in this embodiment are the same, and the structure of the strapping robot is as follows: Figures 5 to 17 The strapping robot shown in the figure adopts a multi-arm collaborative strapping robot, which includes at least one strapping mechanism 1, two groups of support columns 3 and at least one support beam 5, a buffer mechanism 13 and a swing cylinder 14, wherein a continuous wire feeding mechanism 2 is provided on the strapping mechanism 1; the two groups of support columns 3 are arranged opposite to each other, and each group of support columns 3 is provided with a longitudinal support beam 4; the support beam 5 is arranged between the two longitudinal support beams 4, and is movably connected to the longitudinal support beam 4 through a mobile drive mechanism 6; the strapping mechanism 1 is arranged on the support beam 5, and is movably connected to the support beam 5 through a second mobile drive mechanism 7; as shown Figures 4 to 15 The multi-arm tying cooperative tying robot of the present invention shown in the figure, the tying mechanism 1 includes a tying structure 11, the tying structure 11 is fixed on a fixed frame 12, and a wire guide tube 18 is provided on one side of the fixed frame 12; the buffer mechanism 13 is arranged on the top of the fixed frame 12; the swing cylinder 14 is arranged above the buffer mechanism 13 and is connected to the buffer mechanism 13, and the top of the swing cylinder 14 is fixed on the tying arm 15; it also includes a calibration rod 16 for calibrating the tying arm 15, and the calibration rod 16 is arranged on the fixed frame 12.

[0105] like Figure 6 The second mobile driving mechanism 7 shown includes a transverse moving mounting plate 71, a transverse moving driving motor 72 and a transverse driving gear 73, the binding arm 15 is installed on the transverse moving mounting plate 71, the transverse moving driving motor 72 is fixed on the transverse moving mounting plate 71, the transverse driving gear 73 is installed on the output end of the transverse moving driving motor 72, and it is meshed with the transverse rack 53, and the top of the transverse moving mounting plate 71 is connected to the corresponding chain box through an adapter plate.

[0106] like Figure 8 The buffer mechanism 13 shown includes a top plate 131, a bottom plate 132 and a group of buffer springs 133. The top plate 131 and the bottom plate 132 are arranged opposite to each other up and down. The buffer spring 133 is arranged between the top plate 131 and the bottom plate 132, and the upper and lower ends are respectively connected to the top plate 131 and the bottom plate 132. The top plate 131 is connected to the output end of the swing cylinder 14, and the bottom plate 132 is connected to the disc on the top of the fixed frame 12; the buffer mechanism 13 also includes a protective cover 134, and the protective cover 134 is arranged on the outside of the buffer spring 133. The setting of the buffer mechanism can play a good buffering effect, and the setting of the spring can not only protect the binding structure, but also play a certain fine-tuning role. Figures 7 to 9As shown, it also includes a longitudinal horizontal moving module 17, which is fixed to the tying arm 5 through a cylinder fixing seat, and the top of the swing cylinder 14 is slidably connected to the slide of the longitudinal horizontal moving module 17 through a sliding seat. The setting of the swing cylinder allows it to drive the tying structure to rotate 90° to achieve cross tying. At the same time, the setting of the calibration rod can calibrate the installation position of the tying gun to improve its tying quality. The setting of the longitudinal horizontal moving module allows it to adjust the longitudinal position of the tying structure to better meet the tying needs.

[0107] like Figures 11 to 15 The continuous wire feeding mechanism 2 shown includes a wire feeding mounting frame 21, a wire guide disc assembly 22, a wire guide mechanism 23 and a wire disc brake mechanism 24. The wire guide disc assembly 22 is arranged on the wire feeding mounting frame 21, the wire guide mechanism 23 is arranged on the upper part of the wire feeding mounting frame 21, and the wire disc brake mechanism 24 is arranged at the rear of the wire guide disc assembly 22 and connected to the wire guide mechanism 23 through a connecting rod 25. The wire guide disc assembly 22 includes a wire disc support frame 221, a drive shaft 222, a wire disc 223 and a wire disc drive mechanism 224. The wire disc support frame 221 is arranged on the wire feeding mounting frame 21, the drive shaft 222 is arranged on the wire disc support frame 221, the wire disc 223 is arranged outside the drive shaft 222, and the wire disc drive mechanism 224 is connected to the wire feeding mounting frame 21 through a drive fixing frame 225, and its output end is connected to the drive shaft 222. The wire guide assembly continuously feeds wire to the wire guide mechanism. Once the binding wire becomes loose, the wire brake mechanism will brake the wire guide assembly and stop feeding wire. Fig.10The longitudinal support beam 4 shown is provided with a rack 41 on the inner side, a linear rail 42 on the upper part, and a group of drag chain boxes 43 on the outer side. The drag chain box 43 is connected to the end of the support beam 5 through an adapter frame 44, and a guide rail 45 is provided above the longitudinal support beam 4; a mobile driving plate 51 is provided at both ends of the support beam 5, a group of sliding blocks 52 are provided on the mobile driving plate 51, and the sliding blocks 52 are slidably connected to the guide rail 45, and the adapter frame 44 is connected to the mobile driving plate 51; and a transverse rack 53 is provided on the support beam 5. The mobile driving mechanism 6 includes a driving shaft 61, a first driving motor 62 and a group of driving gears 63. The driving shaft 61 is arranged parallel to the support beam 5, one end of which passes through the end of the support beam 5, and the other end is provided with a gear box, and the gear box is meshed and connected with the main driving gear at the output end of the first driving motor 62, and the driving gear 63 is arranged at both ends of the driving shaft 61, and the driving gear 63 is meshed with the corresponding rack 41. The multi-arm collaborative tying robot described in the present invention has an optimized structure, and is provided with at least one supporting beam and at least one tying structure on the beam, so that it can realize multi-arm simultaneous tying according to production needs. At the same time, a longitudinal support beam is also provided, so that the supporting beam can move along the longitudinal support beam according to actual needs, thereby completing the tying task within its own tying area, and effectively improving the tying efficiency.

[0108] like Figure 16-18The multi-arm collaborative binding robot multi-working arm collaborative motion planning method shown is as follows: 1): first, place the steel bars on the mold platform to form a steel mesh; 2): point screening, that is, the camera takes a picture of the steel bars placed on the mold platform, and the host computer processes the captured image through the image recognition program. According to the image recognition result, the coordinate information of the complete steel mesh to be tied is obtained, and a one-dimensional list, i.e., list A, is first formed according to the X coordinate in ascending order, and then the lists are sorted in ascending order according to the Y coordinate, and used as the input of this algorithm; if the user selects the full bundling mode, no processing is performed; when the user selects the plum blossom bundling mode, that is, all the edge points of the mesh are tied, and the internal points are tied one by one, and the corresponding points to be tied are selected from list A to form list A'; 3): area division, according to list A or list A', the working area of ​​each working arm is divided by area division method or quantity division method; 4): coordinate processing, that is, the point coordinates are processed using the path planning algorithm, Get the point where each working arm needs to complete the bundling, and calculate the coordinate information for driving each motor to reach the specified point; 5): The motion control algorithm drives each motor to move according to the coordinate information obtained by the previous processing, and performs real-time collision detection during the movement; 6): Bundling, that is, the mobile drive mechanism 6 drives the support beam 5 to move to the first column of bundling points, and the second mobile drive mechanism 7 simultaneously drives the working arm on the support beam 5 to move, so that each binding mechanism 1 reaches above the first binding point, and fine-tunes and compensates for the coordinate deviation of the binding mechanism 1. After the binding mechanism 1 is accurately aligned with the steel bar point in the XY direction, the working arm Z-axis motor moves to make the binding mechanism 1 descend to the working height for bundling; after bundling is completed, the Z-axis motor moves, the binding mechanism 1 rises to a safe height, and the Y-axis motor continues to move to the next point, and repeats the bundling action; when all the points in a column are bundled, the beam moves to the next column of points, and the above process is repeated until all the points are bundled.

[0109] In step 6) of the present embodiment, during the binding movement, there is a risk of collision between the supporting beams 5 and the working arms on the same supporting beam 5. Therefore, during the movement, the distance between the supporting beams 5 and the working arms is detected in real time. When the distance is less than the safe distance, one of them stops moving and enters a waiting state. When the collision risk is eliminated, the movement continues.

[0110] In the method for motion planning of multi-working arms in the present invention, the steel mesh exists in two states, as follows: (1) The steel mesh is laid flat in the entire working area of ​​the strapping machine. In this case, the area division method is adopted: first, the working area is divided into two equal areas in the X direction, and then the two areas are divided into three sub-areas in the Y direction, that is, the area is divided into six sub-areas, and the X, Y coordinates of each point and the sub-area interval range are compared, so that the point list is divided into six sub-lists, and each corresponds to a working arm; (2) The steel mesh only exists in the local working area of ​​the strapping machine. In this case, the quantity division method is adopted: the number of columns of points along the X direction is divided equally, corresponding to two beams respectively, and then the number of points in each column occupied by the beam along the Y direction is divided equally, so that each working arm can be divided into a certain number of points, avoiding the problem of excessive difference in workload of each working arm due to uneven distribution of points. Since it is impossible to ensure that each working arm obtains the same number of points in the same column (the difference is only one), the number of points allocated to each working arm is the same by adding virtual strapping points. In the multi-working arm collaborative motion planning method described in the present invention, the specific method of fine-tuning and compensating the coordinate deviation of the binding mechanism (1) in step 6) is as follows: Assume that the binding machine is provided with 6 working arms, namely W1, W2, W3, W4, W5, and W6, wherein W1, W2, and W3 are arranged on the same supporting beam Xa, and W4, W5, and W6 are arranged on another supporting beam Xb. Each working arm is equipped with two sets of driving mechanisms to realize the movement in the Y and Z axis directions respectively. Among them, the working arms W1, W3, W4, and W6 are also equipped with a fine-tuning axis driving motor, denoted as the C axis, and its movement direction is the same as the X axis; the working arms located in the middle of the beam, namely W2 and W5, are not equipped with the C axis. Therefore, the coordinate of each movement of the supporting beam 5 is the average of the maximum and minimum values ​​of the X coordinates of the column of points in the S2 and S5 areas, that is:

[0111]

[0112] max Indicates the maximum X coordinate of this column of points, x min Indicates the minimum X coordinate of this column of points, and x0 indicates the target coordinate for controlling the movement of the beam;

[0113] When the point deviates from the muzzle of the tying mechanism (1) within a certain range d, the tying action can still be completed normally. Therefore, when the deviation of the X coordinates of the points in the S2 and S5 areas from the above average exceeds the allowable error, that is:

[0114] QUOTE , x n Indicates the X coordinate of this column of points;

[0115] Then the point is divided into two outer areas (dividing the point into the area it is close to). For the working arms on both sides, the deviation between their coordinates and x0 is compensated by the C-axis. Therefore, the C-axis motion of the points on both sides is:

[0116] QUOTE , c n Indicates the target coordinates of the fine-tuning axis motion calculated by the algorithm.

[0117] In the multi-working arm collaborative motion planning method of the present invention, the movement trajectory of the strapping head of the strapping mechanism 1 during the strapping process is S-shaped, that is, after the strapping head has completed strapping a row of points along the positive direction Y, when strapping the next row of points, it moves along the negative direction Y. After the point processing is completed, the point information of each area is reordered according to the S-shaped path. Through the S-shaped motion trajectory, the idle stroke of the working arm can be shortened, the strapping efficiency can be improved, and the working time can be reduced.

[0118] In this embodiment, the motion control algorithm for multiple robotic arms working together is as follows:

[0119] (1) The X, Y, and C axis movements are independent of each other. For each working arm, the Z axis motor can only move when the X, Y, and C axes have reached the target coordinates;

[0120] (2) The host computer sends the information of each point in turn, including X, Y, and C coordinate information (the C coordinate of the point in the S2 and S5 areas is 0). Since the bundling height is fixed, it is not included in the coordinate information;

[0121] (3) Anti-collision measures, including coordinate soft limit and collision detection, are as follows: 301): Before the control mechanism sends the point information, it detects whether the point coordinate exceeds the working arm motor movement range to avoid exceeding the range. If it exceeds the range, the point movement is handed over to other working arms that can reach the position to execute;

[0122] 302): Real-time detection of whether the safety distance requirements are met between the supporting beams 5 and the working arms of the binding mechanism 1. For example, when W1 and W2 move along the positive direction of Y, if the distance between the two is less than the safety distance, W1 stops and waits; if it moves along the negative direction of Y, W2 should stop and wait.

[0123] In this embodiment, a multi-working arm collaborative motion planning method described in the present invention can effectively improve the production efficiency of prefabricated components by adopting multiple working arms to work collaboratively. During the bundling process, the bundling point is photographed by a camera, and the bundling point information is identified by image processing. The motion planning algorithm is used to complete the motion planning of multiple bundling arms, so that multiple bundling arms can quickly perform bundling. It can adapt to multiple working modes and can quickly and accurately complete the motion planning of the working arms of the bundling machine. At the same time, in the motion planning, the coordination of coordinate soft limit and collision detection can avoid collision between the working arms, providing multiple safety guarantees for the operation of the machine.

[0124] The multi-working arm collaborative motion planning method described in the present invention can choose to adopt the area division method or the quantity division method according to the state of the steel mesh, which can balance the workload of each working arm and enable it to adapt to various working modes to avoid the problem of excessive workload differences.

Claims

1. A deep learning training method and a data set optimization method in a strapping robot vision system, characterized in that: include: 1): Carefully sample the steel bar bundling operations at industrial sites and complete the creation and annotation of data sets. That is, the staff uses industrial array cameras to collect all-round image data of the steel bar bundling operations at different time periods and under different lighting conditions, and then classifies and labels the collected image data and analyzes the category distribution; the image annotation tool is used to accurately locate and mark the steel bar bundling points in each image, and its category information is recorded. The number of bundling points of different categories is counted through statistical analysis software, and a category distribution map is generated to identify the categories of bundling points with insufficient sample size in the data set; 2): Image processing and data set construction, that is, according to the category distribution of the bundling points in the current data set, mosaic image enhancement optimization is used for the bundling point categories with insufficient sample size to obtain a preliminary dataset of steel bar bundling points, and the dataset is enhanced and trained; 3): Targeted deep learning model training: Use the processed data set to train the deep learning model and verify the accuracy of the model under different working conditions; 4): Scenario adaptability optimization and model verification. After the model training is completed, the staff will conduct rigorous testing on the model in a real steel bar bundling work environment, including environmental adaptability testing and performance stability testing, as well as final accuracy evaluation; The specific process of image enhancement in 2) image processing and dataset construction is as follows: 201): To analyze the imbalance of dataset categories, the technicians first used data analysis tools to perform detailed statistics on the image categories in the dataset and identified the categories of tying points with insufficient sample size in the collected dataset; 202): For the categories with insufficient sample size identified in step 201), mosaic technology is used to enhance the image. Four images are read each time, and then the four images are flipped, translated, scaled, cropped and the color gamut is changed. After that, the four images are arranged in different directions, which can greatly enhance the diversity and complexity of the training data set, thereby improving the robustness and generalization ability of the deep learning model; the four images are randomly cropped according to their respective positions, and finally the images and frames are combined to obtain a more balanced and diversified data set; 203): Enhanced dataset training: The balanced and diversified dataset obtained in the previous step is trained using a deep learning model. During the training process, the performance of the model in each category is closely monitored. The category labels with a small number are segmented out in the image, processed with 40% transparency, and randomly added to other random datasets for mixed training. This method balances the number of category labels in the dataset and can adjust the model parameters as needed. The 202) mosaic operation execution standard is determined by the following formula: in: is the preset image enhancement probability, is the number of samples that need to be enhanced, is the total number of samples in the dataset, is the class imbalance, To ultimately enhance the probability; The specific process of scenario adaptability optimization and model verification in 4) is as follows: 401): Real-time monitoring of model performance, through a real-time performance monitoring system, to continuously track and evaluate the performance of the model in the new environment, including real-time monitoring of key indicators of model recognition accuracy, response time and stability, and analyzing the performance trends and potential problems of the model through regularly generated performance reports; 402): Analyze the differences between the new environment data and the original training set. When the model is deployed in a new working environment, the staff first conducts a series of target recognition tests and collects the recognition data of the model in the new environment. By comparing it with the original training set, the difference in category distribution between the two is analyzed to determine whether the data set needs to be updated. This analysis will take into account the new strapping point styles, background changes, and lighting conditions that appear in the new environment; 403): Dynamically update the data set. According to the analysis results in step 402), if the data set needs to be updated, the staff will collect image data in the new environment and include it in the training set. At the same time, in order to maintain the timeliness and accuracy of the data set, those samples that are no longer representative in the new environment will be deleted. This process includes manually reviewing the quality of the newly added samples to ensure that the quality of the newly added data meets the training requirements. 404): Optimize the model and repeat training and validation. Using the updated data set, the staff will restart the model training and validation process. During this process, the model parameters, learning rate, regularization term and training cycle will be carefully adjusted to adapt to the update of the data set. At the same time, through multiple rounds of training and cross-validation, it is ensured that the model maintains or exceeds the original recognition accuracy in the new environment; The specific process of dynamically updating the data set in 403) is as follows: 4031): According to the monitor created previously, the distribution of category labels is analyzed in real time to obtain the number of category labels in the data set; 4032): When the visual system enters a new industrial environment, the recognition results are analyzed and the difference between the recognized label category distribution and the label category distribution of the training set is calculated as follows: in, is the difference in the distribution of class labels, is the total number of label categories, is the number of samples of a certain category in the new environment, is the total number of samples in the new environment, is the number of samples of this category in the original data set, is the total number of samples in the original data set, and the threshold α is set. When the preset threshold α is exceeded, the system will trigger an alarm and upload new environmental data, store and package the images taken in the new scene and upload them to the local training server, delete the labels in the old dataset that are too different from the new dataset categories, and add the new dataset for retraining.

2. The deep learning training method and data set optimization method in the strapping robot vision system according to claim 1, characterized in that: The specific process of data set creation and labeling in 1) is as follows: Create a data set monitor, classify and count the three types of labels in different production line scenarios, collect image data of steel bar bundling points through industrial cameras, and perform 4 ~ The 8cm rectangular calibration frame is annotated. Through the image data features of the bundling points, they can be distinguished into tied bundling points, untied bundling points and the intersection points of the steel bars and the side molds. The intersection points of the steel bars that have been tied with steel wires are distinguished as tied bundling points, and the intersection points of the steel bars that have not been tied with steel wires are distinguished as untied bundling points. The intersection points of the steel bars and the side molds are identified as the intersection points of the steel bars and the side molds, thereby generating the original data set and its feature labels.

3. The deep learning training method and data set optimization method in the strapping robot vision system according to claim 1, characterized in that: The specific process of the enhanced data set training in 3) is as follows: 2031): The enhanced image is normalized and the pixel value range is modified to be between 0 and 1. The normalization formula is as follows: in: is the original pixel value, is the minimum pixel value of the image, which is 0; is the maximum pixel value of the image, which is 255; is the normalized value; Through the above processing, all pixel values ​​are scaled to the range of 0 to 1, which helps to improve the numerical stability and convergence speed of model training; 2032): Resize the image to 640×640, and input the image into the YOLO deep learning model with a channel number of 64×640×640×3, where 64 is the batch size, 640×640 is the height × width of the image, and 3 is the number of color RGB channels; 2033): Set the training parameters and divide the data set into training set, validation set and test set in the proportion of 70%, 15% and 15%. The loss function in training adopts the cross entropy loss function for iterative training; 2034): Use accuracy to visualize whether the training results are good or not. The accuracy formula is as follows: Among them, A is the recognition accuracy; TP is the number of true positives: the model correctly predicts the positive class as positive; TN is the number of true negatives: the model correctly predicts the negative class as negative; FP is the number of false positives: the model incorrectly predicts a negative class as positive; FN is the number of false negatives: the model incorrectly predicts a positive class as a negative class; 2035): Finally, the learning rate is set to 0.

001. In each training batch, the model calculates the loss through forward propagation, and then updates the weights through backpropagation. The loss and accuracy during training and validation are recorded and monitored.

4. The deep learning training method and data set optimization method in the strapping robot vision system according to claim 3, characterized in that: The recognition process adopts a confidence-based label recognition method, specifically: when the visual model recognizes an object, it will be accompanied by a confidence standard to visualize the correct probability of recognition. When the recognition confidence of the visual model for a certain category label is lower than the threshold, the system will automatically trigger the model's retraining process. During this process, those categories with low confidence will be given priority for intensive training to improve the overall recognition accuracy and generalization ability of the model.

5. The deep learning training method and data set optimization method in the strapping robot vision system according to claim 1, characterized in that: The specific calculation process of the model through forward propagation to calculate the loss is as follows: After calculating the loss through the BCE (Binary Cross-Entropy) loss function, calculate the gradient descent trend of the loss function obtained in each training, and perform weight analysis through back propagation. The BCE loss function formula is as follows: Where p(x) is the model output and y is the true label.

Images