Automatic welding method and equipment for tunnel steel grating and storage medium

Abstract

The invention provides an automatic welding method and device for a tunnel steel grating and a storage medium. The method comprises the steps that gray level image data of the tunnel steel grating are collected; executing a first target detection operation on the grayscale image data to obtain a first welding point to be welded on a main reinforcing steel bar in the tunnel steel grid; the tunnel steel grating collects original infrared image data; adjusting the original infrared image data according to the current of the welding robot to obtain target infrared image data; converting the target infrared image data into optical flow field image data; under the reference of the gray level image data, second target detection operation is carried out on the optical flow field image data, and a second welding point generated when the welding robot carries out welding on the tunnel steel grid is obtained; and controlling the welding robot to continuously weld the tunnel steel grating according to the first welding point and the second welding point. According to the embodiment, welding tracking of the tunnel steel grid is achieved, correction is conducted in real time during welding, and the accuracy of welding of the tunnel steel grid is improved.

Description

Technical Field

[0001] The embodiments of the present application belong to the technical field of computer vision, and particularly relate to an automatic welding method, device and storage medium for tunnel steel grids. Background Art

[0002] A tunnel steel grid is a primary support steel frame for large-section soft surrounding rock tunnels, and is mostly used for the initial support and protection of tunnels.

[0003] Currently, welding robots are mostly used to assist in welding. Before welding, computer vision technology is used to locate the welding points on the tunnel steel grid, so as to control the welding robot to weld the welding points.

[0004] However, the welding robot generates high temperature and heat during welding, which hinders the welding tracking of the tunnel steel grid and may cause deviations during welding. Summary of the Invention

[0005] In view of this, the embodiments of the present application provide an automatic welding method, device and storage medium for tunnel steel grids, so as to perform full-process tracking on the welding of tunnel steel grids and improve the welding accuracy.

[0006] The first aspect of the embodiments of the present application provides an automatic welding method for tunnel steel grids, including:

[0007] Before starting the welding robot to weld the tunnel steel grid, collect grayscale image data of the tunnel steel grid;

[0008] Perform a first target detection operation on the grayscale image data to obtain the first welding points waiting to be welded on the main steel bars in the tunnel steel grid;

[0009] When starting the welding robot to weld the tunnel steel grid according to the first welding points, collect original infrared image data of the tunnel steel grid;

[0010] Adjust the original infrared image data according to the current of the welding robot to obtain target infrared image data;

[0011] Convert the target infrared image data into optical flow field image data;

[0012] Under the reference of the grayscale image data, perform a second target detection operation on the optical flow field image data to obtain the second welding points generated when the welding robot welds on the tunnel steel grid;

[0013] Control the welding robot to continue welding the tunnel steel grid according to the first welding points and the second welding points.

[0014] The second aspect of the embodiments of the present application provides a tunnel steel grid automatic welding device, including:

[0015] A grayscale image data acquisition module, configured to acquire grayscale image data of the tunnel steel grid before starting the welding robot to weld the tunnel steel grid;

[0016] A first welding point detection module, configured to perform a first target detection operation on the grayscale image data to obtain a first welding point waiting to be welded on the main steel bars in the tunnel steel grid;

[0017] An infrared image data acquisition module, configured to acquire original infrared image data of the tunnel steel grid when starting the welding robot to weld the tunnel steel grid according to the first welding point;

[0018] An infrared image data adjustment module, configured to adjust the original infrared image data according to the current of the welding robot to obtain target infrared image data;

[0019] An optical flow field image data conversion module, configured to convert the target infrared image data into optical flow field image data;

[0020] A second welding point detection module, configured to perform a second target detection operation on the optical flow field image data with reference to the grayscale image data to obtain a second welding point generated when the welding robot welds on the tunnel steel grid;

[0021] A welding control module, configured to control the welding robot to continue welding the tunnel steel grid according to the first welding point and the second welding point.

[0022] The third aspect of the embodiments of the present application provides a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the tunnel steel grid automatic welding method described in the first aspect above is implemented.

[0023] The fourth aspect of the embodiments of the present application provides a computer-readable storage medium, where the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the tunnel steel grid automatic welding method described in the first aspect above is implemented.

[0024] The fifth aspect of the embodiments of the present application provides a computer program product, which when running on a computer causes the computer to execute the tunnel steel grid automatic welding method described in the first aspect above.

[0025] In this embodiment, before starting the welding robot to weld the tunnel steel grid, grayscale image data of the tunnel steel grid is collected; a first target detection operation is performed on the grayscale image data to obtain the first welding points waiting to be welded on the main steel bars in the tunnel steel grid; when starting the welding robot to weld the tunnel steel grid according to the first welding points, original infrared image data of the tunnel steel grid is collected; the original infrared image data is adjusted according to the current of the welding robot to obtain target infrared image data; the target infrared image data is converted into optical flow field image data; under the reference of the grayscale image data, a second target detection operation is performed on the optical flow field image data to obtain the second welding points generated when the welding robot welds on the tunnel steel grid; the welding robot is controlled to continue welding the tunnel steel grid according to the first welding points and the second welding points. In this embodiment, by using the grayscale image data when not welding as a reference and suppressing the flash generated during the welding of the welding robot, etc., the welding points generated when the welding robot welds on the tunnel steel grid are detected, so as to realize the welding tracking of the tunnel steel grid, perform real-time correction during welding, and improve the accuracy of welding the tunnel steel grid. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings required for use in the embodiments or the description of the prior art. Obviously, the following drawings are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0027] Figure 1 is a schematic diagram of an automatic welding method for a tunnel steel grid provided by an embodiment of the present application;

[0028] Figure 2 is a schematic diagram of the first target detection network and the second target detection network provided by an embodiment of the present application;

[0029] Figure 3 is a schematic diagram of the fusion module provided by an embodiment of the present application;

[0030] Figure 4 is a schematic diagram of an automatic welding device for a tunnel steel grid provided by an embodiment of the present application;

[0031] Figure 5 is a schematic diagram of a terminal device provided by an embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0032] In the following description, specific details such as specific system architectures and technologies are presented for the purpose of illustration rather than limitation, so as to thoroughly understand the embodiments of the present application. However, those skilled in the art should clearly understand that the present application can also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid unnecessary details from interfering with the description of the present application.

[0033] The technical solution of the present application will be described below through specific embodiments.

[0034] Refer to Figure 1 , which shows a schematic diagram of an automatic welding method for tunnel steel grids provided by an embodiment of the present application. Specifically, it may include the following steps:

[0035] Step 101: Before starting the welding robot to weld the tunnel steel grid, collect grayscale image data of the tunnel steel grid.

[0036] Generally, a tunnel steel grid includes six longitudinal main steel bars, stirrups fixed on the periphery of the longitudinal main steel bars, and structural bars welded between adjacent longitudinal main steel bars. Among them, the structural bars are arranged at intervals of 8-shaped steel frames and X-shaped steel frames, and adjacent tunnel steel grids are connected by corresponding A joints, B joints, C joints, and D joints.

[0037] In this embodiment, a first industrial camera and a second industrial camera are configured in support of the welding robot. Among them, the first industrial camera is an RGB (red, green, blue) industrial camera, and the second industrial camera is an IR (infrared) industrial camera.

[0038] The welding robot, the first industrial camera, and the second industrial camera have been jointly calibrated, and a transformation matrix (such as a rotation matrix, a translation matrix, etc.) has been established between the two coordinate systems.

[0039] Before starting the welding robot to weld the tunnel steel grid, the first industrial camera can be controlled to collect color image data of the tunnel steel grid, and the color image data can be grayscale processed to obtain grayscale image data.

[0040] Step 102: Perform a first target detection operation on the grayscale image data to obtain the first welding points waiting to be welded on the main steel bars in the tunnel steel grid.

[0041] In this embodiment, a target detection algorithm can be used to perform a first target detection operation on the grayscale image data to obtain the first welding points waiting to be welded on the main steel bars in the tunnel steel grid, that is, the first welding points are the welding points planned by the welding robot.

[0042] When there are multiple first welding points waiting to be welded on the grayscale image data, track and weld the first welding points in a preset order (such as from left to right, from top to bottom, etc.).

[0043] In a specific implementation, a first target detection network can be loaded, such as the YOLO (You Only Look Once) series, etc. Among them, as Figure 2 shown, the first target detection network includes a first backbone network Backbone_1, a first neck network Neck_1, and a first head network Head_1.

[0044] Among them, the first backbone network Backbone_1 is the main component of the first target detection network, usually a convolutional neural network (CNN) or a residual neural network (ResNet), etc. Then, the grayscale image data can be input into the first backbone network Backbone_1 to extract the first grayscale image features for subsequent processing and analysis. The first backbone network Backbone_1 usually has many layers and many parameters and can extract high-level feature representations of the grayscale image data.

[0045] The first neck network Neck_1 is an intermediate layer connecting the first backbone network Backbone_1 and the first head network Head_1, usually using convolutional layers, pooling layers, or fully connected layers, etc. The first grayscale image features can be input into the first neck network Neck_1 to be converted into second grayscale image features, thereby reducing the dimension or adjusting the first grayscale image features from the first backbone network Backbone_1 to better meet the requirements for detecting welding points.

[0046] The first head network Head_1 is the last layer of the first target detection network, usually a classifier or a bounding box regressor. The second grayscale image features can be input into the first head network Head_1 to detect the first welding points waiting to be welded on the main steel bars in the tunnel steel grid.

[0047] Step 103: When starting the welding robot to weld the tunnel steel grid according to the first welding points, collect the original infrared image data of the tunnel steel grid.

[0048] In this embodiment, using the transformation matrix between the first industrial camera and the welding robot, the first welding points are transformed from the coordinate system of the first industrial camera to the coordinate system of the welding robot, so as to start the welding robot to weld the tunnel steel grid according to the first welding points.

[0049] At this time, the second industrial camera can be controlled to collect the original infrared image data of the tunnel steel grid.

[0050] Step 104: Adjust the original infrared image data according to the current of the welding robot to obtain the target infrared image data.

[0051] The welding robot has a welding torch. A welding wire is arranged in the welding torch. The welding torch, the welding wire and the first welding point form a circuit. The welding robot supplies current to the welding wire, and an electric arc is generated between the welding wire and the first welding point. The high temperature of the electric arc melts the welding wire to form a molten droplet, and the molten droplet drops into the first welding point to achieve welding. At the same time, a large amount of heat is generated during welding and a flash is radiated.

[0052] Generally, the current of the welding robot is positively correlated with the intensity of the flash. That is, the greater the current of the welding robot, the more heat is generated and the higher the intensity of the radiated flash.

[0053] Therefore, the original infrared image data can be adaptively adjusted according to the current of the welding robot to alleviate the influence of the flash, obtain the target infrared image data, and improve the accuracy of tracking welding.

[0054] In practical applications, the infrared values (IR values) of each pixel point in the original infrared image data can be sorted to obtain an infrared sequence, and a specified quantile (such as 50%) is taken in the infrared sequence as the threshold.

[0055] Traverse each pixel point in the original infrared image data in a certain order (such as from left to right, from top to bottom, etc.).

[0056] During the process of traversing each pixel point in the original infrared image data, a window is added with the current pixel point as the center, and the average value of the infrared values of the pixel points within the window is calculated to obtain a filtered value, and the filtered value is compared with the threshold.

[0057] If the filtered value is less than or equal to the threshold, the infrared value of the current pixel point remains unchanged.

[0058] If the filtered value is greater than the threshold, the infrared value of the current pixel point is adaptively attenuated according to the current of the welding robot.

[0059] Exemplarily, the current of the welding robot is mapped to a reference infrared value; wherein, the reference infrared value is positively correlated with the current of the welding robot.

[0060] The current of the welding robot, the reference infrared value and the infrared value of the current pixel point are input into the following formula for operation to attenuate the infrared value of the current pixel point:

[0061]

[0062] wherein, IR new is the infrared value after attenuation of the current pixel point, IR oldIR is the infrared value before the attenuation of the current pixel point, G is the preset gain coefficient, and IR base is the reference infrared value, and γ is the preset attenuation coefficient.

[0063] In this way, high-frequency infrared signals near the welding point can be filtered out.

[0064] If all pixel points in the original infrared image data are traversed, then output each pixel point to obtain the target infrared image data.

[0065] Step 105: Convert the target infrared image data into optical flow field image data.

[0066] In this embodiment, the optical flow method can be used to convert multiple frames of target infrared image data into multiple frames of optical flow field image data.

[0067] In the optical flow method, the difference in the gray-scale distribution of different target infrared image data in the sequence composed of multiple frames of target infrared image data can be utilized to obtain the description of the motion field of the moving image, and transfer the motion field in space to the image, which is represented as optical flow field image data.

[0068] The two-dimensional vector field characteristics of the optical flow field can reflect the change trend of the gray scale at each point on the target infrared image data, as well as the instantaneous velocity field generated by the movement of the pixel points with gray scale on the plane of the target infrared image data.

[0069] Step 106: Under the reference of the gray-scale image data, perform a second target detection operation on the optical flow field image data to obtain the second welding point generated when the welding robot welds on the tunnel steel grid.

[0070] When the first industrial camera and the second industrial camera are fixed, the content collected by the first industrial camera and the second industrial camera remains basically unchanged. Therefore, the gray-scale image data collected by the first industrial camera before welding can be used as a reference. In this case, use the target detection algorithm to perform a second target detection operation on the optical flow field image data to obtain the second welding point generated when the welding robot welds on the tunnel steel grid, that is, the second welding point is the actual welding point of the welding robot.

[0071] In an embodiment of the present application, step 106 may include the following steps:

[0072] Step 1061: Load the second target detection network.

[0073] In a specific implementation, the second target detection network can be loaded, such as the YOLO series, etc. Among them, such as Figure 2As shown in the figure, the second object detection network includes a second backbone network Backbone_2, a fusion module (Fusion Module, FM), a second neck network Neck_2, and a second head network Head_2.

[0074] In one design, the structure of the first backbone network Backbone_1 is the same as that of the second backbone network Backbone_2, the structure of the first neck network Neck_1 is the same as that of the second neck network Neck_2, and the structure of the first head network Head_1 is the same as that of the second head network Head_2.

[0075] Step 1062: Input the optical flow field image data into the first backbone network to extract the first optical flow field image features.

[0076] The second backbone network Backbone_2 is the main component of the second object detection network and is usually a convolutional neural network (CNN) or a residual neural network (ResNet), etc. Then, the optical flow field image data can be input into the second backbone network Backbone_2 to extract the first optical flow field image features for subsequent processing and analysis. The second backbone network Backbone_2 usually has many layers and many parameters and can extract high-level feature representations of the optical flow field image data.

[0077] Step 1063: Input the first grayscale image features and the first optical flow field image features into the fusion module to fuse them into the second optical flow field image features.

[0078] In this embodiment, the first grayscale image features and the first optical flow field image features can be input into the fusion module FM to fuse them into the second optical flow field image features.

[0079] In one design, as Figure 3 shown, the first backbone network Backbone_1 outputs the first grayscale image feature E1 in the first stage, the first grayscale image feature E2 in the second stage, and the first grayscale image feature E3 in the third stage. Among them, the scale of the first grayscale image feature E1 in the first stage is larger than the scale of the first grayscale image feature E2 in the second stage, and the scale of the first grayscale image feature E2 in the second stage is larger than the scale of the first grayscale image feature E3 in the third stage.

[0080] Accordingly, the second backbone network Backbone_2 outputs the first optical flow field image feature F1 in the first stage, the first optical flow field image feature F2 in the second stage, and the first optical flow field image feature F3 in the third stage. Among them, the scale of the first optical flow field image feature F1 in the first stage is larger than that of the first optical flow field image feature F2 in the second stage, and the scale of the first optical flow field image feature E2 in the second stage is larger than that of the first optical flow field image feature F3 in the third stage.

[0081] The fusion module FM includes a first attention layer Attention_1, a second attention layer Attention_2, and a third attention layer Attention_3.

[0082] In this design, the first grayscale image feature E1 in the first stage and the first optical flow field image feature F1 in the first stage are input into the first attention layer Attention_1 for fusion into the first fusion image feature.

[0083] The first grayscale image feature E2 in the second stage and the first optical flow field image feature F2 in the second stage are input into the second attention layer Attention_2 for fusion into the second fusion image feature.

[0084] The first grayscale image feature E3 in the third stage and the first optical flow field image feature F3 in the third stage are input into the third attention layer Attention_3 for fusion into the third fusion image feature.

[0085] The third fusion image feature is upsampled by UpSample to obtain the first intermediate image feature H1.

[0086] The first intermediate image feature H1 and the second fusion image feature are concatenated by Concat to obtain the second intermediate image feature H2.

[0087] The second intermediate image feature is upsampled by UpSample to obtain the third intermediate image feature H3.

[0088] The third intermediate image feature H3 and the first fusion image feature are concatenated by Concat to obtain the fourth intermediate image feature H4.

[0089] The fourth intermediate image feature H4 is downsampled by DownSample to obtain the second optical flow field image feature G1 in the first stage.

[0090] The second optical flow field image feature G1 in the first stage and the third intermediate image feature H3 are concatenated by Concat to obtain the fifth intermediate image feature H5.

[0091] Downsample the fifth intermediate image feature H5 to obtain the second optical flow field image feature G2 of the second stage.

[0092] Concatenate the second optical flow field image feature G2 of the second stage with the first intermediate image feature H1 to obtain the sixth intermediate image feature H6.

[0093] Downsample the sixth intermediate image feature H6 to obtain the second optical flow field image feature G3 of the third stage.

[0094] Step 1064: Input the second optical flow field image feature into the second neck network to convert it into the third optical flow field image feature.

[0095] The second neck network Neck_2 is an intermediate layer connecting the fusion module FM and the second head network Head_2, usually using convolutional layers, pooling layers, or fully connected layers, etc. The second optical flow field image feature can be input into the second neck network Neck_2 to be converted into the third optical flow field image feature, thereby reducing the dimension or adjusting the second optical flow field image feature from the fusion module FM to better meet the requirements of detecting welding points.

[0096] Step 1065: Input the third optical flow field image feature into the second head network to detect the second welding points generated when the welding robot welds on the tunnel steel grid.

[0097] The second head network Head_2 is the last layer of the second object detection network, usually a classifier and a bounding box regressor. The third optical flow field image feature can be input into the second head network Head_2 to detect the second welding points generated when the welding robot welds on the tunnel steel grid.

[0098] Step 107: Control the welding robot to continue welding the tunnel steel grid according to the first welding point and the second welding point.

[0099] In this embodiment, using the transformation matrix between the second industrial camera and the welding robot, the second welding point is transformed from the coordinate system of the second industrial camera to the coordinate system of the welding robot, and the welding of the welding robot is corrected according to the deviation between the first welding point and the second welding point, so as to control the welding robot to continue welding the tunnel steel grid.

[0100] In specific implementation, a welding range is generated with the first welding point as the center and a specified distance as the radius, and the second welding point is compared with the welding range.

[0101] If the second welding point is within the welding range, continue to observe the welding robot welding the tunnel steel grid.

[0102] If the second welding point is outside the welding range, calculate the vector pointing from the second welding point to the first welding point. Use PID (Proportion-Integral-Differential) to control the movement of the observation welding robot according to the vector, and weld the tunnel steel grid.

[0103] In this embodiment, before starting the welding robot to weld the tunnel steel grid, grayscale image data of the tunnel steel grid is collected; a first target detection operation is performed on the grayscale image data to obtain a first welding point waiting to be welded on the main steel bar in the tunnel steel grid; when starting the welding robot to weld the tunnel steel grid according to the first welding point, original infrared image data of the tunnel steel grid is collected; the original infrared image data is adjusted according to the current of the welding robot to obtain target infrared image data; the target infrared image data is converted into optical flow field image data; under the reference of the grayscale image data, a second target detection operation is performed on the optical flow field image data to obtain a second welding point generated when the welding robot welds on the tunnel steel grid; the welding robot is controlled to continue welding the tunnel steel grid according to the first welding point and the second welding point. In this embodiment, by using the grayscale image data during non-welding as a reference and suppressing the flash generated during the welding of the welding robot, etc., the welding points generated when the welding robot welds on the tunnel steel grid are detected, thereby realizing welding tracking of the tunnel steel grid, performing real-time correction during welding, and improving the accuracy of welding the tunnel steel grid.

[0104] It should be noted that the magnitudes of the sequence numbers of the steps in the above embodiments do not mean the order of execution. The order of execution of each process should be determined according to its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of the present application.

[0105] Refer to Figure 4 , which shows a schematic diagram of an automatic welding device for a tunnel steel grid provided by an embodiment of the present application, and may specifically include the following modules:

[0106] The grayscale image data acquisition module 401 is used to collect grayscale image data of the tunnel steel grid before starting the welding robot to weld the tunnel steel grid;

[0107] The first welding point detection module 402 is used to perform a first target detection operation on the grayscale image data to obtain a first welding point waiting to be welded on the main steel bar in the tunnel steel grid;

[0108] The infrared image data acquisition module 403 is used to collect original infrared image data of the tunnel steel grid when starting the welding robot to weld the tunnel steel grid according to the first welding point;

[0109] An infrared image data adjustment module 404, configured to adjust the original infrared image data according to the current of the welding robot to obtain target infrared image data;

[0110] An optical flow field image data conversion module 405, configured to convert the target infrared image data into optical flow field image data;

[0111] A second welding point detection module 406, configured to perform a second target detection operation on the optical flow field image data with reference to the grayscale image data to obtain a second welding point generated when the welding robot welds on the tunnel steel grid;

[0112] A welding control module 407, configured to control the welding robot to continue welding the tunnel steel grid according to the first welding point and the second welding point.

[0113] In an embodiment of the present application, the first welding point detection module 402 is further configured to:

[0114] Load a first target detection network; the first target detection network includes a first backbone network, a first neck network, and a first head network;

[0115] Input the grayscale image data into the first backbone network to extract first grayscale image features;

[0116] Input the first grayscale image features into the first neck network to convert them into second grayscale image features;

[0117] Input the second grayscale image features into the first head network to detect a first welding point waiting to be welded on the main steel bars in the tunnel steel grid.

[0118] In an embodiment of the present application, the second welding point detection module 406 is further configured to:

[0119] Load a second target detection network; the second target detection network includes a second backbone network, a fusion module, a second neck network, and a second head network;

[0120] Input the optical flow field image data into the first backbone network to extract first optical flow field image features;

[0121] Input the first grayscale image features and the first optical flow field image features into the fusion module to fuse them into second optical flow field image features;

[0122] Input the second optical flow field image features into the second neck network to convert them into third optical flow field image features;

[0123] Input the third optical flow field image feature into the second head network to detect the second welding point generated when the welding robot welds on the tunnel steel grid.

[0124] In one embodiment of the present application, the first backbone network outputs the first grayscale image feature in the first stage, the second stage, and the third stage in sequence, and the second backbone network outputs the first optical flow field image feature in the first stage, the second stage, and the third stage in sequence; the fusion module includes a first attention layer, a second attention layer, and a third attention layer;

[0125] The second welding point detection module 406 is further configured to:

[0126] Input the first grayscale image feature in the first stage and the first optical flow field image feature in the first stage into the first attention layer to fuse them into a first fused image feature;

[0127] Input the first grayscale image feature in the second stage and the first optical flow field image feature in the second stage into the second attention layer to fuse them into a second fused image feature;

[0128] Input the first grayscale image feature in the third stage and the first optical flow field image feature in the third stage into the third attention layer to fuse them into a third fused image feature;

[0129] Upsample the third fused image feature to obtain a first intermediate image feature;

[0130] Concatenate the first intermediate image feature and the second fused image feature to obtain a second intermediate image feature;

[0131] Upsample the second intermediate image feature to obtain a third intermediate image feature;

[0132] Concatenate the third intermediate image feature and the first fused image feature to obtain a fourth intermediate image feature;

[0133] Downsample the fourth intermediate image feature to obtain the second optical flow field image feature in the first stage;

[0134] Concatenate the second optical flow field image feature in the first stage and the third intermediate image feature to obtain a fifth intermediate image feature;

[0135] Downsample the fifth intermediate image feature to obtain the second optical flow field image feature in the second stage;

[0136] Concatenate the second optical flow field image feature in the second stage and the first intermediate image feature to obtain a sixth intermediate image feature;

[0137] Downsample the sixth intermediate image feature to obtain the second optical flow field image feature in the third stage.

[0138] In an embodiment of the present application, the structure of the first backbone network is the same as that of the second backbone network, the structure of the first neck network is the same as that of the second neck network, and the structure of the first head network is the same as that of the second head network.

[0139] In an embodiment of the present application, the infrared image data adjustment module 404 is further configured to:

[0140] Sort the infrared values of each pixel point in the original infrared image data to obtain an infrared sequence;

[0141] Take a specified quantile in the infrared sequence as the threshold;

[0142] Add a window centered on the current pixel point during the process of traversing each pixel point in the original infrared image data;

[0143] Calculate the average value of the infrared values of the pixel points within the window to obtain a filtered value;

[0144] If the filtered value is less than or equal to the threshold, keep the infrared value of the current pixel point unchanged;

[0145] If the filtered value is greater than the threshold, attenuate the infrared value of the current pixel point according to the current of the welding robot;

[0146] If all pixel points in the original infrared image data have been traversed, output each pixel point to obtain the target infrared image data.

[0147] In an embodiment of the present application, the infrared image data adjustment module 404 is further configured to:

[0148] Map the current of the welding robot to a reference infrared value; the reference infrared value is positively correlated with the current of the welding robot;

[0149] Input the current of the welding robot, the reference infrared value, and the infrared value of the current pixel point into the following formula for calculation to attenuate the infrared value of the current pixel point:

[0150]

[0151] where, IR new is the infrared value after attenuation of the current pixel point, IR old is the infrared value before attenuation of the current pixel point, G is a preset gain coefficient, IR base is the reference infrared value, and γ is a preset attenuation coefficient.

[0152] In one embodiment of the present application, the welding control module 407 is further configured to:

[0153] Generate a welding range centered on the first welding point;

[0154] If the second welding point is within the welding range, continue to observe the welding robot welding the tunnel steel grid;

[0155] If the second welding point is outside the welding range, calculate the vector pointing from the second welding point to the first welding point;

[0156] Control and observe the movement of the welding robot according to the vector, and weld the tunnel steel grid.

[0157] An automatic welding device for tunnel steel grids provided by an embodiment of the present application can implement each step in the foregoing method embodiments when this device is applied.

[0158] For the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple. For related parts, refer to the description in the method embodiment part.

[0159] Refer to Figure 5 , which shows a schematic diagram of a terminal device provided by an embodiment of the present application. As Figure 5 shown, the terminal device 500 in the embodiment of the present application includes: a processor 510, a memory 520, and a computer program 521 stored in the memory 520 and executable on the processor 510. When the processor 510 executes the computer program 521, the steps in each embodiment of the above-mentioned automatic welding method for tunnel steel grids are implemented. Alternatively, when the processor 510 executes the computer program 521, the functions of each module / unit in each device embodiment above are implemented.

[0160] Exemplarily, the computer program 521 can be divided into one or more modules / units. The one or more modules / units are stored in the memory 520 and executed by the processor 510 to complete the present application. The one or more modules / units can be a series of computer program instruction segments capable of performing specific functions, and these instruction segments can be used to describe the execution process of the computer program 521 in the terminal device 500.

[0161] The terminal device 500 may include, but is not limited to, a processor 510 and a memory 520. Those skilled in the art can understand that Figure 5This is only an example of the terminal device 500, which does not constitute a limitation on the terminal device 500. It may include more or fewer components than those shown in the figure, or combine certain components, or different components. For example, the terminal device 500 may also include input / output devices, network access devices, buses, etc.

[0162] The processor 510 may be a central processing unit (CPU), or may also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor, or the processor may also be any conventional processor, etc.

[0163] The memory 520 may be an internal storage unit of the terminal device 500, such as the hard disk or memory of the terminal device 500. The memory 520 may also be an external storage device of the terminal device 500, such as a plug-in hard disk equipped on the terminal device 500, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. Further, the memory 520 may also include both the internal storage unit and the external storage device of the terminal device 500. The memory 520 is used to store the computer program 521 and other programs and data required by the terminal device 500. The memory 520 may also be used to temporarily store data that has been output or is to be output.

[0164] The embodiments of the present application also disclose a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the automatic welding method of the tunnel steel grid as described in the foregoing various embodiments.

[0165] The embodiments of the present application also disclose a computer-readable storage medium storing a computer program, and when the computer program is executed by a processor, it implements the automatic welding method of the tunnel steel grid as described in the foregoing various embodiments.

[0166] The embodiment of the present application also discloses a computer program product. When the computer program product runs on a computer, it causes the computer to execute the automatic welding method of the tunnel steel grid described in each of the foregoing embodiments.

[0167] The foregoing embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of each embodiment of the present application, and should all be included in the protection scope of the present application.

Claims

1. A tunnel steel grating automatic welding method, characterized in that: include: Before starting the welding robot to weld the tunnel steel grating, grayscale image data of the tunnel steel grating is collected; Performing a first target detection operation on the grayscale image data to obtain a first welding point on the main steel bar in the tunnel steel grid waiting for welding; When the welding robot is started to weld the tunnel steel grating according to the first welding point, original infrared image data of the tunnel steel grating is collected; The original infrared image data is adjusted according to the current of the welding robot to obtain target infrared image data; Converting the target infrared image data into optical flow field image data; Under the reference of the grayscale image data, performing a second target detection operation on the optical flow field image data to obtain a second welding point generated when the welding robot welds on the tunnel steel grid; The welding robot is controlled to continue welding the tunnel steel grille according to the first welding point and the second welding point.

2. The method according to claim 1, characterized in that The performing a first target detection operation on the grayscale image data to obtain a first welding point on the main steel bar in the tunnel steel grid waiting for welding comprises: Loading a first target detection network; the first target detection network includes a first backbone network, a first neck network and a first head network; Inputting the grayscale image data into the first backbone network to extract first grayscale image features; Inputting the first grayscale image feature into the first neck network to convert it into a second grayscale image feature; The second grayscale image feature is input into the first head network to detect the first welding point on the main steel bar in the tunnel steel grid waiting for welding.

3. The method according to claim 2, characterized in that The step of performing a second target detection operation on the optical flow field image data with reference to the grayscale image data to obtain a second welding point generated when the welding robot welds on the tunnel steel grid comprises: Loading a second target detection network; the second target detection network includes a second backbone network, a fusion module, a second neck network and a second head network; Inputting the optical flow field image data into the first backbone network to extract first optical flow field image features; Inputting the first grayscale image feature and the first optical flow field image feature into the fusion module and merging them into a second optical flow field image feature; Inputting the second optical flow field image feature into the second neck network to convert it into a third optical flow field image feature; The third optical flow field image feature is input into the second head network to detect the second welding point generated by the welding robot when welding on the tunnel steel grid.

4. The method according to claim 3, characterized in that The first backbone network outputs the first grayscale image features in the first stage, the second stage and the third stage in sequence, and the second backbone network outputs the first optical flow field image features in the first stage, the second stage and the third stage in sequence; the fusion module includes a first attention layer, a second attention layer and a third attention layer; The step of inputting the first grayscale image feature and the first optical flow field image feature into the fusion module and merging them into a second optical flow field image feature includes: Inputting the first grayscale image feature of the first stage and the first optical flow field image feature of the first stage into the first attention layer to fuse them into a first fused image feature; Inputting the first grayscale image feature of the second stage and the first optical flow field image feature of the second stage into the second attention layer to fuse them into a second fused image feature; Inputting the first grayscale image feature of the third stage and the first optical flow field image feature of the third stage into the third attention layer to fuse them into a third fused image feature; Upsampling the third fused image feature to obtain a first intermediate image feature; splicing the first intermediate image feature and the second fused image feature into a second intermediate image feature; Upsampling the second intermediate image feature to obtain a third intermediate image feature; splicing the third intermediate image feature and the first fused image feature into a fourth intermediate image feature; Down-sampling the fourth intermediate image feature to obtain a second optical flow field image feature of the first stage; splicing the second optical flow field image feature and the third intermediate image feature of the first stage into a fifth intermediate image feature; Downsampling the fifth intermediate image feature to obtain a second optical flow field image feature of the second stage; splicing the second optical flow field image feature of the second stage and the first intermediate image feature into a sixth intermediate image feature; The sixth intermediate image feature is downsampled to obtain the second optical flow field image feature of the third stage.

5. The method according to claim 3, characterized in that: The structure of the first backbone network is the same as that of the second backbone network, the structure of the first neck network is the same as that of the second neck network, and the structure of the first head network is the same as that of the second head network.

6. The method according to claim 1, characterized in that The adjusting the original infrared image data according to the current of the welding robot to obtain target infrared image data includes: Sorting the infrared values ​​of each pixel in the original infrared image data to obtain an infrared sequence; Taking a specified quantile in the infrared sequence as a threshold; In the process of traversing each pixel point in the original infrared image data, adding a window with the current pixel point as the center; Calculating the average value of infrared values ​​for the pixels in the window to obtain a filtered value; If the filtering value is less than or equal to the threshold, the infrared value of the current pixel is maintained unchanged; If the filtering value is greater than the threshold, the infrared value of the current pixel point is attenuated according to the current of the welding robot; If all pixel points in the original infrared image data are traversed, each pixel point is output to obtain the target infrared image data.

7. The method according to claim 6, characterized in that The attenuating the infrared value of the current pixel point according to the current of the welding robot includes: Mapping the current of the welding robot to a reference infrared value; the reference infrared value is positively correlated with the current of the welding robot; The current of the welding robot, the reference infrared value and the infrared value of the current pixel are input into the following formula for calculation to attenuate the infrared value of the current pixel: Among them, IR new is the infrared value of the current pixel after attenuation, IR old is the infrared value of the current pixel before attenuation, G is the preset gain coefficient, IR base is the reference infrared value, and γ is the preset attenuation coefficient.

8. The method according to any one of claims 1 to 7, characterized in that The controlling the welding robot to continue welding the tunnel steel grille according to the first welding point and the second welding point includes: Generating a welding range with the first welding point as the center; If the second welding point is within the welding range, continue to observe the welding robot welding the tunnel steel grille; If the second welding point is outside the welding range, calculating a vector from the second welding point to the first welding point; The welding robot moves according to the vector control observation, and the tunnel steel grating is welded.

9. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the automatic welding method for tunnel steel grating as described in any one of claims 1 to 8 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the automatic welding method for tunnel steel grating as described in any one of claims 1 to 8 is implemented.

Images