Method and device for repairing cracks of steel box girder

Abstract

The invention provides a steel box girder crack repairing method and device, and relates to the technical field of robots, and the method comprises the following steps: collecting an image based on a camera component on the steel box girder crack repairing device; obtaining a to-be-repaired position prediction result; determining a point cloud coordinate of the position to be repaired; determining a relative pose between the camera coordinate system and the ground coordinate system; coordinates of a to-be-repaired position are obtained; and determining a target path of the repair operation. According to the method, the depth camera and the RGB camera are used for collecting images on the steel box girder, and the crack position, namely the position to be repaired, is predicted and the point cloud coordinates are determined in combination with the crack position prediction model; obtaining a relative pose of a camera coordinate system and a ground coordinate system through the I MU data, and further converting a point cloud coordinate of a to-be-repaired position into a position in the ground coordinate system; and finally, a repairing path is determined according to the coordinates of the to-be-repaired position, the mechanical arm is controlled to conduct steel box girder crack repairing operation, and therefore accurate repairing operation is achieved, and the repairing efficiency and accuracy are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of robots, and more particularly, to a method and device for repairing steel box girder cracks. Background Art

[0002] Currently, the repair of steel box girder cracks mainly relies on manual operation, but there are many drawbacks to manual repair of steel box girder cracks: on the one hand, manual repair requires maintenance personnel to work in a closed and complex environment, which not only has great operation difficulty for maintenance personnel, but also has a high safety risk; on the other hand, manual repair is inefficient, and maintenance personnel need to spend a lot of time on crack positioning, measurement, and operation of repair tools. Moreover, for some cracks with complex shapes or cracks distributed on the surface of a large area of steel box girders, it is difficult to ensure the consistency of repair quality by manual repair.

[0003] In addition, the accuracy of manual repair is also limited by the skill level and work experience of maintenance personnel. Different maintenance personnel may have differences in the operation methods and standards for crack repair, which may lead to unsatisfactory repair effects and affect the structural performance and service life of steel box girders. Therefore, it is of great practical significance to develop a device and method that can repair steel box girder cracks efficiently, accurately, and safely. Summary of the Invention

[0004] The purpose of the present invention is to provide a method and device for repairing steel box girder cracks to improve the above problems. To achieve the above purpose, the technical solutions adopted by the present invention are as follows:

[0005] In the first aspect, the present application provides a method for repairing steel box girder cracks, including:

[0006] Collecting images based on the camera components on the steel box girder crack repair device, respectively collecting a plurality of first images and a plurality of second images. The camera components include a depth camera and an RGB camera. The first images are collected by the RGB camera, and the second images are collected by the depth camera;

[0007] Inputting the plurality of first images into a trained crack position prediction model for crack position prediction to obtain a predicted result of the position to be repaired. The crack position prediction model is a model constructed based on the YOLO algorithm;

[0008] Determining the first images and second images corresponding to the positions based on the predicted result of the position to be repaired, and determining the point cloud coordinates of the position to be repaired through the first images and the second images. The point cloud coordinates are coordinates in the camera coordinate system;

[0009] Determining the relative pose between the camera coordinate system and the ground coordinate system based on the IMU data. The relative pose is used to represent the spatial relationship between the two coordinate systems;

[0010] Convert the point cloud coordinates of the position to be repaired into the ground coordinate system based on the relative pose to obtain the coordinates of the position to be repaired;

[0011] Determine the target path of the repair operation based on the coordinates of the position to be repaired, and control the steel box girder repair robotic arm of the steel box girder crack repair device to move along the path and perform crack repair based on the target path.

[0012] In a second aspect, the present application also provides a steel box girder crack repair device, including:

[0013] A steel box girder repair robotic arm, which is composed of an upper arm, a forearm and a rotating joint, and one end of the upper arm is connected to the forearm through the rotating joint;

[0014] A robotic arm magnetic adsorption base, which is arranged at one end of the upper arm far from the rotating joint, and the robotic arm magnetic adsorption base is magnetically adsorbed and fixed on the steel box girder;

[0015] A terminal repair mechanism, which is fixedly connected to one end of the forearm far from the rotating joint through a flexible connector;

[0016] A phase mechanism component, which is an integrated component including a depth camera and an RGB camera, and the phase mechanism component is fixed on a bracket at one end of the forearm close to the terminal repair mechanism.

[0017] The beneficial effects of the present invention are as follows:

[0018] The present invention uses a depth camera and an RGB camera to collect images on the steel box girder, combines a trained crack position prediction model based on the YOLO algorithm to predict the crack position, that is, the position to be repaired, and determine the point cloud coordinates; obtains the relative pose between the camera coordinate system and the ground coordinate system through IMU data, and then converts the point cloud coordinates of the position to be repaired into the position in the ground coordinate system; finally, determines the repair path according to the coordinates of the position to be repaired, and controls the robotic arm to perform the steel box girder crack repair operation, thereby realizing precise repair operations and improving the efficiency and accuracy of repair.

[0019] Other features and advantages of the present invention will be described in the subsequent specification, and some will become obvious from the specification or be understood by implementing the embodiments of the present invention. Description of the Drawings

[0020] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as limiting the scope. For those of ordinary skill in the art, without creative efforts, other related drawings can also be obtained based on these drawings.

[0021] Figure 1 Schematic flow diagram of the steel box girder crack repair method described in the embodiments of the present invention;

[0022] Figure 2 Schematic diagram of the vision calibration system described in the embodiments of the present invention;

[0023] Figure 3 Schematic diagram of the improved CS_YOLO model structure described in the embodiments of the present invention;

[0024] Figure 4 Schematic diagram of the channel shuffle operation described in the embodiments of the present invention;

[0025] Figure 5 Schematic diagram of the overall process of the steel box girder crack repair method described in the embodiments of the present invention;

[0026] Figure 6 Steel box girder crack repair device described in the embodiments of the present invention;

[0027] Figure 7 Schematic diagram of the flexible connector structure described in the embodiments of the present invention.

[0028] Markings in the figure: 1, control host; 2, steel box girder repair robotic arm; 3, camera component; 4, magnetic base of the end repair mechanism; 5, end repair mechanism; 6, flexible connector; 7, steel box girder; 8, magnetic base of the robotic arm; 21, upper arm; 22, rotary joint; 23, forearm; 61, steel box girder; 62, fixed platform; 63, moving platform; 611, first connecting rod; 612, spring; 613, second connecting rod; 614, spherical pair; 615, sliding pair; 616, rotating pair. Detailed implementation manners

[0029] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Apparently, the described embodiments are some, but not all, of the embodiments of the present invention. Components of the embodiments of the present invention described and illustrated herein generally can be arranged and designed in a variety of different configurations. Therefore, the detailed description of the embodiments of the present invention provided herein is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts fall within the scope of protection of the present invention.

[0030] It should be noted that like reference numerals and letters denote like items in the following drawings. Therefore, once an item is defined in one drawing, it does not require further definition and explanation in subsequent drawings. At the same time, in the description of the present invention, terms such as "first" and "second" are only used for descriptive distinction and cannot be construed as indicating or implying relative importance.

[0031] Embodiment 1:

[0032] This embodiment provides a method for repairing cracks in a steel box girder.

[0033] See Figure 1 , which shows that this method includes Step S10, Step S20, Step S30, Step S40, Step S50, and Step S60.

[0034] Step S10. Based on the camera member 3 on the steel box girder crack repair device, images are collected, and a plurality of first images and a plurality of second images are respectively collected. The camera member 3 includes a depth camera and an RGB camera. The first images are collected by the RGB camera, and the second images are collected by the depth camera;

[0035] Specifically, the RGB camera and the depth camera provided on the steel box girder crack repair device are used to obtain the surface image information of the steel box girder from different dimensions. The RGB camera provides a clear visual image of the crack, while the depth camera can capture the depth information of the crack to help more comprehensively evaluate the situation of the crack.

[0036] Considering that the working environment and the initial position of the steel box girder crack repair device will change, it is necessary to position the steel box girder crack repair device in the camera coordinate system, that is, the coordinate system corresponding to the RGB camera. Through calibration, the position and attitude, that is, the pose, of the device relative to the RGB camera are obtained, so as to obtain the three-dimensional coordinates of the device in the camera coordinate system. This can ensure that during subsequent repair operations, the system can accurately know the position of the device in the actual scene and avoid repair errors caused by inaccurate positions.

[0037] Specifically, the calibration steps of the steel box girder crack repair device include: Step S1, Step S2, Step S3, Step S4, and Step S5:

[0038] Step S1. Based on the laser device and RGB camera fixedly connected to the steel box girder crack repair device, images are collected to obtain a laser image, and the laser image is an image of the laser emitted by the laser device projected onto the Apriltag plane;

[0039] Specifically, first, a vision calibration system including an Apriltag label, a laser rangefinder, and a steel box girder crack repair device is established, as Figure 2 shown. The laser rangefinder includes a laser device that emits laser and a laser distance camera for collection. The laser rangefinder is fixedly connected to the steel box girder crack repair device. The Apriltag moves along the z-axis. At each position on the z-axis, the Apriltag is rotated around the x-axis and y-axis to ensure that the laser point falls within the Apriltag plane. The distance from the Apriltag plane to the laser rangefinder is recorded, and the Apriltag plane image is collected by the RGB camera as the laser image.

[0040] Step S2. Based on the laser image and a preset calibration method, determine the relative pose between the Apriltag coordinate system and the camera coordinate system;

[0041] Specifically, obtaining the extrinsic parameters of the camera for the Apriltag mainly involves establishing the relative pose relationship between the Apriltag coordinate system and the camera coordinate system, that is, the rotation matrix and translation matrix between the two coordinate systems.

[0042] Assume that the coordinates of the Apriltag in the world coordinate system are (X w , Y w , Z w ). Its coordinates in the pixel coordinate system (u, v) can be obtained through the coordinate transformation relationship described by formula (1):

[0043]

[0044] where Z c is the depth value of the pixel point with coordinates (u, v); M r is the intrinsic parameter matrix of the RGB camera; E is the extrinsic parameter matrix of the RGB camera, is the rotation matrix between the world coordinate system and the camera coordinate system, is the translation matrix between the world coordinate system and the camera coordinate system,

[0045] Among them, u0 r is the abscissa of the optical center; v0 r is the ordinate of the optical center; fx r is the focal length of the RGB camera's horizontal axis; fy r is the focal length of the RGB camera's vertical axis.

[0046] Based on the coordinates of the four corner points in the known Apriltag coordinate system and the corresponding pixel coordinates in the RGB image, the homography matrix H can be calculated using the direct linear transformation DLT to map the projective transformation relationship between the Apriltag coordinate system and the camera coordinate system. The homography matrix H is expressed as the product of the intrinsic matrix and the extrinsic matrix of the RGB camera. Since the z-axis coordinate of each point on the Apriltag plane is 0 in the Apriltag coordinate system, the third column of the rotation matrix is removed, and the homography matrix H can be expressed as formula (2):

[0047] H = sM r E’ (2)

[0048] where H is the homography matrix, s is an unknown scale factor; E’ is the extrinsic matrix of the RGB camera after removal processing, M r is the intrinsic matrix of the RGB camera,

[0049] Expanding formula (2) gives formula (3):

[0050]

[0051] where H 11 、H 12 、H 13 、H 21 、H 22 、H 31 、H 32 、H 33 are the elements in the homography matrix; s is an unknown scale factor; R 11 、R 12 、R 21 、R 22 、R 31 、R 32 are the parameters in the rotation matrix; T1, T2, T3 are the parameters in the translation matrix; u0 r is the abscissa of the optical center; v0 r is the ordinate of the optical center; fx r is the focal length of the RGB camera's horizontal axis; fy r is the focal length of the RGB camera's vertical axis.

[0052] Since the rotation matrix is an orthonormal matrix and the norm of the column vectors in the matrix is 1, the magnitude of the unknown scale factor s is obtained by calculating the geometric mean of the norms of two column vectors in the rotation matrix, as shown in Equation (4).

[0053]

[0054] Where a is the magnitude of the first column of the rotation matrix; b is the magnitude of the second column of the rotation matrix; s is the unknown scale factor; R 11 、R 12 、R 21 、R 22 、R 31 、R 32 are the parameters in the rotation matrix.

[0055] Combining the property that each column of the rotation matrix is an identity matrix, the values of the elements in the first two columns are obtained, and then the cross product operation is performed on the elements in the first two columns using the property of the orthogonal matrix to obtain the magnitude of the elements in the third column of the rotation matrix. However, the rotation matrix obtained by DLT does not necessarily have the orthonormal property, so the rotation matrix needs to be decomposed by SVD, and finally the transformation matrix from the Apriltag coordinate system to the camera coordinate system, that is, the relative pose, is obtained.

[0056] Step S3. Based on the relative pose between the Apriltag coordinate system and the camera coordinate system, calculate the plane parameters of the Apriltag plane in the camera coordinate system;

[0057] Specifically, Step S3 includes Step S301, Step S302, and Step S303:

[0058] Step S301. Calculate the product of the rotation matrix and the normal vector of the Apriltag plane in the Apriltag coordinate system to obtain the normal vector in the camera coordinate system;

[0059] Step S302. Calculate the inner product of the transpose of the normal vector in the camera coordinate system and the translation matrix, and take the opposite number to obtain the constant term in the camera coordinate system;

[0060] Step S303. Based on the normal vector and the constant term in the camera coordinate system, form the plane parameters;

[0061] Specifically, the coordinates of the Apriltag plane in the camera coordinate system are represented by (n c ,d c ). Assuming that nc is the normal vector of the Apriltag plane in the camera coordinate system and n w is the normal vector of the Apriltag plane in the world coordinate system, according to n w =[0,0,1] T , nc is expressed as formula (5):

[0062]

[0063] where n c is the normal vector of the Apriltag plane in the camera coordinate system; n w is the normal vector of the Apriltag plane in the world coordinate system; is the rotation matrix between the world coordinate system and the camera coordinate system.

[0064] The plane equation of the Apriltag plane in the camera coordinate system can be expressed as formula (6):

[0065] n c T P c + d c = 0 (6)

[0066] where P c is the coordinate of the origin of the Apriltag plane in the camera coordinate system, is the translation matrix between the world coordinate system and the camera coordinate system; n c T is the transpose of the normal vector of the Apriltag plane in the camera coordinate system; d c is the constant term of the Apriltag plane in the camera coordinate system.

[0067] where Based on formulas (5) and (6), the following formulas (7) and (8) can be obtained:

[0068]

[0069] where, is the rotation matrix; n w is the normal vector of the Apriltag plane in the world coordinate system; is the translation matrix between the world coordinate system and the camera coordinate system; d c is the constant term of the Apriltag plane in the camera coordinate system.

[0070] The results calculated by formulas (5) and (8) are the plane parameters (n c , d c ) of the Apriltag plane in the camera coordinate system.

[0071] Step S4. Calculate the relative pose of the laser coordinate system and the camera coordinate system based on the preset constraints of the laser points on the laser image;

[0072] Step S5. Determine the three-dimensional coordinates of the steel box girder crack repair device in the camera coordinate system based on the relative pose between the Apriltag coordinate system and the camera coordinate system, the plane parameters, and the relative pose between the laser coordinate system and the camera coordinate system;

[0073] Specifically, the laser rangefinder is connected to the steel box girder crack repair device. Let the rotation and translation matrices for transforming the laser coordinate system to the camera coordinate system be and The laser point P emitted by the laser rangefinder L =[0, 0, l] T , where the coordinate representation of the laser point in the camera coordinate system is where R L c is the rotation matrix between the laser coordinate system and the camera coordinate system, and the translation matrix between the laser coordinate system and the camera coordinate system.

[0074] According to the fact that the laser point is on the Apriltag calibration plane, it can be known that formula (9) is satisfied, that is:

[0075]

[0076] where is the rotation matrix between the laser coordinate system and the camera coordinate system; the translation matrix between the laser coordinate system and the camera coordinate system; P L is the laser point emitted by the laser rangefinder; n c T is the transpose of the normal vector of the Apriltag plane in the camera coordinate system; d c is the constant term of the Apriltag plane in the camera coordinate system.

[0077] where n c T =[n 11 n 21 n 31 , Substituting the above formula (9), formula (10) can be obtained:

[0078]

[0079] Transform formula (10) into a system of linear equations AX = b, where A = [n 11 l, n 21 l, n 31 l, n 11 , n 21 , n 31 , X = [R 13 , R 23,R 33 ,t 11 ,t 21 ,t 31 T , b = [-d c , when there are m laser points, the A matrix is an m×6 coefficient matrix, X is a 6×1 vector, and b is an m×1 vector. Since is an orthogonal matrix and satisfies the constraint that the modulus of the vector is 1, X satisfies the constraint condition of formula (11), that is:

[0080] R 13 2 +R 23 2 +R 33 2 = 1 (11)

[0081] where, R 13 、R 23 and R 33 are the parameters in the rotation matrix between the laser coordinate system and the camera coordinate system.

[0082] Solving the rotation matrix and translation matrix for transforming the laser coordinate system to the camera coordinate system can be transformed into solving a system of multivariate nonlinear equations. Since the analytical solution of the system of multivariate nonlinear equations cannot be obtained, the problem of solving the rotation and translation matrices can be transformed into a nonlinear optimization problem of multivariate variables. The optimization function is established as formula (12):

[0083]

[0084] where, H(x) is the optimization function; A = [n 11 l, n 21 l, n 31 l, n 11 , n 21 , n 31 , l is the depth coordinate of the laser point, n 11 、n 21 、n 31 are the parameters in the transposed matrix of the normal vector of the Apriltag plane in the camera coordinate system; X = [R 13 , R 23 , R 33 , t 11 , t 21 , t 31 T , R 13 、R 23 and R 33 are the parameters in the rotation matrix between the laser coordinate system and the camera coordinate system, t 11 、t 21 ​​, t 31 The parameters in the translation matrix between the laser coordinate system and the camera coordinate system; b = [-d c , d c is the constant term of the Apriltag plane in the camera coordinate system.

[0085] First, solve the optimization function without non-linear constraints, and use the obtained X as the initial value of the non-linear optimization problem. Then, use the method of trigonometric substitution to transform the non-linear optimization problem with constraints into a non-linear optimization problem without constraints. The substitution is shown in Equation (13):

[0086] R 13 = sinα cosβ, R 23 = sinα sinβ, R 33 = cosα (13)

[0087] where, R 13 , R 23 and R 33 are the parameters in the rotation matrix between the laser coordinate system and the camera coordinate system; α, β are substitution variables used to replace the original variables to simplify the optimization function.

[0088] The optimization function established based on Equation (13) is shown in Equation (14):

[0089]

[0090] where, l is the depth coordinate of the laser point; n 11 , n 21 , n 31 are the parameters in the transposed matrix of the normal vector of the Apriltag plane in the camera coordinate system; α, β are substitution variables; t 11 , t 21 , t 31 are the parameters in the translation matrix between the laser coordinate system and the camera coordinate system; f(x) is the fitting error; H(x) is the optimization function.

[0091] f(x) is used to measure the deviation degree of any laser point under the transformation relationship between the laser coordinate system and the camera coordinate system. At this time, the variable to be optimized is x = (α, β, t 11 , t 21 , t 31 ).

[0092] The first-order partial derivatives of the optimization function are as follows:

[0093]

[0094] where, l is the depth coordinate of the laser point; n 11 , n21 , n 31 is the parameter in the transposed matrix of the normal vector of the Apriltag plane in the camera coordinate system; α, β are substitution variables; t 11 , t 21 , t 31 are the parameters in the translation matrix between the laser coordinate system and the camera coordinate system; f(x) is the fitting error; H(x) is the optimization function.

[0095] The quasi - Newton method is used for iterative optimization and solution. The algorithm starts from the initial point, calculates the gradient of the objective function, and determines the search direction according to the gradient and the approximate Hessian matrix; then selects the step size through the line search method, updates the solution and updates the approximation of the Hessian matrix; the iterative process continues until the gradient satisfies the stopping condition, and finally outputs the optimal solution. The quasi - Newton method is an unconstrained optimization algorithm that avoids directly calculating the second - order derivative by approximating the Hessian matrix of the objective function, that is, the second - order derivative matrix, thus improving the calculation efficiency.

[0096] Since the repair mechanism is fixedly connected to the laser rangefinder, the coordinates of the repair mechanism in the laser coordinate system are defined as the origin (0, 0, 0) of the laser coordinate system. According to the coordinate transformation relationship between the laser coordinate system and the camera coordinate system the three - dimensional coordinates of the steel box girder crack repair device in the camera coordinate system are calculated.

[0097] Step S20. Input multiple first - images into the trained crack - position prediction model for crack - position prediction to obtain the predicted result of the position to be repaired. The crack - position prediction model is a model constructed based on the YOLO algorithm;

[0098] Specifically, step S20 specifically includes steps S21, S22, S23, S24, S25, S26, and S27:

[0099] Step S21. Construct an initial model based on the YOLO algorithm. The initial model consists of a feature extraction module, a feature fusion module, and a multi - scale detection module.

[0100] Specifically, step S21 specifically includes steps S211, S212, S213, and S214:

[0101] Step S211. Construct an extraction unit based on sequentially stacking convolutional modules and C3 feature extraction modules;

[0102] Step S212. Combine the output of the extraction unit with the SPPF feature pooling module to form a feature extraction module;

[0103] Step S213. A feature fusion module is constructed based on a sequentially stacked CS convolution module, an upsampling module, a feature splicing module, and a C3 feature extraction module;

[0104] Step S214. An initial model is constructed based on a feature extraction module, a feature fusion module, and a multi-scale detection module;

[0105] Specifically, an improved CS_YOLO model is proposed in this application to construct a crack location prediction model. The model inputs the collected image and outputs a prediction box for the position to be repaired, where the pixel points included in the prediction box for the position to be repaired are the feature points of the predicted position to be repaired.

[0106] As Figure 3 shown, the improved CS_YOLO model includes three parts: Backbone, i.e., the feature extraction module, Neck, i.e., the feature fusion module, and Head, i.e., the multi-scale detection module.

[0107] Backbone, i.e., the feature extraction module, mainly includes a Conv module, a C3 module, and an SPPF module. The C3 module is used to add a residual structure, which can increase the gradient value of backpropagation between layers, avoid the vanishing gradient caused by the deepening of the network, and thus can extract finer-grained features without worrying about network degradation; the SPPF structure greatly increases the receptive field of the feature map and separates the most important context features, but has little impact on the running speed of the model; the Conv module is a key component for extracting local features from the input image. Backbone, i.e., the feature extraction module, gradually transmits spatial information to the channels, and each compression of the spatial width and height and expansion of the channels of the feature map will cause partial loss of semantic information.

[0108] Neck, i.e., the feature splicing module, scales and fuses feature maps of different scales to enhance the network's detection ability for targets to be repaired at different scales. Neck, i.e., the feature splicing module, mainly includes a Concat module, a C3 module, a CSConv module, and an Upsample module. When the feature reaches Neck, the channel dimension reaches the maximum and the width and height dimensions reach the minimum. In order to interact the position information of different channels, in Neck, the CSConv module is involved. The CSConv module performs a channel shuffle operation on the features after fusing different convolution operations. As Figure 4 shown, first, the input feature map is divided into two groups according to the number of channels. At this time, the dimension of the channels is Then these two groups of feature maps are transposed to The channel order of the transposed feature map has been shuffled, and then the shuffled feature map is re-divided into 2 groups. The new 2 groups of features are concatenated and fused along the channel direction. By shuffling the feature map along the channel direction, re-grouping, and then concatenating, information interaction between different channels is achieved. Through the introduction of the CSConv module, the model not only realizes the fusion of feature map information of different layers in the Neck, i.e., the feature concatenation module, but also realizes information interaction in the feature map with deeper channels in the same layer.

[0109] And the Neck adopts the combination of FPN + PAN. The FPN (Feature Pyramid Network) structure is from top to bottom, that is, after upsampling the deep features, they are fused with the shallow features again to enhance the propagation of deep semantic features to the shallow layer. The PAN (Path Aggregation Network) structure is from bottom to top, that is, after downsampling the shallow features, they are fused with the deep features again to enhance the upward propagation of shallow target location information.

[0110] The Head, i.e., the multi-scale detection module, is used to detect the targets to be repaired at 3 scales. For example, when the size of the input image is 640×640×3, the sizes of the output feature maps are 20×20×18, 40×40×18, and 80×80×18 respectively, which are responsible for detecting large-scale, medium-scale, and small-scale targets. Each grid of the feature map predicts 3 bboxes (bounding boxes). Each bbox contains the class confidence, the height and width of the prediction box of the target to be repaired, the coordinates of the center point, and the confidence of whether there is a position to be repaired within the prediction box, a total of 3×1 + 4 + 1 kinds of information.

[0111] Step S22. Obtain the steel box images in the normal state and the cracked state. The steel box image in the cracked state carries the crack position annotation box.

[0112] Step S23. Input the steel box image into the initial model for prediction to obtain the prediction result, which is the prediction box marking the position to be repaired on the steel box image.

[0113] Step S24. Calculate the loss function based on the prediction result and the crack position annotation box to obtain the first loss function.

[0114] Step S25. Calculate the loss function based on the prediction result and the closest crack position annotation box to obtain the second loss function.

[0115] Step S26. Construct the target loss function based on the first loss function and the second loss function.

[0116] Step S27. Adjust the parameters of the initial model until the target loss function is less than the set threshold. The corresponding initial model is the crack position prediction model.

[0117] Specifically, the calculation formula of the loss function is as follows:

[0118] L = L Attr + L Rej

[0119] where L is the loss function, L Attr is the first loss function; L Rej is the second loss function.

[0120] Among them, the first loss function L Attr is the original loss function of the YOLO network, representing the loss value generated by the predicted bounding box at the position to be repaired and the labeled bounding box at the position to be repaired. When the predicted bounding box at the position to be repaired is closer to the labeled bounding box at the position to be repaired, the L Attr loss value will be smaller, and L Attr will make the predicted bounding box at the position to be repaired and the corresponding labeled bounding box at the position to be repaired as close as possible. The second loss function L Rej is the loss value generated by the predicted bounding box at the position to be repaired and the other labeled bounding boxes around it, that is, except for the corresponding labeled bounding box at the position to be repaired, the labeled bounding box with the largest IoU (Intersection over Union) with the predicted bounding box at the position to be repaired. The closer the predicted bounding box at the position to be repaired is to the other labeled bounding boxes around it, the L Rej larger the loss value, and L Rej makes the predicted bounding box at the position to be repaired as far away as possible from the other labeled bounding boxes around it.

[0121] The calculation of L Rej is as follows:

[0122]

[0123] where L Rej is the second loss function; B P is the predicted bounding box at the position to be repaired for positive samples; is the labeled bounding box with the largest IoU with the predicted bounding box at the position to be repaired except for the corresponding labeled bounding box at the position to be repaired, P + is the set of all positive samples, P ∈ P + ; area() is the area function; T overlap is the preset overlap threshold; Smooth ln () is the smoothing function.

[0124] Among them, positive samples are samples whose overlapping area between the corresponding predicted bounding box at the position to be repaired and the other labeled bounding boxes around it exceeds the threshold T overlap , and the L Rej loss needs to be calculated; when the overlapping area does not exceed the threshold Toverlap , then there is no need to calculate L Rej loss.

[0125] Step S30. Determine the first image and the second image corresponding to the position based on the prediction result of the position to be repaired, and determine the point cloud coordinates of the position to be repaired through the first image and the second image. The point cloud coordinates are the coordinates in the camera coordinate system;

[0126] Specifically, determine the corresponding image containing the crack of the steel box girder from multiple first images through the above crack position prediction model, and only perform subsequent processing on the corresponding images with cracks.

[0127] Step S30 specifically includes Step S31, Step S32, Step S33 and Step S34:

[0128] Step S31. Combine the pixel coordinates and depth values of each pixel on the second image based on the internal parameters of the depth camera, and convert them into three-dimensional coordinates in the depth camera coordinate system;

[0129] Step S32. Based on the external parameters of the depth camera and the RGB camera, convert the three-dimensional coordinates in the depth camera coordinate system into three-dimensional coordinates in the camera coordinate system;

[0130] Step S33. Based on the internal parameters of the RGB camera, convert the three-dimensional coordinates in the camera coordinate system into two-dimensional pixel coordinates in the camera coordinate system;

[0131] Step S34. Align and fuse the depth value of each pixel on the second image with the corresponding two-dimensional pixel coordinates to obtain the point cloud coordinates of the position to be repaired;

[0132] Specifically, before fusing the first image collected by the RGB camera and the second image collected by the depth camera, first align the first image and the second image. The steps are as follows:

[0133] (1. Based on the internal parameters of the depth camera, convert the depth pixel coordinate system to the depth camera coordinate system:

[0134] Let the pixel point of the depth pixel coordinate system be (u d , v d ), the internal parameter of the depth camera is M d , and the value of the pixel point (u d , v d ) is the vertical distance from the spatial point corresponding to the pixel point to the depth camera, that is, the z coordinate Z of the spatial point in the depth camera coordinate system dc , and according to formula (15), the three-dimensional coordinates (X d , Y d ) of the pixel coordinates (u dc , vdc , Z dc ):

[0135]

[0136] Among them, (X dc , Y dc , Z dc ) is the three-dimensional coordinate of the pixel coordinate in the depth camera coordinate system; M d is the internal parameter of the depth camera; (u d , v d ) is the pixel coordinate.

[0137] (2. Based on the external parameters of the depth camera and the RGB camera, convert the depth camera coordinate system to the RGB camera coordinate system:

[0138] Let the external parameters of the depth camera and the RGB camera be Among them, is the rotation matrix from the depth camera coordinate system to the camera coordinate system; is the translation matrix from the depth camera coordinate system to the camera coordinate system. Through the coordinate transformation relationship of formula (16), obtain its coordinate (X rc , Y rc , Z rc ) in the RGB camera coordinate system:

[0139]

[0140] Among them, (X rc , Y rc , Z rc ) is the three-dimensional coordinate of the pixel coordinate in the camera coordinate system; is the transformation matrix from the depth camera coordinate system to the camera coordinate system; (X dc , Y dc , Z dc ) is the three-dimensional coordinate of the pixel coordinate in the depth camera coordinate system.

[0141] (3. Based on the internal parameter of the RGB camera, convert the camera coordinate system to the RGB pixel coordinate system:

[0142] Let the internal parameter of the RGB camera be M r , and the pixel point of the RGB pixel coordinate system be (u r , v r ). Based on formula (17), convert the coordinate (X rc , Y rc , Z rc ) in the RGB camera coordinate system to the pixel coordinate (u r , v r ) of the RGB pixel coordinate system:

[0143]

[0144] Among them, Z rc is the depth value in the three-dimensional coordinates of the pixel coordinates in the depth camera coordinate system; (u r , v r ) is the two-dimensional pixel coordinate in the pixel coordinate system; (X rc , Y rc , Z rc ) is the three-dimensional coordinate of the pixel coordinate in the camera coordinate system; M r is the internal parameter of the RGB camera.

[0145] According to the above steps, it can be known that the depth value corresponding to the pixel coordinate (u r , v r ) in the first image collected by the RGB camera is the pixel value Z d , v d ) coordinate in the depth image. dc .

[0146] Step S34 specifically includes steps S341, S342, S343, S344, S345 and step S346:

[0147] Step S341. Calculate the ratio of the depth value in the second image to the preset conversion ratio to obtain the depth coordinate value of the point cloud coordinate;

[0148] Step S342. Calculate the difference between the abscissa in the two-dimensional pixel coordinate and the abscissa of the optical center, and calculate the product of the difference and the depth coordinate value to obtain the first product;

[0149] Step S343. Calculate the ratio of the first product to the horizontal axis focal length of the RGB camera to obtain the abscissa value of the point cloud coordinate;

[0150] Step S344. Calculate the difference between the ordinate in the two-dimensional pixel coordinate and the ordinate of the optical center, and calculate the product of the difference and the depth coordinate value to obtain the second product;

[0151] Step S345. Calculate the ratio of the second product to the vertical axis focal length of the RGB camera to obtain the ordinate value of the point cloud coordinate;

[0152] Step S346. Based on the abscissa value, ordinate value and depth coordinate value, form the point cloud coordinate;

[0153] Specifically, the internal parameters of the RGB camera include the horizontal axis focal length of the RGB camera, the vertical axis focal length of the RGB camera, the abscissa of the optical center and the ordinate of the optical center. Therefore, the calculation of the point cloud coordinate at the position to be repaired is shown in formula (18):

[0154]

[0155] Among them, x sc is the abscissa value in the camera coordinate system; y sc is the ordinate value in the camera coordinate system; z sc is the depth coordinate value in the image polar coordinate system; (x ij , y ij , d ij ) is the coordinate value of the i-th row and j-th column; u0 r is the abscissa of the optical center; v0 r is the ordinate of the optical center; fx r is the horizontal axis focal length of the RGB camera; fy r is the vertical axis focal length of the RGB camera; r is the conversion ratio.

[0156] Step S40. Determine the relative pose between the camera coordinate system and the ground coordinate system based on the IMU data, and the relative pose is used to characterize the spatial relationship between the two coordinate systems;

[0157] Step S50. Convert the point cloud coordinates of the position to be repaired into the ground coordinate system based on the relative pose to obtain the coordinates of the position to be repaired;

[0158] Specifically, the relative position, i.e., the pose, of the camera coordinate system with respect to the ground coordinate system can be directly obtained through the imu. The coordinates of the position to be repaired in the camera coordinate system are multiplied by the transformation matrix, and then they are converted into the coordinates of the position to be repaired in the ground coordinate system.

[0159] Step S60. Determine the target path of the repair operation based on the coordinates of the position to be repaired, and control the steel girder box repair manipulator 2 of the steel girder box crack repair device to move along the path and perform crack repair;

[0160] Specifically, according to the current coordinates of the position to be repaired, the control host 1 of the steel girder box crack repair device uses inverse kinematics to calculate the joint angles of the steel girder box repair manipulator 2 to ensure that the end repair mechanism 5 on the steel girder box repair device can accurately move to the position to be repaired. The inverse kinematics formula is formula (19):

[0161] q i = f -1 (P i )(19)

[0162] Among them, q i is the joint angle vector of the steel girder box repair manipulator 2 corresponding to the i-th position coordinate to be repaired, P i is the i-th position coordinate to be repaired, f -1 () is the solution function of inverse kinematics, which calculates the joint angle values of each joint of the steel girder box repair manipulator 2.

[0163] To ensure the smoothness and safety of the steel girder box repair manipulator 2 during the entire movement process, a trajectory smoothing algorithm is used to interpolate the calculated joint angles, thereby generating a continuous trajectory curve.

[0164] To avoid the manipulator colliding with suddenly emerging obstacles during the movement process, the movement path is optimized, that is, the distance between adjacent repair points is minimized, min||P i+1 -P i ||, where P i+1 is the coordinate of the (i + 1)-th position to be repaired; P i is the coordinate of the i-th position to be repaired. Thus, the movement time of the manipulator is minimized, and at the same time, the change of the joint angle is ensured to be smooth.

[0165] After the steel girder box repair manipulator 2 reaches the position to be repaired, the control host 1 issues a repair instruction to perform the repair operation. The repair operation may include punching, bolt tightening, etc., and corresponding operations are performed according to the crack type and repair plan.

[0166] As Figure 5 shown, it is a schematic diagram of the overall process of the steel box girder crack repair method. Through the calibration of the steel box girder crack repair device, combined with the acquisition of RGB images, depth images, and IMU data, the precise positioning and path planning of the position to be repaired are completed. First, feature points of the position to be repaired are extracted using the RGB image, and depth information is obtained by combining the image collected by the depth camera. At the same time, the relative pose between the camera coordinate system and the ground coordinate system is updated through the IMU data. Subsequently, point cloud data of the position to be repaired is generated based on the RGB image and the depth image, and it is transformed from the camera coordinate system to the ground coordinate system through the IMU data. Finally, the target path of the repair operation is planned based on the point cloud data in the ground coordinate system to guide the device to complete the crack repair task.

[0167] Embodiment 2:

[0168] As Figure 6 shown, this embodiment provides a steel box girder crack repair device, and the device includes:

[0169] A steel girder box repair manipulator 2, which is composed of an upper arm 21, a forearm 23, and a rotary joint 22. One end of the upper arm 21 is connected to the forearm 23 through the rotary joint 22;

[0170] A manipulator magnetic base 8, which is arranged at one end of the upper arm 21 far from the rotary joint 22, and the manipulator magnetic base 8 is magnetically fixed on the steel box girder 7;

[0171] A terminal repair mechanism 5, which is fixedly connected to one end of the forearm 23 far from the rotary joint 22 through a flexible connector 6;

[0172] The camera component 3 is an integrated component composed of a depth camera and an RGB camera. The camera component 3 is fixed on the bracket at one end of the forearm 23 near the end repair mechanism 5.

[0173] During use, the steel box girder repair robotic arm 2 can achieve flexible movement through the upper arm 21, forearm 23 and rotary joint 22. The robotic arm magnetic adsorption base 8 fixes the steel box girder repair robotic arm 2 on the steel box girder 7 to ensure stability. The end repair mechanism 5 is connected to the forearm 23 through the flexible connector 6. The flexible connector 5 can buffer stress and prevent the steel box girder repair robotic arm 2 from being damaged due to stress concentration during the repair process. The camera component 3 integrates a depth camera and an RGB camera and is installed on the bracket of the forearm 23 to collect images and depth information of the crack position in real time and accurately guide the repair operation.

[0174] The device also includes an end repair mechanism magnetic adsorption base 4 and a control host 1. The end repair mechanism magnetic adsorption base 4 is arranged at one end of the end repair mechanism 5 far from the flexible connector 6. The control host 1 is connected to the steel box girder repair robotic arm 2 through a cable.

[0175] The control host 1 is connected to the steel box girder repair robotic arm 2 through a cable and is responsible for the operation control, data transmission and task coordination of the overall system. When the control host 1 determines the position to be repaired, it sends a movement instruction to the steel box girder repair robotic arm 2. The instruction carries a target path, and the cable and the steel box girder repair robotic arm 2 move according to the target path. When moving to the position to be repaired, the end repair mechanism magnetic adsorption base 4 performs an adsorption operation to firmly adsorb the steel box girder repair robotic arm 2 on the surface of the steel box girder 7 to ensure the stability of the repair operation.

[0176] The flexible connector 6 includes multiple groups of support rods 61, a fixed platform 63 and a moving platform 62. Each group of support rods 61 includes a first connecting rod 611, a second connecting rod 613 and a spring 612. One end of the first connecting rod 611 is connected to the spring 612 and the second connecting rod 613. The other end of the first connecting rod 611 is fixedly connected to the fixed platform 63. The ends of the second connecting rod 613 and the spring 612 far from the first connecting rod 611 are fixedly connected to the moving platform 62. As Figure 7 shown, the flexible connector 6 is used to offset the stress between the end repair mechanism 5 and the steel box girder repair robotic arm 2 caused by the adsorption of the end repair mechanism magnetic adsorption base 4. The three groups of support rods 61 are evenly distributed at 120° between the fixed platform 63 and the moving platform 62 to form a symmetric support structure, ensuring high structural stability when the flexible connector is stressed and guaranteeing the smooth operation of the end repair mechanism 5.

[0177] Each group of support rods 61 further includes a revolute pair 616, a prismatic pair 615 and a spherical pair 614. The revolute pair 616 is arranged between the first connecting rod 611 and the fixed platform 63. The prismatic pair 615 is arranged between one end of the first connecting rod 611, the spring 612 and the second connecting rod 613. The spherical pair 614 is arranged between the spring 612, the second connecting rod 613 and the moving platform 62. The revolute pair 616 enhances the angle adjustment ability between the first connecting rod 611 and the fixed platform 63. The prismatic pair 615 provides axial buffering for the spring 612 and the second connecting rod 613. The spherical pair 614 improves the multi-directional movement adaptability of the moving platform 62. The three work together to enable the flexible connector 6 to efficiently absorb and relieve stress, adapt to complex stress environments, and at the same time ensure the stability and accuracy of the operation of the end repair mechanism 5, greatly improving the flexibility and reliability of the steel box girder crack repair device.

[0178] The above are only the preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention can have various changes and modifications. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

[0179] The above is only the specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present invention can easily think of changes or replacements, which should all be covered within the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A method for repairing cracks in a steel box girder, characterized in that: include: Based on the image acquisition by a camera component (3) on the steel box girder crack repair device, a plurality of first images and a plurality of second images are acquired respectively, wherein the camera component (3) comprises a depth camera and an RGB camera, the first images are acquired by the RGB camera, and the second images are acquired by the depth camera; Inputting the plurality of first images into a trained crack position prediction model to predict the crack position, and obtaining a prediction result of the position to be repaired, wherein the crack position prediction model is a model constructed based on the YOLO algorithm; Determine a first image and a second image of a corresponding position based on the prediction result of the position to be repaired, and determine the point cloud coordinates of the position to be repaired through the first image and the second image, wherein the point cloud coordinates are coordinates in a camera coordinate system; Determine the relative position and posture between the camera coordinate system and the ground coordinate system based on the IMU data, wherein the relative position and posture is used to characterize the spatial relationship between the two coordinate systems; Based on the relative position and posture, the point cloud coordinates of the position to be repaired are converted into a ground coordinate system to obtain the coordinates of the position to be repaired; A target path for the repair operation is determined based on the coordinates of the position to be repaired, and a steel box girder repair mechanical arm (2) of the steel box girder crack repair device is controlled based on the target path to move the path and perform crack repair.

2. The method for repairing cracks in a steel box beam according to claim 1, characterized in that , input the plurality of the first images into a trained crack position prediction model to predict the crack position, and obtain a prediction result of the position to be repaired, wherein the crack position prediction model is a model constructed based on the YOLO algorithm, and includes: Build an initial model based on the YOLO algorithm; Acquire images of the steel box in a normal state and a cracked state, wherein the images of the steel box in the cracked state carry a crack position marking frame; Input the steel box image into the initial model for prediction to obtain a prediction result, wherein the prediction result is a prediction box marking a position to be repaired on the steel box image; Calculating a loss function based on the prediction result and the crack position annotation box to obtain a first loss function; Calculating a loss function based on the prediction result and the closest crack position annotation box to obtain a second loss function; Constructing a target loss function based on the first loss function and the second loss function; The parameters of the initial model are adjusted until the target loss function is less than a set threshold, and the corresponding initial model is the crack position prediction model.

3. The method for repairing cracks in a steel box beam according to claim 1, characterized in that , based on the prediction result of the position to be repaired, determining the first image and the second image of the corresponding position, and determining the point cloud coordinates of the position to be repaired through the first image and the second image, including: Combining the pixel coordinates and the depth value of each pixel on the second image based on the intrinsic parameters of the depth camera, and converting them into three-dimensional coordinates in the depth camera coordinate system; Based on the external parameters of the depth camera and the RGB camera, convert the three-dimensional coordinates in the depth camera coordinate system into the three-dimensional coordinates in the camera coordinate system; Based on the intrinsic parameters of the RGB camera, convert the three-dimensional coordinates in the camera coordinate system into two-dimensional pixel coordinates in the camera coordinate system; The depth value of each pixel on the second image is aligned and fused with the corresponding two-dimensional pixel coordinates to obtain the point cloud coordinates of the position to be repaired.

4. The method for repairing cracks in a steel box beam according to claim 3, characterized in that , align and fuse the depth value of each pixel on the second image with the corresponding two-dimensional pixel coordinates to obtain the point cloud coordinates of the position to be repaired, the internal parameters of the RGB camera include the horizontal axis focal length of the RGB camera, the vertical axis focal length of the RGB camera, the optical center horizontal coordinate and the optical center vertical coordinate, including: Calculating a ratio of a depth value in the second image to a preset conversion ratio to obtain a depth coordinate value of a point cloud coordinate; Calculating the difference between the abscissa in the two-dimensional pixel coordinates and the abscissa of the optical center, and calculating the product of the difference and the depth coordinate value to obtain a first product; Calculate the ratio of the first product to the horizontal axis focal length of the RGB camera to obtain the horizontal coordinate value of the point cloud coordinate; Calculating the difference between the ordinate in the two-dimensional pixel coordinates and the ordinate of the optical center, and calculating the product of the difference and the depth coordinate value to obtain a second product; Calculate the ratio of the second product to the vertical axis focal length of the RGB camera to obtain the ordinate value of the point cloud coordinates; The point cloud coordinates are composed based on the abscissa value, the ordinate value, and the depth coordinate value.

5. The method for repairing cracks in a steel box beam according to claim 4, characterized in that ,The point cloud coordinate calculation formula of the position to be repaired is: Among them, x sc is the horizontal axis value; y sc is the vertical coordinate value; z sc is the depth coordinate value; x ij is the horizontal coordinate in the two-dimensional pixel coordinate; y ij is the vertical coordinate in the two-dimensional pixel coordinate; d ij is the depth value; u0 r is the abscissa of the optical center; v0 r fx is the ordinate of the optical center; r fy is the horizontal focal length of the RGB camera; r is the vertical axis focal length of the RGB camera; r is the conversion ratio.

6. The method for repairing cracks in a steel box beam according to claim 2, characterized in that ,Based on the YOLO algorithm, an initial model is constructed, which includes a feature extraction module, a feature fusion module and a multi-scale detection module, including: An extraction unit is formed based on sequentially stacking convolution modules and C3 feature extraction modules; Combining the output of the extraction unit with the SPPF feature pooling module to form the feature extraction module; The feature fusion module is formed based on the CS convolution module, the upsampling module, the feature splicing module and the C3 feature extraction module stacked in sequence; The initial model is constructed based on the feature extraction module, the feature fusion module and the multi-scale detection module.

7. A steel box girder crack repair device, characterized in that: include: A steel box girder repairing mechanical arm (2), the steel box girder repairing mechanical arm (2) comprising an upper arm (21), a forearm (23) and a rotary joint (22), one end of the upper arm (21) being connected to the forearm (23) via the rotary joint (22); A mechanical arm magnetic base (8), wherein the mechanical arm magnetic base (8) is arranged at one end of the upper arm (21) away from the rotating joint (22), and the mechanical arm magnetic base (8) is magnetically fixed on the steel box beam; An end repair mechanism (5), wherein the end repair mechanism (5) is fixedly connected to an end of the forearm (23) away from the rotary joint (22) via a flexible connector (6); A camera component (3), wherein the camera component (3) is an integrated component comprising a depth camera and an RGB camera, and the camera component (3) is fixed to a bracket at one end of the forearm (23) close to the terminal repair mechanism (5).

8. The steel box girder crack repairing device according to claim 7, characterized in that: The device also includes a terminal repair mechanism magnetic base (4) and a control host (1). The end repair mechanism magnetic base (4) is arranged at an end of the end repair mechanism (5) away from the flexible connector (6); The control host (1) is connected to the steel beam box repair mechanical arm (2) via a cable.

9. The steel box girder crack repairing device according to claim 7, characterized in that: The flexible connector (6) comprises a plurality of support rods (61), a fixed platform (63) and a movable platform (62). Each group of support rods (61) comprises a first connecting rod (611), a second connecting rod (613) and a spring (612); one end of the first connecting rod (611) is connected to the spring (612) and the second connecting rod (613); the other end of the first connecting rod (611) is fixedly connected to the fixed platform (63); and the second connecting rod (613) and one end of the spring (612) away from the first connecting rod (611) are fixedly connected to the movable platform (62).

10. The steel box girder crack repairing device according to claim 9, characterized in that: Each group of the support rods (61) further includes a rotation pair (616), a translation pair (615) and a spherical pair (614). The rotating pair (616) is arranged between the first connecting rod (611) and the fixed platform (63), the moving pair (615) is arranged between one end of the first connecting rod (611) and the spring (612) and the second connecting rod (613), and the spherical pair (614) is arranged between the spring (612) and the second connecting rod (613) and the movable platform (62).

Images