Remote repair welding repair method and system for cracked steel bridge structure in combination with machine vision and digital twinning

Abstract

The invention belongs to the technical field of bridge maintenance, and discloses a machine vision and digital twinning combined remote repair welding repair method and system for a cracked steel bridge structure. A transmission steel bridge maintenance method mainly depends on manual detection and field repair, and has many limitations in practical application and is difficult to meet bridge maintenance requirements. The method comprises the following steps: acquiring steel bridge data through a detection robot to identify a steel bridge crack, generating a three-dimensional model of the steel bridge crack based on the steel bridge crack data, performing finite element stress analysis, generating a crack welding strategy, and issuing the crack welding strategy to a welding robot to execute a welding action; further dynamically mapping the welding information to the three-dimensional model of the crack, fusing the three-dimensional dynamic model of crack repair into a VR operation interface, and monitoring the welding process of the welding robot through the VR operation interface; and finally, the welding robot detects the repair integrity of the steel bridge crack, and the repair integrity information is updated to the three-dimensional dynamic model of steel bridge crack repair.

Description

Technical Field

[0001] This invention relates to the field of bridge maintenance technology, specifically to a method and system for remote welding repair of cracked steel bridge structures that combines machine vision and digital twins. Background Technology

[0002] Steel bridges are widely used in modern bridge engineering due to their advantages such as lightweight, high load-bearing capacity, and rapid construction, especially in long-span bridges and urban viaducts. However, with long-term service, steel bridge structures inevitably experience problems such as fatigue cracks, weld failures, and pavement damage.

[0003] These problems not only affect the service life of steel bridges but may also lead to serious safety accidents. Traditional steel bridge maintenance methods mainly rely on manual inspection and on-site repair, but this method has many limitations in practical applications and is difficult to meet the needs of modern steel bridge maintenance.

[0004] First, working at heights carries high risks. Steel bridge structures typically have considerable height and complex engineering structures. For example, the internal height of a steel box girder may be less than 1 meter, and the working space at truss joints may be less than 0.8 meters. Manual maintenance requires the erection of suspended platforms, ladders, or the use of aerial work platforms. The setup and use of these devices are not only time-consuming and labor-intensive but also subject to limitations imposed by weather conditions and the working environment. Working at heights is inherently dangerous, and workers are prone to fatigue when working in confined spaces for extended periods, further increasing the risk of accidents.

[0005] Secondly, traditional repair techniques have shortcomings. Traditional steel bridge repair methods mainly rely on reinforcement techniques, such as welding studs and then pouring concrete or using external steel plates for reinforcement. While these methods improve the load-bearing capacity of the structure to some extent, they do not directly repair the cracks in the parent material.

[0006] Residual stress generated during welding can lead to secondary cracking, especially under fatigue loads, where crack propagation rates accelerate. Furthermore, traditional repair methods typically require large-scale structural modifications, increasing construction costs and time, and significantly disrupting the normal use of the bridge.

[0007] Finally, traditional crack location methods lack sufficient accuracy. While visual inspection is currently the primary method for damage detection in steel bridges, this approach struggles to identify micron-sized cracks (such as hidden damage smaller than 0.1 mm). Early crack identification is crucial for preventing crack propagation, but traditional methods often rely on experienced inspectors, leading to subjective results and a high risk of missed detections. Furthermore, visual inspection cannot obtain real-time crack geometry parameters (such as length, depth, and angle), making it difficult to develop precise repair plans. Summary of the Invention

[0008] The purpose of this invention is to provide a method and system for remote welding repair of cracked steel bridge structures that combines machine vision and digital twins, so as to solve the problems mentioned in the background art.

[0009] The main design concept of this invention is as follows: Based on the above problems, this application proposes to use machine vision technology in conjunction with digital twin models to automatically detect cracks in cracked steel bridge structures, create 3D crack models, generate welding strategies, remotely monitor the welding process, and intelligently detect repair quality to form a repair closed loop.

[0010] Machine vision technology can identify cracks in real time and generate high-precision 3D point cloud data. By equipping a high-resolution industrial camera and an infrared thermal imager, the system based on the YOLOv5 algorithm can quickly and accurately detect information such as the location, length and depth of cracks, with a positioning accuracy of ±0.05 mm.

[0011] Digital twin technology uses the Unity3D engine to build a 3D model of a cracked steel bridge, dynamically simulates the repair process and optimizes process parameters, automatically generates welding paths, interpass temperatures and weld bead layout strategies, predicts the range of the heat-affected zone, reduces actual operational risks, and improves welding efficiency and quality.

[0012] The welding robot uses a magnetic adsorption chassis and a six-degree-of-freedom robotic arm, which can move freely on steel surfaces with an inclination angle of ≤85°. Equipped with a pulsed MIG welding torch and a molten pool monitoring module, it can perform multi-layer and multi-pass repair welding and provide real-time feedback on the penetration depth and temperature information to ensure repair quality.

[0013] Combining machine vision with digital twin technology for remote welding repair of cracked steel bridge structures can significantly improve detection and repair accuracy, reduce the risks of manual high-altitude operations, reduce maintenance costs, improve repair efficiency, overcome the limitations of traditional methods, promote the intelligent development of bridge maintenance technology, and provide safer, more efficient, and more economical solutions for modern bridge engineering.

[0014] To achieve the above objectives, the present invention provides the following technical solution: A method for identifying cracks in steel bridge structures using machine vision includes the following steps: Data augmentation processing is performed on the original steel bridge image to obtain an enhanced crack image; The YOLOv5 model was used to detect cracks in the enhanced crack image to obtain the crack recognition box; The crack recognition frame is superimposed onto the original steel bridge image to obtain the crack recognition image.

[0015] More preferably, the step of performing data augmentation processing on the original steel bridge image to obtain the crack-enhanced image includes the following steps: Grayscale analysis and corrosion processing were performed on the original steel bridge image; A second-order differential operator is used to perform edge detection on the eroded image data; A mask image is generated for the detected closed internal edge region, and the original steel bridge image is masked using the mask image to obtain a crack enhancement image.

[0016] More preferably, the convolution kernel of the second-order differential operator is: (1) (2) When the edge is unclear after detection by formula (1), it is further detected by formula (2).

[0017] A remote welding repair method for cracked steel bridge structures combining machine vision and digital twins includes the following steps: Acquire image data of the steel bridge; The steel bridge image data was processed using the machine vision-integrated steel bridge crack identification method to obtain steel bridge crack data. A 3D model of steel bridge cracks was generated based on the crack data of the steel bridge. Based on the three-dimensional model of the crack in the steel bridge, a crack welding strategy is generated using the finite element analysis method. The welding process is executed according to the crack welding strategy, and the welding process information is dynamically mapped to the three-dimensional model of the steel bridge crack to form a three-dimensional dynamic model for steel bridge crack repair. Based on a three-dimensional dynamic model for steel bridge crack repair, the welding process is remotely monitored. The integrity of the repair of cracks in the steel bridge is detected, and the repair integrity information is updated in the three-dimensional dynamic model of the steel bridge crack repair.

[0018] More preferably, the steel bridge data is collected by a detection robot, which is equipped with an industrial camera, an infrared thermal imager, a laser displacement sensor and a control unit, and the industrial camera, infrared thermal imager and laser displacement sensor are all electrically connected to the control unit.

[0019] More preferably, the crack data of the steel bridge includes the crack length, depth, and stress concentration area.

[0020] More preferably, the physical dimensions of the crack length and depth are determined by the following formula: in, The physical dimension representing the crack length. Let be the pixel length of the crack in the image. To calibrate the distance, The focal length of an image pixel. Physical dimensions representing crack depth. The pixel depth of the crack in the image.

[0021] More preferably, the crack welding strategy includes welding path, interpass temperature, and weld bead arrangement.

[0022] More preferably, the welding process is performed by a welding robot, which is equipped with a pulsed MIG welding torch, an ultrasonic flaw detector, an industrial camera, and a control unit, and the pulsed MIG welding torch, the ultrasonic flaw detector, and the industrial camera are all electrically connected to the control unit.

[0023] A remote welding repair system for cracked steel bridge structures combining machine vision and digital twins, executing the aforementioned remote welding repair method for cracked steel bridge structures combining machine vision and digital twins, includes the following modules: The crack recognition module, installed on the inspection robot, is used to collect image data of the steel bridge to identify cracks in the steel bridge and transmit the crack data to the digital twin module. The digital twin module, installed on a remote server, is used to generate a three-dimensional model of the steel bridge crack based on the crack data, generate a crack welding strategy based on the three-dimensional model using the finite element analysis method, and then send the crack welding strategy to the crack welding execution module. The crack welding module is installed on the welding robot to perform welding actions according to the crack welding strategy and dynamically map the welding information onto the crack 3D model. The remote monitoring module, installed on a remote server, monitors the welding process of the welding robot through a VR operating interface based on a three-dimensional dynamic model of crack repair. The repair detection module, installed on the welding robot, detects the integrity of crack repair and updates the repair integrity information to the three-dimensional dynamic model of crack repair.

[0024] Compared with the prior art, the beneficial effects of the present invention are: The machine vision-based crack recognition method for steel bridge structures provided in this application significantly improves the accuracy and reliability of micro-crack recognition by performing targeted preprocessing and intelligent fusion of steel bridge images. First, by using principal grayscale region erosion and second-order differential operator adaptive edge detection, the characteristics of micro-cracks are effectively amplified while suppressing false edge interference such as steel bridge surface textures, thus reducing the probability of false detection. Second, masking technology is used to enhance the crack region, further improving target saliency. Finally, precise bounding box localization and overlay algorithms ensure accurate visualization of the detection results in the original image.

[0025] This application provides a method and system for remote welding repair of cracked steel bridge structures that combines machine vision and digital twins. Through an integrated solution of "detection-planning-repair-verification", it solves the problems of high risk of high-altitude operations, insufficient crack positioning accuracy, and defects in repair process in traditional steel bridge maintenance.

[0026] Technically, the inspection robot is equipped with a 48-megapixel industrial camera and a laser displacement sensor, achieving a positioning accuracy of ±0.05mm and 3D modeling resolution of 0.1mm, accurately identifying micron-level cracks; the magnetic adsorption six-degree-of-freedom robot can move on steel surfaces with an inclination of ≤85°, adapting to complex parts such as U-rib welds; the digital twin platform, based on Unity3D, integrates the 3D crack model with finite element stress analysis, automatically generating welding paths and dynamically optimizing interpass temperature and weld bead strategies, supporting VR remote parameter correction.

[0027] This application improves repair efficiency by 8-10 times compared to manual methods, reduces maintenance costs by over 40%, directly repairs the parent material to avoid secondary damage, and establishes a quality closed loop through molten pool monitoring and ultrasonic testing. It is suitable for harsh environments such as high altitudes and narrow spaces, providing a safe and efficient end-to-end solution for intelligent maintenance of steel bridges, promoting the upgrading of bridge structure long-life service technology, and possessing significant engineering application value and market promotion potential. Attached Figure Description

[0028] Figure 1 This is a flowchart of the remote welding repair method for cracked steel bridge structures that combines machine vision and digital twins according to the present invention. Figure 2 This is a schematic diagram of the remote welding repair system for cracked steel bridge structures that combines machine vision and digital twins according to the present invention. Figure 3 This is a schematic diagram of the steel bridge structure crack recognition method combined with machine vision according to the present invention. Figure 4 This is a simulation diagram comparing the crack identification accuracy of the present invention with that of traditional methods; Figure 5 This is a comparison diagram showing the efficiency improvement of the remote welding repair solution for cracks in steel bridges according to the present invention compared with traditional solutions. Figure 6 This is a cost control comparison chart between the remote welding repair solution for steel bridge cracks of the present invention and the traditional solution; Figure 7 This is the original steel bridge image used in the simulation experiment of this invention; Figure 8 This is an image of the simulation experimental data after enhancement processing according to the present invention; Figure 9 The image is a result of the YOLOv5 model detection in the simulation experiment of this invention. Figure 10 This is an image of the simulation experiment frame after the transfer of the present invention. Detailed Implementation

[0029] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0030] In the description of this invention, it should be noted that the terms "upper," "lower," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance. Furthermore, the terms "installation" and "connection" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances. Example 1

[0031] like Figures 1 to 10 As shown, this embodiment provides a method for identifying cracks in steel bridge structures using machine vision, including the following steps: Data augmentation processing is performed on the original steel bridge image to obtain an enhanced crack image; The YOLOv5 model was used to detect cracks in the enhanced crack image to obtain the crack recognition box; The crack recognition frame is superimposed onto the original steel bridge image to obtain the crack recognition image.

[0032] This embodiment also provides a remote welding repair method for cracked steel bridge structures that combines machine vision and digital twins, including the following steps: The inspection robot collects image data of the steel bridge. The steel bridge structure crack identification method described above is used to process the collected steel bridge image data to obtain steel bridge crack data, and the corresponding steel bridge crack data is transmitted to the digital twin platform. The digital twin platform generates a 3D model of the cracks in the steel bridge based on the crack data. The digital twin platform uses the finite element analysis method to generate a crack welding strategy based on a three-dimensional model of a crack in a steel bridge, and then distributes the crack welding strategy to the welding robot. The welding information of the welding robot is dynamically mapped to the three-dimensional model of the steel bridge crack, forming a three-dimensional dynamic model for steel bridge crack repair. The three-dimensional dynamic model of steel bridge crack repair is integrated into the VR operation interface, and the welding process of the welding robot is monitored through the VR operation interface. The welding robot inspects the repair integrity of cracks in the steel bridge and updates the repair integrity information into the three-dimensional dynamic model of the crack repair.

[0033] The steel bridge inspection robot is a tracked robot equipped with a magnetic rectangular tracked chassis, ensuring stable movement on the inclined surface of the steel bridge. The robot is also equipped with a six-degree-of-freedom robotic arm, ensuring precise control over any position and orientation in three-dimensional space.

[0034] The steel bridge inspection robot is equipped with an industrial camera, an infrared thermal imager, a laser displacement sensor, and a control unit. The industrial camera captures high-definition images in real time, the infrared thermal imager detects temperature changes, and the laser displacement sensor accurately measures the geometric features of cracks. The control unit integrates the YOLOv5 algorithm, enabling efficient and accurate identification of cracks in steel bridges.

[0035] As one specific implementation method, the industrial camera is installed at the front end of the robotic arm and connected to the robot's control unit. It adopts the Keyence VS-L320MX industrial camera, which has 3.2 megapixels, IP67 waterproof, PoE power supply + Gigabit Ethernet, and directly connects to the control unit. It has a built-in edge computing module that can preprocess images in real time (such as noise reduction and contrast enhancement) to reduce the backend computing power pressure.

[0036] The VS-L320MX industrial camera, with a 48-megapixel resolution, can achieve a physical resolution of 0.5μm / pixel. Combined with the Zernike subpixel edge detection algorithm, it can identify crack width and length with micrometer-level accuracy (≤1μm).

[0037] An infrared thermal imager is mounted on the front end of the robotic arm and connected to the robot's control unit. It uses a Testo 885 infrared thermal imager with a 320×240 uncooled focal plane and supports super-resolution technology (interpolation to 640×480); temperature range: -20℃ to 1200℃ (extended via a high-temperature module); accuracy: ±2℃ or ±2% of the reading; USB 3.0 + MicroSD card support for real-time data transmission and local storage; automatic generation of overall thermal images of the steel bridge; identification of large-scale temperature anomalies; and prediction of corrosion risk in cracked areas using humidity sensor data.

[0038] A laser displacement sensor is mounted on the front end of the robotic arm and connected to the robot's control unit. It employs the Mi-Iridium OptoNCDT1320 laser displacement sensor, with a measurement range of 10-500mm, linear accuracy of ±0.1%F·S, a measurement frequency of 4kHz (0.5μm), and supports dynamic crack propagation monitoring. It features analog input (0-10V) + RS422, allowing direct connection to the control unit's PLC. The sensor uses 658nm red light to reduce diffuse reflection interference from metal surfaces and employs algorithms to filter false signals from rust particles on the steel bridge surface. Based on the triangulation principle, the laser displacement sensor achieves a spatial positioning accuracy of ±0.05mm and a resolution of 0.1mm, accurately capturing the three-dimensional features of crack depth and spatial orientation.

[0039] The ICP algorithm is used to register visual images and laser data with a registration error of ≤0.1mm. This achieves high-precision positioning of cracks in three-dimensional space in terms of "position-shape-depth", which is far superior to traditional manual visual inspection (positioning error ≥2mm) or single sensor solutions.

[0040] YOLOv5 (You Only Look Once version 5) is a high-efficiency real-time object detection algorithm launched by the Ultralytics team in 2020, implemented based on the PyTorch framework. As an important branch of the YOLO series, it has become one of the preferred object detection tools in the industry due to its excellent engineering optimization and ease of use.

[0041] The steel bridge inspection robot used in this application, as well as the industrial camera, infrared thermal imager, laser displacement sensor, control unit, and control system configured on it, the YOLOv5 algorithm used, and the identification of steel bridge cracks through machine vision technology, are all existing technologies.

[0042] Scanning rules are preset in the control unit, including the range, acquisition heartbeat, path, etc. After the scanning rules are preset, the steel bridge inspection robot moves on the surface of the steel bridge. The industrial camera, infrared thermal imager, and laser displacement sensor collect data according to the scanning rules and transmit the collected data to the control unit via wired transmission.

[0043] The control system of the control unit uses the YOLOv5 algorithm to identify the transmitted steel bridge image data. If a crack is detected in the steel bridge, the three-dimensional data of the crack is further extracted from the corresponding image data. Finally, the corresponding steel bridge crack image data and crack three-dimensional data are transmitted wirelessly to a remote digital twin platform.

[0044] When using the YOLOv5 algorithm to identify cracks in steel bridges based on acquired image data, data augmentation processing of the image is required before crack identification in order to reduce the probability of false detection of false edges.

[0045] Considering that cracks in steel bridges are relatively small in their early stages, traditional methods are difficult to identify them. Therefore, for steel bridge image data, grayscale analysis is first performed to obtain the main grayscale values ​​of the steel bridge image. Then, an erosion operation is performed in the main grayscale value region with an erosion kernel size of 3×3 to amplify the crack features in the steel bridge image.

[0046] Next, a second-order differential operator is used to perform edge detection on the eroded image. The convolution kernel of the second-order differential operator is represented as follows: (1) (2) Equation (1) is a second-order differential enhancement operator that can detect possible edges around the target pixel at various angles. Equation (2) is another second-order differential enhancement operator, which is mainly used to enhance the detection of possible edges in the four directions of the target pixel (up, down, left, and right), and can also detect possible edges in the remaining four directions.

[0047] When the edge detected by (1) is unclear, edge detection can be performed again using (2). First, determine the number of pixels of the edge detected by (1), and then compare it with twice the number of pixels in the width direction of the steel bridge segment. If it is less than twice the number of pixels in the width direction of the steel bridge segment, it means that the edge is unclear, and if it is greater than twice the number of pixels in the width direction of the steel bridge segment, it means that the edge is clear.

[0048] Next, a mask image is generated based on the detected internal closed edges. This mask image is then used to mask the original steel bridge image, resulting in a crack-enhanced image. Finally, the YOLOv5 model is used for crack detection to obtain the original bounding box image.

[0049] Finally, the recognition boxes are superimposed on the original steel bridge image data with the same relative image positions to obtain the final crack recognition image.

[0050] Assuming the bounding box is a red box (color code FF0000), when the bounding box is transferred, the image of the bounding box is first filtered using a red bandpass filter to obtain an image containing only the red box. Then, the endpoint detection operator is used to detect the endpoints of the image to obtain the four endpoints of the red box.

[0051] The endpoint detection operator is expressed as follows: (3) Next, in the pixel matrix of the recognition box image, the relative positions of these four endpoints in the image are determined (with the top left corner of the image as the origin). Then, the pixel values ​​of the same position pixels in the original steel bridge image are replaced with red. After that, the four endpoints are connected in sequence using red to obtain the final crack recognition image.

[0052] Its mathematical form is as follows: Assuming the coordinates of the four endpoints are (a, b), (a+m, b), (a, b+n), and (a+m, b+n), then the pixel... To pixel It also includes m-1 pixels, whose set is This involves changing the hue of all pixels in the set to red (FF0000), thus completing the pixel mapping. To pixel The connection.

[0053] pixel To pixel , arrive ,as well as arrive The pixels between them are processed in a similar way to complete the transfer of the red frame.

[0054] When extracting three-dimensional data of steel bridge cracks based on steel bridge image data, it is necessary to convert the pixel values ​​of the image into actual physical dimensions. The three-dimensional data of steel bridge cracks in this application includes the crack length, depth, and stress concentration region. The crack length and depth information are obtained using the following technical methods: (4) or (5) in, The length represents the physical size of the crack. Let be the pixel length of the crack in the image. To calibrate the distance, The focal length of an image pixel. The depth, representing the physical size of the crack. This represents the image pixel depth value.

[0055] In the extraction of three-dimensional data of cracks in steel bridges, stress concentration areas cannot be directly reflected by images alone. They need to be based on the crack geometric parameters (such as length, depth, tip shape, etc.) obtained from images and sensors, and realized through stress concentration factors.

[0056] (6) in, The crack depth is the depth value converted from image pixels corresponding to the physical dimensions measured by the laser displacement sensor. The radius of curvature at the crack tip is obtained by extracting the micro-morphology of the crack tip from the image and combining it with pixel-to-physical size conversion. The thickness of the steel bridge components is a known design parameter or obtained through laser scanning.

[0057] When the local stress concentration factor When the stress concentration factor of the material is exceeded (for steel structures, it is usually taken as 1.5 to 3.0, depending on the strength of the material), the corresponding area is the stress concentration area.

[0058] The digital twin platform described in this application is built on the Unity3D engine. Unity3D is a cross-platform, comprehensive development platform that supports the creation of 3D game videos, architectural visualizations, real-time 3D animations, and more. It is suitable for creating 3D architectural models and simulating construction processes. Supporting virtual reality and augmented reality, it is one of the mainstream engines for VR development, supporting various VR devices and providing an immersive interactive experience.

[0059] Unity3D uses computer vision technology to transform two-dimensional images into three-dimensional models based on the crack image data and three-dimensional crack data of the steel bridge transmitted by the inspection robot, and then integrates them into the Unity engine to achieve interactive visualization.

[0060] The 3D crack data extracted by the detection robot is based on the sensor coordinate system. Therefore, when constructing a 3D model of the steel bridge crack on the digital twin platform, it is necessary to convert the sensor coordinate system of the crack 3D data into the world coordinate system to obtain accurate geometric data of the steel bridge crack for model construction.

[0061] A sensor coordinate system is a local reference system that describes the sensor's measurement reference and data direction. Its origin and direction are determined by the sensor's physical structure and installation location. This coordinate system is crucial for multi-sensor fusion, data calibration, and spatial positioning.

[0062] Sensor coordinate transformation needs to be based on the sensor's installation parameters on the robotic arm, namely the sensor's pose relative to the robotic arm's end effector, such as offset and rotation angle, and the robot's positioning data in the steel bridge environment, namely the pose of the robotic arm base relative to the world coordinate system. This can be obtained through GPS, SLAM, or pre-calibrated reference points, and then calculated through a coordinate transformation matrix to ensure that the position and orientation of the crack's three-dimensional data in the world coordinate system are completely consistent with the actual crack in the steel bridge.

[0063] The sensor coordinate system is a local reference system that describes the sensor's measurement benchmark and data direction. Its origin and direction are determined by the sensor's physical structure and installation position. Sensor coordinate transformation is the core link connecting "front-end multi-sensor data fusion" and "back-end digital twin virtual-real mapping": it supports the spatial unification of multi-sensor data in the front and enables the precise positioning of the virtual model in the back, running through the entire process from "data acquisition" to "virtual-real mapping". This coordinate system is crucial for multi-sensor fusion, data calibration and spatial positioning.

[0064] Specifically, the formula for transforming 3D point cloud coordinates is: (7) In equation (7), These are the coordinates of a point in the actual coordinate system. The coordinates of the point in the sensor coordinate system. R It is a rotation matrix (3×3). T The translation vector is 3×1, obtained through hand-eye calibration. (Hand-eye calibration means obtaining the vector manually.) R The rotational relationship with the target along the X, Y, and Z axes is reflected by the following formula: (8) (9) (10) (11) In the above formula, Represents the rotation matrix. Indicates the rotation component along the vertical axis. Indicates the rotational component along the vertical axis. Represents the rotation component along the horizontal axis; Indicates the rotation angle of the horizontal axis. Indicates the rotation angle along the vertical axis. This indicates the rotation angle along the vertical axis.

[0065] The translation vector T is related to the target's relative coordinate axes, as shown in the following equation: (12) In the above formula, This indicates relative displacement along the horizontal axis. This indicates the relative displacement along the vertical axis. Indicates relative displacement along the vertical axis. The digital twin platform of this application, namely the Unity3D platform, integrates a finite element analysis plugin to perform finite element stress analysis. On the three-dimensional model of the crack in the steel bridge, it simulates the welding process of the crack and generates a crack welding strategy.

[0066] The crack welding strategy of this application includes welding path, interpass temperature, and weld bead arrangement. A segmented welding path is generated based on the three-dimensional point cloud data of the crack to avoid heat concentration. The interpass temperature is set according to the steel material, interpass temperature, and cooling strategy. A multi-layer, multi-bead strategy is adopted, with narrow weld beads used in the bottom layer to ensure penetration, and wide weld beads used in the surface layer to improve appearance.

[0067] The welding path refers to the continuous welding trajectory generated by the welding robot based on the crack three-dimensional point cloud data (acquired by the inspection robot) and the finite element stress distribution results, which "fills from the crack edge to the center and avoids stress concentration". The core objective is to achieve uniform filling of the crack area and minimize the heat-affected zone (HAZ).

[0068] Welding path planning logic: Based on the "minimum bounding rectangle" of the crack 3D model, a "spiral inward filling" strategy is adopted (to avoid the stacking defect caused by the overlap of the start / end point). Priority is given to welding the "high stress area" in the finite element analysis (such as the crack tip, and areas with stress values ​​≥200MPa are filled first).

[0069] Key parameters for welding path specifications include: 1. Path spacing: Based on the welding wire diameter (e.g., for 1.2mm welding wire, the spacing is set to 1.0-1.2mm to ensure that the fusion rate of adjacent weld passes is ≥95%).

[0070] 2. Starting point selection: Avoid the crack tip within 5mm (to prevent the tip from overheating and becoming embrittled), and start from the midpoint in the length direction of the crack and the outer edge in the width direction.

[0071] 3. Path accuracy: Trajectory coordinate error ≤ ±0.05mm (matching the positioning accuracy of the detection robot).

[0072] 4. Interpass temperature refers to the lowest temperature threshold on the surface of the previous weld bead during the interval between welding two adjacent weld bead layers. It needs to be determined based on the thermophysical properties (thermal conductivity, specific heat capacity) of the steel bridge base material (such as Q345 steel) and the results of finite element thermal field analysis. The core objective is to prevent interpass cold cracks and grain coarsening.

[0073] Temperature reference value: For Q345 steel (the common base material for steel bridges), the interpass temperature reference value is set at 150-200℃ (below 150℃, cold cracks are likely to occur, and above 200℃, the heat-affected zone is likely to soften). Welding dynamic adjustment rules: 1. Ambient temperature compensation: If the ambient temperature is ≤5℃, the lower limit of the inter-floor temperature is increased to 180℃; if the ambient temperature is ≥30℃, the lower limit is decreased to 130℃. 2. Crack depth compensation: When the crack depth is >5mm (≥3 weld layers are required), the lower limit of the interpass temperature is increased to 170℃ (to enhance interpass fusion). 3. Monitoring frequency: The temperature of the previous weld layer is collected every 0.5 seconds during the welding process (via an infrared temperature sensor), and the welding interval is adjusted in real time.

[0074] Weld bead layout refers to the number of layers, the number of weld beads per layer, and the offset of weld beads in a "multi-layer, multi-pass welding" process, determined based on the depth and width of the crack and a three-dimensional model. The core objective is to ensure that the crack is completely filled (penetration depth ≥ crack depth + 1mm) and that there are no unfused defects between weld beads.

[0075] Layer calculation rule: Number of weld layers = ceil(crack depth / effective weld height), where the effective weld height is equal to the welding wire diameter × 0.8 (e.g., for a 1.2mm welding wire, the effective height is 0.96mm; if the crack depth is 5mm, the number of layers ceil(5 / 0.96) is 6).

[0076] Number of weld passes per layer: Number of weld passes per layer = ceil(crack width / effective weld pass width), where the effective weld pass width = wire diameter × 1.2 (e.g., 1.2mm wire, effective width 1.44mm; if the crack width is 3mm, the number of passes per layer is ceil(3 / 1.44) = 3). Weld offset rules: adjacent welds are offset by “effective width × 0.5” along the crack width direction (e.g., 1.44mm effective width, offset by 0.72mm), and odd-numbered layers and even-numbered layers are arranged in a “centrally symmetrical manner” (to avoid repeated stacking at the same position).

[0077] Welding heat input, interpass cooling time, welding linear deformation, local stress concentration coefficient are strongly coupled with welding strategy, and are also deeply related to the digital twin platform (finite element simulation), crack three-dimensional detection data, steel basic properties and other aspects in the solution, jointly supporting the whole process from "crack detection" to "welding repair simulation and strategy generation".

[0078] Welding heat input is the thermal energy transferred per unit length of weld during the fusion welding process. It is a core parameter for measuring welding energy transfer efficiency and directly affects the metallurgical quality of the weld. Welding heat input needs to meet penetration requirements while controlling the interpass temperature to ≤150℃ to avoid overheating and secondary cracking. Welding heat input is calculated using the following formula: (13) In the above formula, Q is the heat input (J / mm), I is the welding current (A), and V is the welding voltage (V). This is due to the welding effect (approximately 0.7~0.9 for pulsed MIG welding). The welding speed is (mm / s).

[0079] Interpass cooling time is a key process parameter in multi-pass welding of thick plates, directly affecting the microstructure evolution, residual stress distribution, and probability of welding defects in the heat-affected zone (HAZ). It is calculated using the following formula: (14) In the above formula, The thermal conductivity of steel (approximately) ΔT is the interlayer temperature difference; for example, if the temperature drops from 200℃ to 150℃, then ΔT = 50K. q Heat flow per unit area A This refers to the heat dissipation area. A simplified engineering empirical formula is used. The molten pool monitoring module provides real-time feedback and adjustments.

[0080] When calculating the thermal deformation trend of the repair area, the linear deformation caused by welding is predicted using the following formula in order to optimize the welding path.

[0081] (15) In the above formula, Welding linear deformation Where L is the coefficient of thermal expansion of the steel reinforcement (approximately 12 × 10⁻⁶ / ℃), and L is the original length of the component. The value represents the welding temperature rise (°C).

[0082] Stress concentration areas are one of the main causes of structural failure in engineering structures, and the local stress concentration factor is... As a core parameter for assessing crack propagation risk, its detection and quantification are crucial for preventing fracture accidents. Local stress concentration factor for: (16) In the above formula, For half the length of the crack, Let be the radius of curvature at the crack tip.

[0083] The welding robot in this application has a similar structure to the inspection robot, but the core difference lies in the inclusion of a pulsed MIG welding torch and a molten pool monitoring module. The pulsed MIG welding torch achieves high-quality welding through dual-pulse precision heat control and digital inverter response; the molten pool monitoring relies on near-infrared vision and multi-sensor fusion to overcome the bottleneck of perception in highly interfering environments.

[0084] Inverse kinematics and adsorption force calculation are core technologies for welding robots to perform tasks such as climbing and welding, achieving precise control by combining kinematic modeling and physical and mechanical analysis. This application utilizes homogeneous transformation matrices... Reverse calculation of joint angles This enables precise positioning of the welding torch tip.

[0085] The calculation of the adsorption force of the magnetic adsorption chassis relies on theoretical formulas under ideal conditions, but in practical engineering, it needs to be comprehensively modified in combination with material properties, structural design, environmental factors, etc. The extreme formula for the adsorption force of the magnetic adsorption chassis of the welding robot in this application is: (17) In the above formula, B The value is the air gap magnetic flux density (unit: T). A Adsorption area (unit) ), Vacuum permeability ( ).

[0086] Inspection robots and welding robots rely on magnetic adsorption chassis to firmly adhere to the surface of steel bridges to carry out welding operations. Formula (17) is used to calculate the magnitude of the adsorption force that the chassis can generate. It is the core basis for verifying whether the chassis "adsorption force is sufficient" (after correction based on material properties, structural design, etc.). It can ensure that the robot will not fall off the surface of the steel bridge (including complex working conditions during inspection and welding), thereby ensuring the safety and stability of the welding operation. Therefore, it is directly related to the design and performance verification of the magnetic adsorption chassis.

[0087] The digital twin platform transforms the generated welding path from the Unity coordinate system to the welding robot's base coordinate system, and converts the interpass temperature and weld bead arrangement into welding control commands for the welding robot. Finally, the welding control commands are sent to the welding robot via wireless transmission technology.

[0088] The welding information from the welding robot is dynamically mapped onto a 3D model of the steel bridge crack, forming a 3D dynamic model for steel bridge crack repair. The welding robot collects real-time data of the welding process and transmits it to a digital twin platform via a wireless network. The digital twin platform analyzes the real-time welding data and visualizes and renders it on the 3D model of the steel bridge crack, forming a 3D dynamic model for steel bridge crack repair.

[0089] A 3D dynamic model of steel bridge crack repair is integrated into a VR user interface, allowing users to monitor the welding process of a welding robot. VR user interface design is a core element in creating an immersive experience. The VR user interface and the 3D model form a deeply coupled symbiotic relationship within the virtual reality system. The 3D model serves as the content carrier of the virtual world, while the VR user interface acts as the bridge for users to perceive and manipulate the model.

[0090] In this application, the VR user interface utilizes a VR headset and a large-screen interaction system. Specifically, the VR headset can be the HTC Vive Pro EYE, integrating eye tracking (0.5° accuracy) + an AMOLED screen, 20ms latency, and controller + iris recognition interaction. The large-screen interaction system can be an NEC X554UNS LED splicing screen with a resolution of 3840×2160 / unit, a 240Hz refresh rate, and support for active stereoscopic 3D + multi-touch.

[0091] After welding is completed, ultrasonic testing carried by the welding robot is used to detect the repair integrity of the steel bridge cracks, and the repair integrity information is updated to the three-dimensional dynamic model of the steel bridge crack repair.

[0092] Ultrasonic testing (UT) is a non-destructive testing technique that uses high-frequency sound waves (frequency > 20 kHz) to detect internal defects in materials. It is widely used in industrial quality inspection, engineering safety and other fields.

[0093] Example 2 like Figures 1 to 10 As shown, this embodiment provides a remote welding repair system for cracked steel bridge structures that combines machine vision and digital twins. The system, which executes the remote welding repair method for cracked steel bridge structures combining machine vision and digital twins, includes the following modules: The crack recognition module, installed on the inspection robot, is used to collect image data of the steel bridge to identify cracks in the steel bridge and transmit the crack data to the digital twin module. The digital twin module, installed on a remote server, is used to generate a three-dimensional model of the steel bridge crack based on the crack data, generate a crack welding strategy based on the three-dimensional model using the finite element analysis method, and then send the crack welding strategy to the crack welding execution module. The crack welding module is installed on the welding robot to perform welding actions according to the crack welding strategy and dynamically map the welding information onto the crack 3D model. The remote monitoring module, installed on a remote server, monitors the welding process of the welding robot through a VR operating interface based on a three-dimensional dynamic model of crack repair. The repair detection module, installed on the welding robot, detects the integrity of crack repair and updates the repair integrity information to the three-dimensional dynamic model of crack repair.

[0094] A comparison of data on efficiency improvement and cost control between the remote welding repair method for steel bridge cracks of the present invention and traditional methods in practical applications: Efficiency improvement comparison: The time cost of the traditional manual repair process includes: 1 hour of height protection + 0.5 hours of manual positioning + 6 hours of manual welding + 0.5 hours of manual inspection, with a total time consumption of about 8 hours; the time cost of the solution of this invention includes: 0.1 hours of digital twin planning + 0.9 hours of robotic welding + 0.1 hours of intelligent inspection, with a total time consumption of about 1 hour, improving efficiency by more than 8 times.

[0095] Cost reduction comparison: The cost of traditional manual repair includes: high-altitude operation fee of 3,000 yuan, labor cost of 2,000 yuan, and secondary repair cost of 1,000 yuan, with a total cost of about 6,000 yuan; the cost of this solution includes: equipment loss of 1,000 yuan and consumable cost of 1,000 yuan, with a total cost of about 2,000 yuan, a cost reduction of about 40%.

[0096] Simulation experiment: This experiment simulates and verifies the steel bridge crack recognition method combined with machine vision of this invention. First, the original steel bridge image undergoes data enhancement processing such as grayscale analysis, corrosion detection, and edge detection to generate an enhanced crack image. Then, the YOLOv5 model is used to detect cracks in the enhanced image, outputting bounding boxes containing the crack location and size. Finally, the bounding boxes are superimposed on the original image to generate an intuitive and visual crack recognition image.

[0097] Specifically, Figure 7 The original steel bridge image data shows cracks, multiple welds, and various colors. Figure 8 To perform data enhancement processing such as grayscale analysis, corrosion, and edge detection on the original steel bridge image using this method, it can be seen from the figure that only cracks were identified and retained their original color values, while other areas were set to uniform grayscale values. This avoids the influence of welds and other colors on crack identification and greatly improves the accuracy of identifying small cracks.

[0098] Figure 8 The system uses the YOLOv5 model to detect cracks in the data-augmented image. Once a crack is successfully detected, a red rectangle is drawn around the crack within an appropriate area, and the system outputs a series of specific coordinate data of the crack outline in the background. Figure 9 The crack detection bounding box detected by the YOLOv5 model is transferred to the original steel bridge image based on the corresponding position in the image, forming the final crack detection image.

[0099] The following is a sample of some core code for this algorithm: import cv2 import numpy as np def retain_black_and_fill_white(input_path, output_path): # Read image (supports color and grayscale) img = cv2.imread(input_path) if img is None: Raise a ValueError("Unable to read image, please check file path") # If the image is grayscale, convert it to three channels (for unified processing) if len(img.shape) == 2: img = cv2.cvtColor(img, cv2.COLOR_GRAY2BGR) # Define the black standard (R = 0, G = 0, B = 0) black_standard = np.array([0, 0, 0], dtype=np.uint8) # Create a mask: Mark the non - black areas # Note: The shape of img is (H, W, C), and we compare along the last dimension mask = ~(img == black_standard) mask = mask[:, :, 0] | mask[:, :, 1] | mask[:, :, 2] # Any channel not equal to 0 means non - black # Create a white - all image white_image = np.ones_like(img) * 255 # Copy the black areas in the original image to the white - all image result = np.where(~mask[:, :, np.newaxis], img, white_image) # Save the result cv2.imwrite(output_path, result) print(f"Processing completed, result saved to {output_path}") # Example usage input_image_path = r"D:\picture\original_image.jpg" # Replace with your image path output_image_path = r"D:\picture\output.jpg" # Output path retain_black_and_fill_white(input_image_path, output_image_path) import cv2 import numpy as np def transfer_red_rectangle(image_a_path, image_b_path, output_path): # Read Image img_a = cv2.imread(image_a_path) img_b = cv2.imread(image_b_path) if img_a is None or img_b is None: Raise a ValueError("Unable to read input image, please check file path") # -------------------------------------------------- # Step 1: Detect the red rectangle in image A # -------------------------------------------------- # Convert to HSV color space (more suitable for color segmentation) hsv_a = cv2.cvtColor(img_a, cv2.COLOR_BGR2HSV) # Define the red HSV range (can be adjusted according to actual results) lower_red = np.array([0, 70, 50], dtype=np.uint8) # Lower threshold upper_red = np.array([10, 255, 255], dtype=np.uint8) # High threshold # Create a red mask mask = cv2.inRange(hsv_a, lower_red, upper_red) # Morphological operations for noise removal kernel = np.ones((5, 5), np.uint8) mask = cv2.morphologyEx(mask, cv2.MORPH_CLOSE, kernel) # Find Outlines contours, _ = cv2.findContours(mask, cv2.RETR_EXTERNAL,cv2.CHAIN_APPROX_SIMPLE) If not contours: Raise a ValueError ("No red area detected. Please check the color range or image content"). # Find the outline with the largest area (assuming this is the target rectangle). max_contour = max(contours, key=cv2.contourArea) x, y, w, h = cv2.boundingRect(max_contour) # -------------------------------------------------- # Step 2: Calculate the relative position (normalized coordinates) # -------------------------------------------------- # Normalized coordinates [0~1] norm_x = x / img_a.shape[1] norm_y = y / img_a.shape[0] norm_w = w / img_a.shape[1] norm_h = h / img_a.shape[0] # -------------------------------------------------- # Step 3: Draw rectangles with the same relative positions on image B. # -------------------------------------------------- # Calculate the absolute coordinates on image B new_x = int(norm_x * img_b.shape[1]) new_y = int(norm_y * img_b.shape[0]) new_w = int(norm_w * img_b.shape[1]) new_h = int(norm_h * img_b.shape[0]) # Draw the red rectangle cv2.rectangle(img_b, (new_x, new_y), (new_x + new_w, new_y + new_h), (0, 0, 255), 2) # Save results cv2.imwrite(output_path, img_b) print(f"Processing complete, result saved to {output_path}") # Example usage image_a_path = "image_a.jpg" # Replace with the path to your image A image_b_path = "image_b.jpg" # Replace with your image B path output_path = "result.jpg" # Output path transfer_red_rectangle(image_a_path, image_b_path, output_path) It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered illustrative and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention, and no reference numerals in the claims should be construed as limiting the scope of the claims.

[0100] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for identifying cracks in steel bridge structures using machine vision, characterized in that, Includes the following steps: Data augmentation processing is performed on the original steel bridge image to obtain an enhanced crack image; The YOLOv5 model was used to detect cracks in the enhanced crack image to obtain the crack recognition box; The crack recognition frame is superimposed onto the original steel bridge image to obtain the crack recognition image.

2. The method for identifying cracks in steel bridge structures using machine vision as described in claim 1, characterized in that, The process of performing data augmentation on the original steel bridge image to obtain an enhanced crack image includes the following steps: Grayscale analysis and corrosion processing were performed on the original steel bridge image; A second-order differential operator is used to perform edge detection on the eroded image data; A mask image is generated for the detected closed internal edge region, and the original steel bridge image is masked using the mask image to obtain a crack enhancement image.

3. The method for identifying cracks in steel bridge structures using machine vision as described in claim 2, characterized in that, The convolution kernel of the second-order differential operator is: （1） （2） When the edge is unclear after detection by formula (1), it is further detected by formula (2).

4. A method for remote welding repair of cracked steel bridge structures combining machine vision and digital twins, characterized in that, Includes the following steps: Acquire image data of the steel bridge; The collected steel bridge image data is processed using the method described in claim 1 to obtain steel bridge crack data; A 3D model of steel bridge cracks was generated based on the crack data of the steel bridge. Based on the three-dimensional model of the crack in the steel bridge, a crack welding strategy is generated using the finite element analysis method. The welding process is executed according to the crack welding strategy, and the welding process information is dynamically mapped to the three-dimensional model of the steel bridge crack to form a three-dimensional dynamic model for steel bridge crack repair. Based on a three-dimensional dynamic model for steel bridge crack repair, the welding process is remotely monitored. The integrity of the repair of cracks in the steel bridge is detected, and the repair integrity information is updated in the three-dimensional dynamic model of the steel bridge crack repair.

5. A remote welding repair method for cracked steel bridge structures combining machine vision and digital twins according to claim 4, characterized in that, The steel bridge data is collected by an inspection robot, which is equipped with an industrial camera, an infrared thermal imager, a laser displacement sensor and a control unit. The industrial camera, infrared thermal imager and laser displacement sensor are all electrically connected to the control unit.

6. The method for remote welding repair of cracked steel bridge structures combining machine vision and digital twins according to claim 4, characterized in that, The crack data of the steel bridge includes the length, depth and stress concentration area of ​​the crack.

7. A remote welding repair method for cracked steel bridge structures combining machine vision and digital twins according to claim 6, characterized in that, The physical dimensions of the length and depth of the crack are determined by the following formula: in, The physical dimension representing the crack length. Let be the pixel length of the crack in the image. To calibrate the distance, The focal length of an image pixel. Physical dimensions representing crack depth. The pixel depth of the crack in the image.

8. A remote welding repair method for cracked steel bridge structures combining machine vision and digital twins according to claim 4, characterized in that, The crack welding strategy includes welding path, interpass temperature, and weld bead arrangement.

9. A remote welding repair method for cracked steel bridge structures combining machine vision and digital twins according to claim 4, characterized in that, The welding process is performed by a welding robot, which is equipped with a pulsed MIG welding torch, an ultrasonic flaw detector, an industrial camera, and a control unit. The pulsed MIG welding torch, the ultrasonic flaw detector, and the industrial camera are all electrically connected to the control unit.

10. A remote welding repair system for cracked steel bridge structures combining machine vision and digital twins, characterized in that, The method of any one of claims 4-9 comprises the following modules: The crack recognition module, installed on the inspection robot, is used to collect image data of the steel bridge to identify cracks in the steel bridge and transmit the crack data to the digital twin module. The digital twin module, installed on a remote server, is used to generate a three-dimensional model of the steel bridge crack based on the crack data, generate a crack welding strategy based on the three-dimensional model using the finite element analysis method, and then send the crack welding strategy to the crack welding execution module. The crack welding module is installed on the welding robot to perform welding actions according to the crack welding strategy and dynamically map the welding information onto the crack 3D model. The remote monitoring module, installed on a remote server, monitors the welding process of the welding robot through a VR operating interface based on a three-dimensional dynamic model of crack repair. The repair detection module, installed on the welding robot, detects the integrity of crack repair and updates the repair integrity information to the three-dimensional dynamic model of crack repair.

Images