Welding quality online detection and optimization method and system

Abstract

The application discloses a welding quality online detection and optimization method and system. The method comprises the following steps: performing laser scanning to obtain inertial navigation data of a mobile robot; performing motion compensation and multi-frame alignment on original point cloud data through the inertial navigation data; starting an image acquisition device to perform image acquisition; performing space alignment on visible light images and aligned point cloud data through the inertial navigation data; mapping visible light texture data obtained through aligned image sequences to a 3D point cloud model, and detecting welding quality through the 3D texture model; and adaptively adjusting welding parameters according to the welding quality detection result. The method effectively overcomes the image blurring and misplacement problems caused by trajectory deviation or environmental vibration in long-distance mobile detection in non-special sites, significantly improves the geometric precision and stability of point cloud data, and lays a reliable foundation for subsequent detection.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of visual inspection, in particular to a welding quality online detection and optimization method and system. BACKGROUND

[0002] With the development of technology, automatic detection means for welding quality are gradually applied in practical work.

[0003] Welding quality detection mainly includes two aspects, welding surface detection and welding internal detection. For welding surface detection, current surface detection mainly relies on non-contact laser displacement sensors and vision systems. The laser displacement sensor scans the welding position to construct a 3D image of the welding position, thereby realizing online detection of the welding seam on the welding surface.

[0004] For large special vehicles such as engineering vehicles and semi-trailers, due to the long length of the welding seam (up to tens of meters or even dozens of meters), the complex structure (such as including special-shaped curved surfaces and the like), and the large structural difference between different types of vehicles, the cost of laying a special detection site and detection mechanism is large.

[0005] Based on this, when detecting the appearance of large special vehicles, it is usually detected in a non-special detection site or even directly in a production site, and image acquisition is performed by a mobile robot or a manually held device, and then welding quality detection is performed.

[0006] However, the on-site detection environment is complex, and continuous mobile acquisition when facing long welds is prone to cause deviation of the equipment motion track, which can easily lead to poor image quality of the acquired welding position, and further affect the accuracy of the welding quality detection result. SUMMARY

[0007] To solve the above problems, the present application provides a welding quality online detection and optimization method, comprising:

[0008] starting a mobile robot to acquire images of the welding position of the special vehicle; wherein the mobile robot carries a laser scanning device, an image acquisition device, and an inertial navigation module;

[0009] performing laser scanning by the laser scanning device to obtain a laser image, and acquiring inertial navigation data of the mobile robot by the inertial navigation module;

[0010] generating original point cloud data based on the laser image, and performing motion compensation and multi-frame alignment on the original point cloud data based on the inertial navigation data to obtain aligned point cloud data;

[0011] Based on the aligned point cloud data, real-time feature extraction is performed. When the feature value of the extracted specified image feature exceeds the preset range or the mobile robot moves to the preset position, the image acquisition device is activated to acquire the image and obtain a visible light image.

[0012] The visible light image and the aligned point cloud data are spatially aligned using the inertial navigation data to obtain the aligned image sequence corresponding to the visible light image;

[0013] A 3D point cloud model is obtained by aligning the point cloud data, and the visible light texture data obtained by aligning the image sequence is mapped onto the 3D point cloud model to obtain a 3D texture model. The welding quality is then detected using the 3D texture model.

[0014] In one example, raw point cloud data is generated based on the laser image, and motion compensation and multi-frame alignment are performed on the raw point cloud data using the inertial navigation data to obtain aligned point cloud data, specifically including:

[0015] For each frame of the laser image, filtering and noise reduction are performed, and the Hessian matrix corresponding to each pixel is calculated to obtain the normal direction of the laser line;

[0016] A second-order Taylor expansion is performed along the normal direction to fit the extreme points of light intensity, thus obtaining the set of coordinates of the laser centerline.

[0017] Based on the principle of triangulation, the 2D coordinates of the centerline in the laser centerline coordinate set are converted into a 3D point cloud, and the inertial navigation data is used to apply a reverse motion transformation to each point in the 3D point cloud.

[0018] For each frame point cloud, a preset number of frame point clouds are combined into a first processing unit, and the minimum point-to-point distance is taken as the first objective function. The nearest point search is accelerated by KD tree and the local matching of sliding window is performed by point cloud normal vector constraints to obtain an aligned second processing unit. During the matching process, the relative pose is calculated by the inertial navigation data as the initial value in the first objective function, and abnormal matches are rejected by chi-square test.

[0019] In one example, when the feature value of the extracted specified image feature exceeds a preset range or the mobile robot moves to a preset position, the method further includes:

[0020] The grid is divided according to the top view of the second processing unit to obtain multiple sector grids, and feature encoding is performed on each sector grid;

[0021] Based on the feature encoding, the current second processing unit of the current frame and the historical second processing unit of the historical frame are matched for similarity. If the similarity exceeds a preset level and the pose distance is less than a preset distance, point cloud reconstruction is triggered.

[0022] Extract the region features corresponding to the current second processing unit, and perform feature enhancement based on the feature type of the region features;

[0023] A pose graph is constructed based on the current second processing unit; wherein, in the pose graph, the center position of the current second processing unit and the inertial navigation data of the current frame are defined as nodes, and multiple types of constraint edges are established based on the positional relationship between the inertial navigation data of the current frame and the inertial navigation data of historical frames, and confidence weights are assigned to different types of constraint edges.

[0024] The weighted deviation is obtained by summing the weighted deviations of the constraint edges according to the confidence weights corresponding to each type of constraint edge. The minimized reinforcement deviation is used as the second objective function, and the second objective function is solved iteratively to obtain the optimized pose parameters.

[0025] Based on the optimized pose parameters, the aligned point cloud data is reconstructed into a point cloud.

[0026] In one example, the visible light image and the aligned point cloud data are spatially aligned using the inertial navigation data to obtain an aligned image sequence corresponding to the visible light image, specifically including:

[0027] The inertial navigation data and the visible light image are preprocessed and synchronized.

[0028] For each frame image in the visible light image, its corresponding visual features are extracted, and the first displacement vector of the adjacent frame image is predicted based on the optical flow algorithm.

[0029] Based on the inertial navigation data, the second displacement vector between adjacent frame images is predicted, and the state of the second displacement vector is updated by Kalman filtering.

[0030] Based on the error value corresponding to the first displacement vector and the error value of the second displacement vector after the state update, a third objective function that minimizes the error value is generated to optimize the prediction of the first displacement vector and the second displacement vector.

[0031] Based on the optimized first and second displacement vectors, an affine transformation is performed on the visible light image to obtain the corresponding aligned image sequence.

[0032] In one example, the visible light texture data obtained through the aligned image sequence is mapped onto the 3D point cloud model to obtain the 3D texture model, specifically including:

[0033] By extrinsic parameter calibration, a coordinate mapping relationship between the 3D point cloud model and the aligned image sequence is established, and a spatiotemporal reference is unified based on the coordinate mapping relationship;

[0034] The 3D point cloud model is triangulated to generate a mesh model containing vertices, edges, and faces, and then globally parameterized based on the fourth objective function corresponding to minimizing angular distortion.

[0035] Based on the transformation of camera intrinsic and extrinsic parameters, single-view projection mapping is performed to convert the 3D point cloud model into a 2D pixel model.

[0036] For each candidate viewpoint, a visibility score is calculated based on the proportion of visible triangles according to its corresponding 2D pixel model, and a quality score is calculated based on viewpoint positivity and anti-blurring.

[0037] The visibility score, the quality score, and the structural similarity scores between the current candidate viewpoint and the adjacent candidate viewpoints are weighted and fused to obtain a multi-view fusion result, which serves as a 3D texture model.

[0038] In one example, when the feature value of the extracted specified image feature exceeds a preset range or the mobile robot moves to a preset position, the method further includes:

[0039] A depth map is generated by scanning along the current scanning position of the laser scanning device, and the depth gradient is calculated by the Sobel operator. Regions with depth gradients higher than preset gradients are marked as height jump regions.

[0040] For the height jump region, determine its corresponding 3D point cloud region and corresponding inertial navigation data;

[0041] For each point in the 3D point cloud region, frontal orientation and visual occlusion are determined to obtain the visibility of the visual cone corresponding to that point.

[0042] Based on the visibility of the visual cone of all points, the point cloud occlusion rate corresponding to the 3D point cloud region is obtained, and when the point cloud occlusion rate is higher than the preset occlusion rate, the missed detection region determination is initiated.

[0043] By using a pre-trained missed detection probability prediction model, the 3D point cloud region and the inertial navigation data are taken as input, and the output is a voxel of the missed detection region probability.

[0044] A probability heatmap is obtained based on the probability voxels of the missed detection areas, and the corresponding high-risk missed detection areas are obtained based on the probability heatmap.

[0045] For the high-risk missed detection areas, a dynamic path plan is generated using the probability heatmap, so that the mobile robot can perform laser scanning on the high-risk missed detection areas according to the dynamic path plan.

[0046] In one example, for the high-risk missed detection area, dynamic path planning is generated using the probability heatmap, specifically including:

[0047] For the high-risk missed detection area, a global cost function is generated through the probability heatmap, and a motion planning algorithm is generated sequentially based on the cost value of each target point in the high-risk missed detection area in the global cost function.

[0048] Based on the motion planning algorithm, a search tree is initialized; in fact, the root node of the search tree is the current position of the laser scanning device.

[0049] Risk-guided sampling is performed based on the probability heatmap to generate sampling points, and nearest neighbor node selection is performed to determine the tree node closest to the sampling point;

[0050] Based on the distance constraints between the tree nodes and the obstacles, new nodes are expanded according to the tree nodes.

[0051] For the neighboring nodes corresponding to the new node, calculate the corresponding new path cost, and when the new path cost is lower than the original path cost corresponding to the new node, update the parent node of the neighboring node corresponding to the new path cost to the new node;

[0052] The new node is determined to cover any target point in the high-risk missed detection area.

[0053] In one example, the mobile robot performs laser scanning on the high-risk missed detection area according to the dynamic path planning, specifically including:

[0054] The scanning speed of the mobile robot is determined based on the path curvature radius corresponding to the dynamic path planning; the scanning speed is positively correlated with the path curvature radius.

[0055] Based on the missed detection probability of the target point in the probability heatmap, the laser power of the laser scanning device is determined; the laser power is positively correlated with the missed detection probability.

[0056] Based on the regional risk level of the 3D point cloud region, the scanning density of the laser scanning device is determined; the regional risk level is obtained based on at least one of the point cloud occlusion rate, the feature value of the specified image feature, and the depth gradient, and the scanning density is positively correlated with the regional risk level.

[0057] In one example, welding parameters are adaptively adjusted based on welding quality inspection results, specifically including:

[0058] Based on the welding quality inspection results, the corresponding defect types are determined, and the welding quality indicators are quantified; the welding quality indicators include geometric indicators and texture indicators.

[0059] Based on the defect type, the corresponding welding parameters are matched using a preset defect-parameter mapping rule library;

[0060] The welding parameters are adaptively adjusted based on the difference between the quantified welding quality indicators and the preset standard indicators.

[0061] On the other hand, this application also proposes an online welding quality detection and optimization system, the system comprising:

[0062] A mobile robot, which carries a laser scanning device, an image acquisition device, an inertial navigation module, and a communication module;

[0063] The cloud-based system communicates with the mobile robot via the communication module to execute the online welding quality detection and optimization method as described in any of the above examples.

[0064] The online welding quality detection and optimization method proposed in this application can bring the following beneficial effects:

[0065] 1. By integrating an inertial navigation module, real-time device motion data is acquired, and motion compensation and multi-frame alignment are performed on the raw point cloud generated by laser scanning. This effectively overcomes the problems of image blurring and misalignment caused by trajectory deviation or environmental vibration in non-dedicated sites and long-distance mobile detection, significantly improving the geometric accuracy and stability of point cloud data, and laying a reliable foundation for subsequent detection.

[0066] 2. Real-time feature extraction is performed based on aligned point cloud data, triggering high-resolution visible light image acquisition only when feature values ​​are abnormal (indicating potential defects) or when a critical preset position is reached. This on-demand triggering mechanism avoids the resource waste of indiscriminate, continuous image acquisition of the entire long weld seam in traditional solutions, optimizing data acquisition efficiency and storage and computing resource consumption.

[0067] 3. Utilizing the compensated and aligned point cloud data and inertial navigation information, sparsely triggered visible light images are precisely spatially aligned onto the point cloud model and fused to generate a 3D model with realistic textures. This not only optimizes the completeness of the detection information, seamlessly combining surface textures (visual information such as color and cracks) with precise 3D geometry, but also improves the visualization and positioning accuracy of welding surface defects (such as undercut, weld beads, and surface cracks), making the detection results more intuitive and accurate.

[0068] 4. Welding quality assessment based on a 3D texture model that integrates accurate geometric information and realistic surface texture is more comprehensive and reliable than relying solely on potentially inaccurate point clouds or independently acquired 2D images. This model effectively reproduces the true state of the weld surface, optimizes the criteria for defect identification and judgment, and thus improves the overall accuracy and reliability of online inspection results. Attached Figure Description

[0069] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0070] Figure 1 This is a flowchart illustrating the online welding quality detection and optimization method in the embodiments of this application;

[0071] Figure 2 This is a schematic diagram of the point cloud reconstruction process under one scenario in an embodiment of this application;

[0072] Figure 3 This is a schematic diagram of the visible light texture mapping process in one scenario of this application embodiment;

[0073] Figure 4 This is a schematic diagram of the identification process of the occluded area under one scenario in an embodiment of this application;

[0074] Figure 5 This is a schematic diagram illustrating the process of determining whether a point is visible in one scenario according to an embodiment of this application.

[0075] Figure 6 This is a schematic diagram of the online welding quality detection and optimization equipment in the embodiments of this application. Detailed Implementation

[0076] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0077] The technical solutions provided by the various embodiments of this application are described in detail below with reference to the accompanying drawings.

[0078] like Figure 1 As shown in the embodiments of this application, an online welding quality detection and optimization method is provided, including:

[0079] S101: Activate the mobile robot to acquire images of the welding parts of the special vehicle; wherein the mobile robot carries a laser scanning device, an image acquisition device, and an inertial navigation module.

[0080] A mobile robot is an intelligent robot capable of moving autonomously or semi-autonomously in different environments and completing specific tasks. In some special environments (such as those with uneven terrain or complex obstacles), human intervention can be used instead of mobile robots, with personnel carrying the necessary equipment for image acquisition.

[0081] Special-purpose vehicles refer to vehicles used specifically in special scenarios, including engineering vehicles, semi-trailers, and other vehicles. After the corresponding structure of the special-purpose vehicle (such as the body structure, chassis structure, etc.) is welded, a period of cooling is allowed to allow the welded structure to stabilize. Then, the mobile robot is started under the control of the user (i.e., the on-site inspection personnel).

[0082] Laser scanning equipment is a device that acquires information such as the three-dimensional spatial coordinates, distance, and shape of a target object by emitting laser pulses and measuring their reflection time or phase difference. It includes a lidar system capable of acquiring 3D point cloud data of the target object. Image acquisition equipment is a device that captures two-dimensional or three-dimensional visual information of the environment through optical lenses and image sensors, capable of acquiring visible light images of target objects. An inertial navigation module is a device that calculates the position, velocity, and attitude of an object by measuring inertial quantities such as acceleration, angular velocity, and magnetic field. It can acquire inertial navigation data such as acceleration, angular velocity, and magnetic field of a mobile robot itself.

[0083] By default, when a mobile robot is first started, it only needs to activate the laser scanning device and the inertial navigation module. When it encounters the expected scene, it then activates the image acquisition device.

[0084] The welding area refers to the vehicle structure of a specialized vehicle containing the welding zone. When a mobile robot collects data, the area containing the welding area can be pre-defined, specifying a collection range. Based on this collection range, the movement paths of the mobile robot, as well as the laser scanning and image acquisition devices, are set, enabling them to perform a complete scan of the welding area. Specifically, the laser scanning and image acquisition devices can be carried on a robotic arm, allowing the movement paths of these devices to be set.

[0085] In actual data processing, data processing is usually carried out through the cloud, or some basic data processing can be performed through mobile robots.

[0086] S102: A laser image is obtained by performing a laser scan using the laser scanning device, and the inertial navigation data of the mobile robot is acquired through the inertial navigation module.

[0087] During its movement, the mobile robot acquires laser images using a laser scanning device, facilitating the analysis and recognition of these images to generate corresponding point cloud data. Simultaneously, the inertial navigation module records the robot's inertial navigation data, including acceleration, angular velocity, and magnetic field.

[0088] Of course, if a mobile robot is equipped with a robotic arm to change the position of a laser scanning device or image acquisition device, the common practice is to install multiple inertial navigation modules. During the movement of the mobile robot, the robotic arm does not move, allowing the basic inertial navigation modules to acquire the robot's inertial navigation data. When the robotic arm needs to move, the mobile robot stops moving. At this time, the robotic arm is controlled to move upwards, downwards, etc., and its inertial navigation data is acquired by the inertial navigation module monitoring the robotic arm. Since only one object is moving at any given time, the robotic arm's inertial navigation data can be directly used as the inertial navigation data for the laser scanning device or image acquisition device; it can also be called the inertial navigation data of the mobile robot's internal components during movement. For ease of description, this will be consistently referred to as the mobile robot's inertial navigation data.

[0089] S103: Generate raw point cloud data based on the laser image, and perform motion compensation and multi-frame alignment on the raw point cloud data using the inertial navigation data to obtain aligned point cloud data.

[0090] Specifically, the laser image obtained by the laser scanning device can be regarded as a stripe image (the typical acquisition frequency is 50 to 100 frames per second). Taking a six-axis inertial measurement module as an example, the inertial navigation data includes the real-time output of the X-axis, Y-axis, and Z-axis accelerations and the angular velocities corresponding to pitch, roll, and yaw. The sampling frequency is usually over 200Hz.

[0091] Laser stripe centers need to be extracted for each frame of the laser image. Then, filtering and noise reduction are performed on each frame, and the Hessian matrix corresponding to each pixel is calculated to obtain the normal direction of the laser line. Gaussian filtering can be used for noise reduction, while the Hessian matrix, also known as the Hessian matrix, is composed of the second-order partial derivatives of a multivariate function. In this application, the laser line is treated as a function, and the local curvature of the laser line is described by the Hessian matrix, thereby obtaining the normal direction of the laser line.

[0092] A second-order Taylor expansion is performed along the normal direction to fit the extreme points of light intensity, thus obtaining the set of coordinates for the laser centerline. By fitting the coordinates, the extreme positions of the light intensity distribution (usually the laser centerline) are found along the normal direction of the laser line in the image, thereby achieving sub-pixel-level positioning accuracy. The coordinates of each laser centerline are then used to generate a set, forming the laser centerline coordinate set.

[0093] Based on the principle of triangulation, the 2D coordinates of the centerline in the laser centerline coordinate set are converted into a 3D point cloud. During this conversion, the extrinsic parameters of the laser scanning equipment need to be pre-calibrated, meaning that the relative position and orientation relationship between the laser scanning equipment and the image acquisition equipment must be consistent.

[0094] When transforming coordinates, it can be done as shown in Formula 1:

[0095]

[0096] Where x, y, and z represent the 2D coordinates of the laser centerline, u and v represent the 3D point cloud coordinates, and θ represents the laser plane angle.

[0097] Inverse motion transformation is applied to each point in the 3D point cloud using inertial navigation data. Based on the acceleration and angular velocity in the inertial navigation data, the instantaneous displacement and rotation of the device are calculated, thereby applying an inverse motion transformation to each 3D point. Through compensation, the jitter error of the single-frame point cloud is reduced. The inverse motion transformation can be represented by Equation 2:

[0098] P corrected =R(-Δφ)*(P raw -Δd) Formula 2;

[0099] Among them, P corrected P represents the coordinates of the 3D point cloud after correction by applying an inverse motion transformation.raw The coordinates of the 3D point cloud before applying the reverse motion transformation are given, Δd is the translation error vector of the laser scanning device, Δφ is the rotation error angle of the laser scanning device, and R(-Δφ) is the reverse rotation matrix, which is used to invert the rotation angle to cancel the rotation error in the original point cloud.

[0100] This achieves point cloud generation and distortion compensation. Next, multi-frame point cloud pre-alignment is performed to construct locally consistent segments.

[0101] For each frame of point cloud, a preset number of frame point clouds are grouped into a processing unit, for example, five consecutive frame point clouds are grouped into a processing unit.

[0102] Minimizing the point-to-point distance is taken as the first objective function. KD-tree is used to accelerate the nearest point search and point cloud normal vector constraints are used for local matching of sliding windows to obtain aligned processing units.

[0103] The first objective function can be expressed as shown in Formula 3:

[0104]

[0105] Where R is a 3×3 orthogonal rotation matrix, which satisfies R T R = I, t is a 3×1 translation vector describing the translational offset of the source point cloud before pre-alignment relative to the target point cloud after pre-alignment, P i Let Q be the i-th point of the source point cloud. i Let be the i-th point in the target point cloud, and n be the number of point pairs.

[0106] In addition, during the matching process, the relative pose is calculated using inertial navigation data as the initial value in the first objective function, and abnormal matches are rejected using a chi-square test.

[0107] By calculating the relative pose using inertial navigation data and using this relative pose as the initial value for the rotation matrix R and translation vector t in the first objective function, the initial value can be made closer to the true solution, thereby reducing the number of iterations. Meanwhile, mismatches (e.g., incorrect point pairs caused by noise points or repetitive structures) can cause non-physical jumps in the pose calculation. The chi-square test, which calculates whether the residuals are within the confidence interval using a statistical model, can filter out incorrect matches, reject abnormal point pairs with residuals exceeding the threshold, prevent them from polluting the optimization process, ensure motion continuity, ensure that the pose calculation conforms to physical laws, and improve the stability of the registration results.

[0108] At this point, the final output can be a locally aligned point cloud segment (5 frames from each processing unit are fused into one point cloud segment).

[0109] S104: Real-time feature extraction is performed based on the aligned point cloud data. When the feature value of the extracted specified image feature exceeds a preset range or the mobile robot moves to a preset position, the image acquisition device is activated to acquire an image and obtain a visible light image.

[0110] The specified image feature can be curvature. When the curvature exceeds a preset range, for example, when the curvature is greater than 0.15mm... - 1. If a sudden change in curvature is observed, this usually indicates a geometrically complex region. In this case, the image acquisition device is activated to acquire images.

[0111] Meanwhile, it can be pre-programmed to indicate that a mobile robot has reached a complex area when it moves to a designated location. This designated location can be manually set based on expert experience and the actual situation on site.

[0112] When the feature values ​​of the extracted specified image features are restored to the preset range, and the mobile robot moves to the preset stop position, the image acquisition device can be stopped from acquiring images.

[0113] Laser scanning offers advantages such as high geometric accuracy and independence from surface texture, making it suitable for inspecting the welding quality of conventional components. Furthermore, laser scanning provides strong real-time performance, supporting rapid dynamic object reconstruction and quickly determining whether the feature values ​​of specified image features exceed preset ranges, thereby deciding whether to activate the image acquisition device.

[0114] However, laser scanning struggles to capture surface texture information. Therefore, when the feature values ​​of extracted image features exceed a preset range or the mobile robot reaches a preset position, indicating the entry into a complex welding area, laser scanning alone may not yield accurate results. Consequently, an image acquisition device is activated. Visible light images acquired by this device possess rich texture information, preserving visual features such as weld surface color, oxide spots, and rust, thus increasing the accuracy of weld quality inspection. While laser scanning has limitations in geometric precision and higher computational complexity, supplementing this limitation with laser scanning can mitigate the computational complexity issue. Furthermore, acquiring images only from specific areas also addresses the problem of high computational complexity.

[0115] Furthermore, when complex regions are encountered, the already acquired aligned point cloud data may suffer from poor acquisition or alignment results due to the complex structure of the region. Therefore, in addition to acquiring visible light images using image acquisition equipment, it can be determined, based on requirements, whether point cloud reconstruction of the already acquired aligned point cloud data is necessary.

[0116] like Figure 2As shown, the first step is to determine whether point cloud reconstruction is necessary.

[0117] Specifically, the grid is divided based on the top view of the second processing unit to obtain multiple sector grids, and feature encoding is performed on each sector grid. At this point, the second processing unit is used instead of judging each frame of point cloud data. Although this reduces accuracy to some extent, the reduced accuracy still meets the requirements, and it also reduces the computational load required during real-time judgment.

[0118] In practical partitioning, the top view of the second processing unit can be divided into a 20×60 sector grid, resulting in a radial resolution of 5 cm and an angular resolution of 6°. Feature encoding can utilize a fast point feature histogram to obtain a 128-dimensional feature vector.

[0119] Based on feature encoding, the current second processing unit of the current frame is matched with the historical second processing unit of the historical frame. If the similarity exceeds a preset level and the pose distance is less than a preset distance, point cloud reconstruction is triggered.

[0120] The formula for similarity matching can be shown in Formula 4:

[0121]

[0122] in, This is the current second processing unit for the current frame. The historical second processing unit is a historical frame. Both the current second processing unit and the historical second processing unit contain k sector regions. In this application, 60 sector regions are used as an example for explanation. ⊙ represents the cosine similarity calculation.

[0123] If the similarity exceeds a preset level, for example, greater than 0.75, and the pose distance is less than a preset distance, for example, less than 0.5m, the current frame is considered to be excessively similar to historical frames, and point cloud reconstruction is required.

[0124] At this point, the regional features corresponding to the current second processing unit are extracted, and feature enhancement is performed based on the feature type of the regional features.

[0125] The feature types of regional features can include two types: geometric feature points and process feature points. Geometric feature points are determined based on specified image features (such as curvature), while process feature points are determined by manually marked preset positions.

[0126] For geometric feature points, when the specified image feature is curvature, curvature analysis can be used to extract stable feature points in areas of geometric abrupt change, such as weld intersections and plate corners. When the curvature of a local surface exceeds a preset range, it is automatically marked as a critical point. However, process feature points need to be manually pre-marked according to the process.

[0127] For geometric feature points, feature enhancement has already been performed above, using a fast point feature histogram to obtain a 128-dimensional feature vector. For process feature points, a rotated projection histogram can be generated to obtain a 352-dimensional feature vector for feature enhancement. Furthermore, an adaptive neighborhood radius can be used, employing a 5mm radius in flat areas and extending to 10mm in complex curved surfaces.

[0128] A pose graph is constructed based on the current second processing unit. The pose graph, short for position and attitude graph, is mainly generated from inertial navigation data and is used to describe the pose relationship between the current second processing unit and the historical second processing units, as well as to construct the corresponding constraint network.

[0129] In the pose graph, the center position of the current second processing unit and the inertial navigation data of the current frame are defined as nodes. The nodes contain three-dimensional coordinates and rotation angle information, which can also be called center pose.

[0130] Multiple types of constraint edges are established based on the positional relationship between the inertial navigation data of the current frame and the inertial navigation data of historical frames, and confidence weights are assigned to different types of constraint edges.

[0131] There are three types of constraint edges: continuous edges, closed-loop edges, and inertial edges. Continuous edges are generated by calculating the relative positional relationship between the second processing units of adjacent frames using a registration algorithm. Closed-loop edges establish high-strength constraints (e.g., a 5-fold increase in weight) between the historical and current second units when the registration algorithm successfully matches feature points. Inertial edges are generated by fusing inertial navigation data to create motion trajectory constraints, with their weight increased when vibrations are strong.

[0132] At this point, the weighted deviation is obtained by summing the weighted deviations of the constraint edges according to the confidence weights corresponding to each type of constraint edge. The minimized reinforcement deviation is taken as the second objective function, and the second objective function is solved iteratively to obtain the optimized pose parameters.

[0133] The second objective function can be represented as shown in Formula 5:

[0134]

[0135] Where X is the pose parameter, i and j represent nodes i and j respectively, representing the second processing unit in different frames, (i,j)∈ε represents traversing all constraint edges, and T i T j T represents the pose transformation matrices for nodes i and j, respectively, describing the position and orientation of the mobile robot in the global coordinate system. ij Let be the observed pose transformation from node i to node j, and log represents mapping the error of the transformation matrix to the Lie algebra space. Z represents the Lie algebra error term, used to minimize the geometric error of edges in the pose graph, ensuring consistency between the estimated and observed poses. The norm square of this error vector represents the geometric constraint residual; imu For inertial navigation data, h(X) is the inertial prediction value calculated based on the pose parameter X, Ω imu The weight matrix for inertial constraints is typically a diagonal matrix. λ represents the inertial constraint term, used to improve the robustness of pose estimation, and λ is the inertial navigation weight coefficient.

[0136] At this point, the pose graph is transformed into a large-scale optimization problem, with the goal of minimizing the weighted bias of three types of constraints, including the positional bias of the second processing unit in adjacent frames, the spatial distance bias of closed-loop feature points, and the continuity bias of the inertial navigation trajectory.

[0137] The iterative process can be solved using the damped least squares method in multiple iterations. In each iteration, the constraint deviation under the current pose configuration can be calculated, the pose adjustment amount can be solved by matrix operations, and the step size can be dynamically adjusted to ensure stable convergence until the preset convergence condition is reached. The process terminates when the overall deviation change is less than one ten-thousandth or when the number of iterations reaches one hundred.

[0138] At this point, it is assumed that the optimized pose parameters after iteration meet expectations, and point cloud reconstruction is performed on the aligned point cloud data based on the optimized pose change.

[0139] First, the deformation field is calculated. Based on the optimized pose parameters and the pose change compared to the original pose parameters, a spatial deformation field is constructed. Rigid transformation is applied to local small deformation areas, and flexible spline transformation is applied to thermal deformation areas.

[0140] The original aligned point cloud data is reconstructed into a point cloud, and displacement is performed according to the deformation field. For complex areas, high-precision reconstruction can be used, while other areas can be smoothed.

[0141] After the point cloud reconstruction is completed, quality verification can be performed based on requirements to test the integrity of the reconstructed point cloud.

[0142] Of course, such as Figure 2 As shown, if point cloud reconstruction is not required, incremental stitching of the point cloud can continue, and scanning can continue.

[0143] S105: Spatially align the visible light image with the aligned point cloud data using the inertial navigation data to obtain the aligned image sequence corresponding to the visible light image.

[0144] For reasons similar to those for laser images, visible light images also need to be aligned to ensure accurate detection and treatment.

[0145] Specifically, inertial navigation data and visible light images are preprocessed and synchronized. Preprocessing of the inertial navigation data includes low-pass filtering to eliminate high-frequency mechanical vibration noise. Preprocessing of the visible light images can include grayscale conversion, geometric transformation, and image enhancement. Data synchronization primarily refers to hardware clock synchronization, ensuring that the time deviation between the two is below the expected value and aligning the timestamps.

[0146] For each frame of a visible light image, its corresponding visual features are extracted, and the first displacement vector of the adjacent frame image is predicted based on the optical flow algorithm.

[0147] Visual features can be Oriented Fast and Rotated Brief (ORB) features. The extraction process involves FAST corner detection to identify points of abrupt changes in intensity within the weld area—points with response values ​​exceeding a threshold. Feature orientation is calculated using the centroid method for orientation assignment. The rotated BRIEF descriptor is then used as the descriptor for the generated 256-bit binary string. BRIEF (Binary Robust Independent Elementary Features) is a binary feature descriptor; combining it with FAST corner points yields ORB features, which serve as the visual features.

[0148] The optical flow algorithm can be the LK optical flow algorithm, which can meticulously track visual features between adjacent frames and predict the displacement vectors of adjacent frame images. For ease of description, this will be referred to as the first displacement vector.

[0149] Based on inertial navigation data, the second displacement vector between adjacent frames is predicted, and the state of the second displacement vector is updated using Kalman filtering. Using the acceleration and angular velocity in the inertial navigation data, the instantaneous displacement and rotation angle between adjacent frames can be calculated using the Newton-Euler equations, thus obtaining the predicted second displacement vector.

[0150] Simultaneously, the state is updated using Kalman filtering, where the state vector includes three state vectors: position, velocity, and attitude angle. After predicting the second displacement vector, the prediction covariance matrix is ​​calculated, and the visual features of the tracked adjacent frames are used as observations to correct the error of the second displacement vector.

[0151] At this point, based on the error value corresponding to the first displacement vector and the error value of the second displacement vector after the state update, a third objective function that minimizes the error value is generated to optimize the prediction of the first and second displacement vectors. The third objective function can be shown in Formula 6:

[0152] min T ∑‖Z visual -π(T*P)‖ 2 +λ‖Z imu -T imu || 2 Formula Six;

[0153] Where T is the pose transformation matrix, describing the pose parameters of the mobile robot in the global coordinate system, and Z is the optimization variable. visual Let P be the observed value of the first displacement vector, P be the 3D point cloud coordinates, π be the camera projection model, and ∑‖Z visual -π(T*P)‖ 2 Z is a visual constraint term used to minimize the error in projecting 3D points onto the image, ensuring spatial consistency between visual features and the 3D point cloud; imu For inertial navigation data, T imu Let λ be the predicted value of the second displacement vector calculated from the pose transformation matrix, where λ is the inertial navigation weight coefficient, and λ‖Z imu -T imu || 2 As an inertial navigation constraint term, it incorporates real-time motion constraints of inertial navigation to maintain positioning and reduce drift when visual information is missing.

[0154] In addition, when the inertial navigation system detects a sudden high-frequency vibration, it can automatically increase the weight of the visual constraint term to prevent drift caused by inertial data dominance.

[0155] Based on the optimized first and second displacement vectors, an affine transformation is performed on the visible light image to obtain the corresponding aligned image sequence.

[0156] During the radiometric transformation, the image's pose parameters are extracted, including displacements in the x and y directions to correct positional deviations, and the image's rotation angle to correct tilt or rotation errors. Coordinate transformation is then performed using Equation 7:

[0157]

[0158] Where f is the focal length and d is the pixel size. θ is the scale factor, θ is the rotation angle, and x and y are the displacements in the x and y directions, respectively.

[0159] During the transformation process, the target image grid is first created, then the coordinates of the original image are calculated by inverse transformation, and finally the pixel values ​​are obtained by bicubic interpolation, thus completing the affine transformation.

[0160] At the same time, based on the requirements, corresponding tests can be carried out on the existing static calibration board.

[0161] S106: Obtain a 3D point cloud model through the aligned point cloud data, and map the visible light texture data obtained through the aligned image sequence onto the 3D point cloud model to obtain a 3D texture model, and use the 3D texture model to detect the welding quality.

[0162] like Figure 3 As shown, after obtaining the aligned point cloud data, a 3D point cloud model can be directly generated from it. For the aligned image sequence obtained through visible light images, the corresponding visible light texture data can be mapped onto the 3D point cloud model to obtain a textured 3D point cloud model, which serves as a 3D texture model.

[0163] Specifically, through extrinsic parameter calibration, a coordinate mapping relationship is established between the 3D point cloud model and the aligned image sequence, and spatiotemporal reference unification is performed based on this coordinate mapping relationship. During extrinsic parameter calibration, the transformation matrix between the laser scanning device and the image acquisition device can be calculated using a checkerboard calibration board, thereby obtaining the coordinate mapping relationship and unifying the spatial reference. Simultaneously, through hardware clock synchronization, it is ensured that the time deviation between the two is lower than the expected value, and timestamp alignment is performed, which is equivalent to unifying the time reference.

[0164] The 3D point cloud model is triangulated to generate a mesh model containing vertices, edges, and faces, and then globally parameterized based on the fourth objective function corresponding to minimizing angular distortion.

[0165] By using Delaunay triangulation, the 3D point cloud model is triangulated to ensure that the side length is less than a certain value, control the mesh density, retain details and avoid excessive density, thereby generating a mesh model containing vertices, edges and faces. This can convert disordered point clouds into ordered meshes, laying the foundation for subsequent parameterization.

[0166] The purpose of global parameterization is to map the 3D surface to the 2D texture plane (UV space), minimize angular distortion, and preserve geometric features. At this point, the fourth objective function can be expressed as shown in Equation 8:

[0167]

[0168] Among them, u i u j k represents the coordinates of the vertex in UV space, which is the parameterized result. ij Edge weights are used to reflect the properties of 3D mesh edges, control the degree of mapping stretching, and ensure reasonable local proportions. ij This refers to the degree of angular distortion.

[0169] Based on camera intrinsic and extrinsic parameter transformations, single-view projection mapping is performed to convert the 3D point cloud model into a 2D pixel model. The perspective projection model can be represented by Equation Nine:

[0170]

[0171] Where K is the camera intrinsic parameter, including focal length and principal point, used to define the image coordinate system, and T... C→W T W→S For extrinsic parameter transformation, spatiotemporal reference unification is used to ensure spatiotemporal consistency between 3D point clouds and 2D pixel images. u and v are the coordinates in the 2D pixel image, and x, y, and z are the coordinates in the 3D point cloud.

[0172] At this point, the mesh triangles are traversed, projected, and filled into the visible area. At the same time, it is important to ensure that the angle between the normal vector and the line of sight is less than a certain value to avoid back faces. Bilinear interpolation is used to smooth the texture and improve visual quality.

[0173] However, using only single-view projection mapping results can easily lead to texture loss, distortion amplification, occlusion blind spots, etc. In this case, multi-view projection mapping results can be used for fusion processing.

[0174] For each candidate viewpoint, a visibility score is calculated based on the proportion of visible triangles according to its corresponding 2D pixel model, and a quality score is obtained through viewpoint positivity and anti-blurring calculations.

[0175] For the visibility score, the percentage of visible triangles under each candidate viewpoint is calculated, which is the percentage of unobstructed faces relative to the total number of faces. This measures the viewpoint's ability to cover the target area. The higher the visibility score, the more effective the viewpoint.

[0176] The quality score comprises two parts. For viewpoint positivity, a smaller angle between the surface normal and the line of sight indicates a more positive viewpoint and a higher quality score. A positive viewpoint is generally preferred to reduce texture stretching and improve sharpness. For anti-blurring, lower motion blur indicates the mobile robot is currently stationary or in slow motion, resulting in a higher quality score and preventing blurred textures from affecting the fusion process.

[0177] The multi-view fusion result is obtained by weighting and fusing visibility score, quality score, and structural similarity score between the current candidate view and adjacent candidate views. Structural similarity measures the consistency of texture structure (including color, edge distribution, etc.) between the current view and adjacent views, avoiding color difference and splicing marks during fusion. The higher the structural similarity value, the better the continuity.

[0178] At this point, the weighted fusion formula is as shown in Formula 10:

[0179]

[0180] Among them, I final For the texture pixels corresponding to the multi-view fusion result, I kFor the k-th single-view projection mapping, w k w represents the rating weight corresponding to the k-th single-view projection mapping. k =S vis *S qual *SSIM k S vis S represents the visibility score. qual For quality score, SSIM k Score the structural similarity.

[0181] At this point, the multi-view fusion result can be directly used as a 3D texture model, or the multi-view fusion result can be post-processed, such as extracting SIFT feature points at the UV boundaries for forced matching and performing color correction to eliminate seams.

[0182] S107: Based on the welding quality inspection results, the welding parameters are adaptively adjusted.

[0183] Specifically, the generated 3D texture model can identify defects in welding quality, thereby identifying the corresponding defect types. For example, defect types can include porosity, undercut, lack of fusion, cracks, etc.

[0184] Simultaneously, welding quality indicators are quantified; these indicators include geometric and textural indicators. For example, quantified geometric indicators may include depression depth, protrusion height, weld width deviation, and continuous defect length, while quantified textural indicators may include oxide spot area, surface roughness, and crack density.

[0185] At the same time, corresponding defect level standards are established, which can be based on relevant international, national, or industry standards. This allows for the determination of the existence of a specific defect type and its severity based on the quantified values ​​of each indicator.

[0186] Based on the defect type, the corresponding welding parameters are matched through a preset defect-parameter mapping rule library; the welding parameters are adaptively adjusted according to the difference between the quantified welding quality index and the preset standard index.

[0187] The system includes a pre-set defect-parameter mapping rule base, which records which defects are associated with which welding parameters. The difference between the welding quality index and the preset standard index indicates the severity of the defect. This allows for the identification of matching welding parameters based on the severity level, and by adjusting these parameters, adaptive adjustment of the welding parameters can be achieved.

[0188] For example, when the defect type is porosity, the associated parameter is shielding gas flow rate. In this case, the shielding gas flow rate needs to be increased, and the increase in shielding gas flow rate increases linearly with the porosity density. When the defect type is undercut, the associated parameter is welding current. In this case, the welding current needs to be reduced, and the reduction in welding current is positively correlated with the undercut depth (for example, setting a corresponding coefficient and multiplying it by the undercut depth to obtain the reduction value; the default value of this coefficient can be 0.8). When the defect type is lack of fusion, the associated parameters are welding speed and voltage. In this case, the welding speed needs to be reduced and / or the voltage needs to be increased. The reduction in speed and the increase in voltage can be set to fixed values, for example, reducing the speed by 10% and increasing the voltage by 5% each time. When the defect type is surface crack, the associated parameter is preheating temperature. In this case, the preheating temperature needs to be increased, and the increase in preheating temperature is positively correlated with the length of the surface crack (for example, setting a corresponding coefficient and multiplying it by the length of the surface crack to obtain the increase value; the default value of this coefficient can be 50℃ / cm).

[0189] Furthermore, if multiple defect types are detected simultaneously, the more severe defect types can be addressed first. Additionally, an upper limit can be set for the adaptive adjustment of welding parameters to ensure the adjustment process remains within a safe range. Of course, welding parameter adjustments can also be made manually based on requirements to ensure the rationality of the adjustments.

[0190] 1. By integrating an inertial navigation module, real-time device motion data is acquired, and motion compensation and multi-frame alignment are performed on the raw point cloud generated by laser scanning. This effectively overcomes the problems of image blurring and misalignment caused by trajectory deviation or environmental vibration in non-dedicated sites and long-distance mobile detection, significantly improving the geometric accuracy and stability of point cloud data, and laying a reliable foundation for subsequent detection.

[0191] 2. Real-time feature extraction is performed based on aligned point cloud data, triggering high-resolution visible light image acquisition only when feature values ​​are abnormal (indicating potential defects) or when a critical preset position is reached. This on-demand triggering mechanism avoids the resource waste of indiscriminate, continuous image acquisition of the entire long weld seam in traditional solutions, optimizing data acquisition efficiency and storage and computing resource consumption.

[0192] 3. Utilizing the compensated and aligned point cloud data and inertial navigation information, sparsely triggered visible light images are precisely spatially aligned onto the point cloud model and fused to generate a 3D model with realistic textures. This not only optimizes the completeness of the detection information, seamlessly combining surface textures (visual information such as color and cracks) with precise 3D geometry, but also improves the visualization and positioning accuracy of welding surface defects (such as undercut, weld beads, and surface cracks), making the detection results more intuitive and accurate.

[0193] 4. Welding quality assessment based on a 3D texture model that integrates accurate geometric information and realistic surface texture is more comprehensive and reliable than relying solely on potentially inaccurate point clouds or independently acquired 2D images. This model effectively reproduces the true state of the weld surface, optimizes the criteria for defect identification and judgment, and thus improves the overall accuracy and reliability of online inspection results.

[0194] In one embodiment, for special vehicles, due to their complex structure, there may be some areas that are missed during inspection, especially when there are double or even multiple layers depending on the current scanning angle. There may be welds inside these multiple layers, which may lead to the generation of areas that are missed during inspection.

[0195] Based on this, such as Figure 4 As shown, the presence or absence of obstruction areas can be used to determine missed detection areas, and corresponding path planning can be set to perform laser scanning on the missed detection areas.

[0196] Specifically, when the feature values ​​of the extracted specified image features exceed a preset range or the mobile robot reaches a preset position, it is considered that the current location is a complex region. Complex regions are more likely to have missed detection areas compared to other ordinary regions. As for ordinary regions, whether they are prone to missed detection areas can be determined by manual marking.

[0197] At this point, a depth map is generated by scanning along the current scanning position of the laser scanning device, and the depth gradient is calculated using the Sobel operator. Regions with depth gradients higher than the preset gradient are marked as height transition regions.

[0198] A depth map is constructed along the current laser scanning direction (usually horizontal, but in some special scenarios it can also be vertical or tilted at 45°). The depth map records the depth information of each point in the scene, providing basic data for subsequent gradient calculations.

[0199] The Sobel operator is used to calculate the gradient of the depth map in the x and y directions. It filters the depth map using convolutional kernels to obtain the gradient component for each pixel. The resulting depth gradient represents the amount of depth change per unit horizontal distance.

[0200] When the depth gradient is high (for example, higher than the preset gradient value of 0.3 mm / mm), the area is considered to be a height jump area, which may be the edge of the object, a sudden change in the surface, etc., and there is a greater possibility that there are missed areas in this location.

[0201] For regions with abrupt changes in altitude, the corresponding 3D point cloud region and inertial navigation data are determined. The 3D point cloud region refers to the point cloud data of the 3D point cloud model in the region with abrupt changes in altitude, while the corresponding inertial navigation data is the inertial navigation data collected when acquiring the data for that 3D point cloud region.

[0202] For each point in the 3D point cloud region, frontal orientation and line-of-sight occlusion are determined to obtain the visibility of the corresponding visual frustum.

[0203] Among them, such as Figure 5 As shown, the system first determines the facing direction. If the result is yes, meaning the current facing direction is positive, then the system continues with a line-of-sight occlusion determination. If the result is no, the point is marked as invisible. During the line-of-sight occlusion determination, if the result indicates occlusion, the point is marked as invisible; if the result indicates no occlusion, the point is marked as visible.

[0204] The determination of the front orientation is shown in Formula 11:

[0205] n i ·(T k -P i Formula 11 > 0;

[0206] Where, n i Let P be the normal vector of the i-th point. i Let T be the position of the i-th point. k T represents the location of the inertial navigation module. k -P i Let n be the vector pointing to the inertial navigation module. i When the product is greater than 0, it means that the two are in the same direction, which means that the surface of the point is facing the inertial navigation module. The result is yes, that is, the current orientation is facing forward.

[0207] When determining line-of-sight obstruction, an inertial navigation module T can be constructed. k To point P i For a given ray, use a KD-tree to find points within a 3mm radius of the ray. If a neighboring point is less than a certain distance from the ray (e.g., 1mm), it is considered that the two are almost collinear, and that the neighboring point is located at point P. i In front of, that is, at point P i With the construction of inertial navigation module T k Between, point P is considered i When there is obstruction, the visibility of the vertebral body is considered to be invisible.

[0208] Based on the visibility of the visual frustum of all points, the point cloud occlusion rate corresponding to the 3D point cloud region is obtained. When the point cloud occlusion rate is higher than the preset occlusion rate, the missed area judgment is initiated. The point cloud occlusion rate can be obtained by the ratio of the number of invisible points to the total number of points. Typically, a preset occlusion rate of 40% can be set. When the point cloud occlusion rate is considered to be higher than 40%, it is believed that there is a high probability of missed areas, and the missed area judgment is initiated.

[0209] By using a pre-trained model to predict the probability of missed detections, 3D point cloud regions and inertial navigation data are taken as inputs, and the output is a voxel of probability of missed detection regions.

[0210] The missed detection probability prediction model can adopt a 3D-GAN model architecture, including an encoder, generator, and discriminator.

[0211] The encoder is used to extract features from the 3D point cloud region. Its input is the set of point cloud coordinates corresponding to the 3D point cloud region. Local features are extracted by a multilayer perceptron and a global feature vector is generated by max pooling. A 256-dimensional feature vector is generated as the geometric structure of the compressed point cloud.

[0212] The generator produces a voxel grid based on the features extracted by the encoder. Each voxel represents the false negative probability at its corresponding location. A voxel is the smallest discrete unit in 3D space, and the voxel grid is a grid divided from voxels into a 3D point cloud region. Each grid corresponds to one or more voxels, and each grid includes the false negative probability at that location. The generator expands the features extracted by the encoder to 8192 dimensions through a fully connected layer and upsamples them using 3D transposed convolutions, outputting a 64×64×64 probability voxel grid.

[0213] The discriminator is an explicit module during the training phase, used to output the true probability based on the missed detection probability output by the generator and the actual missed detection voxels. The discriminator is only explicit during the training phase; once the model is trained, it becomes an implicit module and no longer participates in the calculation process.

[0214] The discriminator takes the generator's output of the false negative probability and the real false negative voxels as input. Through 3D convolutional downsampling and fully connected layer classification, it outputs the true probability, which is used to determine whether the input voxel is a real false negative voxel sample or a fake sample generated by the generator, thereby training the generator to generate a more realistic voxel mesh.

[0215] During model training, the first step is sample construction, including positive and negative samples. Positive samples originate from weld seam areas that were actually missed in historical detections. These missed locations are manually marked with 3D bounding boxes, and data augmentation is applied (e.g., random rotation, simulated occlusion). Negative samples originate from point clouds of qualified weld seams from complete scans. Zero-voxel labels are generated to indicate no missed detections. The sample size can be set to 800 positive samples for training the generator and 1200 negative samples for training the discriminator.

[0216] During training, point cloud preprocessing of samples is performed, including downsampling and normalization. Then, feature encoding is performed on the samples to generate probability voxels. Probability mapping is performed based on the probability voxels to corresponding regions. Subsequently, high-risk missed detection regions are extracted, and regions with probability voxels higher than a preset value are identified as high-risk missed detection regions.

[0217] A probability heatmap is obtained from the probability voxels of the missed detection areas, and the corresponding high-risk missed detection areas are then identified based on the probability heatmap. The probability heatmap can be used to describe the probability of voxel missed detections, and its calculation is shown in Formula XII:

[0218]

[0219] Where V(x,y,z) is the original feature value of the probability voxel (x,y,z) of the missed detection region, and H(x,y,z) is the missed detection probability of the probability voxel of the missed detection region, with a value range of [0,1].

[0220] For high-risk missed detection areas, dynamic path planning is generated through probability heatmaps, enabling mobile robots to perform laser scanning on these areas based on the dynamic path planning.

[0221] Specifically, for high-risk missed detection areas, a global cost function is generated using a probability heatmap. Based on the cost value of each target point in the high-risk missed detection area within the global cost function, a motion planning algorithm is generated sequentially. The global cost function is shown in Formula 13.

[0222] C = w1 * Distance + w2 * (1 - H) (Formula Thirteen)

[0223] Where C is the global cost, Distance is the distance from the current point to the target point (which belongs to the high-risk missed detection area), H is the missed detection probability of the target point, w1 is the efficiency weight with a default value of 0.3, and w2 is the risk coverage weight with a default value of 0.7.

[0224] The global cost function is primarily used to evaluate the priority of target points, thus determining which target point to cover first in the next step of the motion planning algorithm. When there are multiple target points, they are visited sequentially according to their cost function values.

[0225] Motion planning algorithms are primarily used to plan the motion of a laser scanning device towards a target point. For example, consider target point A and target point B, with distances of 1.2 meters and 0.8 meters respectively, H values ​​of 0.93 and 0.35 respectively, and calculated cost values ​​C of 0.429 and 0.755 respectively. In this case, although target point B is closer, target point A is prioritized because its cost is lower.

[0226] Based on the motion planning algorithm, a search tree is initialized; in fact, the root node in the search tree is the current position of the laser scanning device. When the laser scanning device is controlled by a robotic arm, the current position of the laser scanning device is also the current position of the robotic arm, which is called N. start .

[0227] Risk-guided sampling is performed based on probability heatmaps to generate sampling points. For example, there is a 70% probability of generating sampling points in high-risk missed detection areas, and a 30% probability of uniform sampling throughout the entire space.

[0228] Perform nearest neighbor selection to determine the tree node closest to the sampling point. Calculate the Euclidean distance between the sampling point and the tree node (which is the node obtained by expanding outwards from the root node in the search tree), and select the tree node N closest to the sampling point. near .

[0229] Based on the distance constraint between the node and obstacles, new nodes are expanded according to the tree nodes. The distance constraint can be set as follows: new node N candidate The distance to obstacles must be greater than a preset distance (e.g., 50mm). New nodes can be obtained by expanding the tree node in a random direction by a certain step size (e.g., 0.1m).

[0230] For the neighboring nodes of the new node, the cost of the new path is calculated, as shown in Formula Fourteen:

[0231] C new =Cost(N) j )+‖N j -N candidate Formula 15;

[0232] Among them, C new Cost is the cost of the new path, which is the path cost between the new node and the starting point (usually the root node). j ) represents neighbor node N j The path cost between the starting point and the origin, ||N j -N candidate ‖ represents the path cost between the neighboring node and the new node.

[0233] When the cost of the new path is lower than the cost of the original path corresponding to the new node, that is, C new <Cost(N j This indicates the new node N. candidate For neighbor node N j If a better path is provided, that is, a lower cost, then the parent node of the neighbor node corresponding to the cost of the new path is updated to the new node.

[0234] Perform path backtracking and execution, determine the new node to cover any target point in the high-risk missed detection area, that is, terminate when the search tree covers any target point, and generate the corresponding backtracking path along the parent node to the target point.

[0235] Additionally, the control parameters of the robotic arm can be linked with motion planning.

[0236] Specifically, the scanning speed of the mobile robot (which can also be considered a robotic arm) is determined based on the path curvature radius corresponding to dynamic path planning; the scanning speed is positively correlated with the path curvature radius, for example, as shown in Formula Sixteen:

[0237] Formula XVI: v = 0.1 + 0.2 · tanh(ρ - 0.5);

[0238] Where v is the scanning speed, ρ is the path curvature radius, and tanh is the hyperbolic tangent function.

[0239] Based on the missed detection probability of the target point in the probability heatmap, the laser power of the laser scanning device is determined; the laser power is positively correlated with the missed detection probability, for example, as shown in Formula 17:

[0240] Formula XVII: P = 70 + 30·H;

[0241] Where P is the laser power and H is the probability of missed detection.

[0242] The scanning density of the laser scanning device is determined based on the regional risk level of the 3D point cloud region. The regional risk level is obtained based on at least one of the following: point cloud occlusion rate, feature value of a specified image feature, and depth gradient. For example, the higher the point cloud occlusion rate, the feature value of the specified image feature, and the depth gradient, the higher the regional risk level. The scanning density is positively correlated with the regional risk level, as shown in Formula 18.

[0243] Formula 18: η = 30 + 20·RiskLevel;

[0244] Where RiskLevel is the level quantification value corresponding to the regional risk level, and its value range can be an integer between [0,2], and η is the scan density.

[0245] like Figure 6 As shown in the embodiments of this application, an online welding quality detection and optimization system is also provided, the system comprising:

[0246] A mobile robot, which carries a laser scanning device, an image acquisition device, an inertial navigation module, and a communication module; wherein the communication module can be a local area network communication module or a wide area network communication module.

[0247] The cloud-based system communicates with the mobile robot via the communication module to execute the online welding quality detection and optimization method as described in any of the above embodiments.

[0248] The various embodiments in this application are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the device and medium embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the description of the method embodiments.

[0249] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A method for online detection and optimization of welding quality, characterized in that, include: A mobile robot is activated to acquire images of the welding areas of a special vehicle; the mobile robot is equipped with a laser scanning device, an image acquisition device, and an inertial navigation module. Laser scanning is performed using the laser scanning device to obtain a laser image, and the inertial navigation module acquires the inertial navigation data of the mobile robot. Based on the laser image, raw point cloud data is generated, and motion compensation and multi-frame alignment are performed on the raw point cloud data using the inertial navigation data to obtain aligned point cloud data. Based on the aligned point cloud data, real-time feature extraction is performed. When the feature value of the extracted specified image feature exceeds the preset range or the mobile robot moves to the preset position, the image acquisition device is activated to acquire the image and obtain a visible light image. The visible light image and the aligned point cloud data are spatially aligned using the inertial navigation data to obtain the aligned image sequence corresponding to the visible light image; A 3D point cloud model is obtained by aligning the point cloud data, and visible light texture data obtained by aligning the image sequence is mapped onto the 3D point cloud model to obtain a 3D texture model. The welding quality is then detected using the 3D texture model. Based on the welding quality inspection results, the welding parameters are adaptively adjusted.

2. The online welding quality detection and optimization method according to claim 1, characterized in that, Based on the laser image, raw point cloud data is generated, and motion compensation and multi-frame alignment are performed on the raw point cloud data using the inertial navigation data to obtain aligned point cloud data, specifically including: For each frame of the laser image, filtering and noise reduction are performed, and the Hessian matrix corresponding to each pixel is calculated to obtain the normal direction of the laser line; A second-order Taylor expansion is performed along the normal direction to fit the extreme points of light intensity, thus obtaining the set of coordinates of the laser centerline. Based on the principle of triangulation, the 2D coordinates of the centerline in the laser centerline coordinate set are converted into a 3D point cloud, and the inertial navigation data is used to apply a reverse motion transformation to each point in the 3D point cloud. For each frame point cloud, a preset number of frame point clouds are combined into a first processing unit, and the minimum point-to-point distance is taken as the first objective function. The nearest point search is accelerated by KD tree and the local matching of sliding window is performed by point cloud normal vector constraints to obtain an aligned second processing unit. During the matching process, the relative pose is calculated by the inertial navigation data as the initial value in the first objective function, and abnormal matches are rejected by chi-square test.

3. The online welding quality detection and optimization method according to claim 2, characterized in that, When the feature value of the extracted specified image feature exceeds a preset range or the mobile robot moves to a preset position, the method further includes: The grid is divided according to the top view of the second processing unit to obtain multiple sector grids, and feature encoding is performed on each sector grid; Based on the feature encoding, the current second processing unit of the current frame and the historical second processing unit of the historical frame are matched for similarity. If the similarity exceeds a preset level and the pose distance is less than a preset distance, point cloud reconstruction is triggered. Extract the region features corresponding to the current second processing unit, and perform feature enhancement based on the feature type of the region features; A pose graph is constructed based on the current second processing unit; wherein, in the pose graph, the center position of the current second processing unit and the inertial navigation data of the current frame are defined as nodes, and multiple types of constraint edges are established based on the positional relationship between the inertial navigation data of the current frame and the inertial navigation data of historical frames, and confidence weights are assigned to different types of constraint edges. The weighted deviation is obtained by summing the weighted deviations of the constraint edges according to the confidence weights corresponding to each type of constraint edge. The minimized reinforcement deviation is used as the second objective function, and the second objective function is solved iteratively to obtain the optimized pose parameters. Based on the optimized pose parameters, the aligned point cloud data is reconstructed into a point cloud.

4. The online welding quality detection and optimization method according to claim 1, characterized in that, The visible light image is spatially aligned with the aligned point cloud data using the inertial navigation data to obtain an aligned image sequence corresponding to the visible light image, specifically including: The inertial navigation data and the visible light image are preprocessed and synchronized. For each frame image in the visible light image, its corresponding visual features are extracted, and the first displacement vector of the adjacent frame image is predicted based on the optical flow algorithm. Based on the inertial navigation data, the second displacement vector between adjacent frame images is predicted, and the state of the second displacement vector is updated by Kalman filtering. Based on the error value corresponding to the first displacement vector and the error value of the second displacement vector after the state update, a third objective function that minimizes the error value is generated to optimize the prediction of the first displacement vector and the second displacement vector. Based on the optimized first and second displacement vectors, an affine transformation is performed on the visible light image to obtain the corresponding aligned image sequence.

5. The online welding quality detection and optimization method according to claim 1, characterized in that, Mapping the visible light texture data obtained from the aligned image sequence onto the 3D point cloud model to obtain the 3D texture model specifically includes: By extrinsic parameter calibration, a coordinate mapping relationship between the 3D point cloud model and the aligned image sequence is established, and a spatiotemporal reference is unified based on the coordinate mapping relationship; The 3D point cloud model is triangulated to generate a mesh model containing vertices, edges, and faces, and then globally parameterized based on the fourth objective function corresponding to minimizing angular distortion. Based on the transformation of camera intrinsic and extrinsic parameters, single-view projection mapping is performed to convert the 3D point cloud model into a 2D pixel model. For each candidate viewpoint, a visibility score is calculated based on the proportion of visible triangles according to its corresponding 2D pixel model, and a quality score is calculated based on viewpoint positivity and anti-blurring. The visibility score, the quality score, and the structural similarity scores between the current candidate viewpoint and the adjacent candidate viewpoints are weighted and fused to obtain a multi-view fusion result, which serves as a 3D texture model.

6. The online welding quality detection and optimization method according to claim 1, characterized in that, When the feature value of the extracted specified image feature exceeds a preset range or the mobile robot moves to a preset position, the method further includes: A depth map is generated by scanning along the current scanning position of the laser scanning device, and the depth gradient is calculated by the Sobel operator. Regions with depth gradients higher than preset gradients are marked as height jump regions. For the height jump region, determine its corresponding 3D point cloud region and corresponding inertial navigation data; For each point in the 3D point cloud region, frontal orientation and visual occlusion are determined to obtain the visibility of the visual cone corresponding to that point. Based on the visibility of the visual cone of all points, the point cloud occlusion rate corresponding to the 3D point cloud region is obtained, and when the point cloud occlusion rate is higher than the preset occlusion rate, the missed detection region determination is initiated. By using a pre-trained missed detection probability prediction model, the 3D point cloud region and the inertial navigation data are taken as input, and the output is a voxel of the missed detection region probability. A probability heatmap is obtained based on the probability voxels of the missed detection areas, and the corresponding high-risk missed detection areas are obtained based on the probability heatmap. For the high-risk missed detection areas, a dynamic path plan is generated using the probability heatmap, so that the mobile robot can perform laser scanning on the high-risk missed detection areas according to the dynamic path plan.

7. The online welding quality detection and optimization method according to claim 6, characterized in that, For the high-risk missed detection areas, dynamic path planning is generated using the probability heatmap, specifically including: For the high-risk missed detection area, a global cost function is generated through the probability heatmap, and a motion planning algorithm is generated sequentially based on the cost value of each target point in the high-risk missed detection area in the global cost function. Based on the motion planning algorithm, a search tree is initialized; in fact, the root node of the search tree is the current position of the laser scanning device. Risk-guided sampling is performed based on the probability heatmap to generate sampling points, and nearest neighbor node selection is performed to determine the tree node closest to the sampling point; Based on the distance constraints between the tree nodes and the obstacles, new nodes are expanded according to the tree nodes. For the neighboring nodes corresponding to the new node, calculate the corresponding new path cost, and when the new path cost is lower than the original path cost corresponding to the new node, update the parent node of the neighboring node corresponding to the new path cost to the new node; The new node is determined to cover any target point in the high-risk missed detection area.

8. The online welding quality detection and optimization method according to claim 7, characterized in that, The mobile robot performs laser scanning on the high-risk missed detection area according to the dynamic path planning, specifically including: The scanning speed of the mobile robot is determined based on the path curvature radius corresponding to the dynamic path planning; the scanning speed is positively correlated with the path curvature radius. Based on the missed detection probability of the target point in the probability heatmap, the laser power of the laser scanning device is determined; the laser power is positively correlated with the missed detection probability. Based on the regional risk level of the 3D point cloud region, the scanning density of the laser scanning device is determined; the regional risk level is obtained based on at least one of the point cloud occlusion rate, the feature value of the specified image feature, and the depth gradient, and the scanning density is positively correlated with the regional risk level.

9. The online welding quality detection and optimization method according to claim 1, characterized in that, Based on the welding quality inspection results, the welding parameters are adaptively adjusted, specifically including: Based on the welding quality inspection results, the corresponding defect types are determined, and the welding quality indicators are quantified; the welding quality indicators include geometric indicators and texture indicators. Based on the defect type, the corresponding welding parameters are matched using a preset defect-parameter mapping rule library; The welding parameters are adaptively adjusted based on the difference between the quantified welding quality indicators and the preset standard indicators.

10. A welding quality online detection and optimization system, characterized in that, The system includes: A mobile robot, which carries a laser scanning device, an image acquisition device, an inertial navigation module, and a communication module; The cloud-based system communicates with the mobile robot via the communication module to execute the online welding quality detection and optimization method as described in any one of claims 1 to 9.

Images