A large-scale welding scene high-fidelity reconstruction and data generation system and method

Abstract

The present application relates to the field of three-dimensional reconstruction and digital twin technology, and particularly relates to a large-scale welding scene high-fidelity reconstruction and data generation system and method, the data generation system comprising a multi-modal data acquisition module, a semantic guidance and analysis module, a multi-modal data preprocessing module, a global hash semantic Gaussian map initialization module, a reconstruction and optimization module, and a high-fidelity data generation module, wherein the reconstruction and optimization module comprises a sliding window optimization submodule, a forward rendering and loss calculation submodule, a semantic gradient dynamic adjustment submodule, an adaptive density control submodule, and a dynamic interference filtering submodule.The large-scale welding scene high-fidelity reconstruction and data generation system and method of the present application realize efficient, high-precision three-dimensional scene reconstruction and high-fidelity data output with re-lighting through multi-modal data perception, adaptive optimization of semantic guidance, and physics-based rendering.

Description

Technical Field

[0001] This invention relates to the field of 3D reconstruction and digital twin technology, specifically to a high-fidelity reconstruction and data generation system and method for large-scale welding scenes, which is particularly suitable for the construction of digital twin systems for welding production lines in industries such as automobile manufacturing, shipbuilding, and aerospace. Background Technology

[0002] With the advancement of intelligent manufacturing, digital twin technology has become an important means to improve production efficiency and ensure product quality. As a key link in the manufacturing process, the accurate reconstruction of the 3D model and the generation of real-time data in the welding scenario are of great significance for process simulation, robot path planning, quality inspection, and condition monitoring.

[0003] Existing 3D reconstruction methods mainly suffer from the following problems:

[0004] Traditional mesh- or point cloud-based representation methods struggle to balance high-fidelity appearance with real-time rendering performance, especially when dealing with intricate structures such as highly reflective metal surfaces and welds.

[0005] Large-scale factory environments (such as welding lines that can reach hundreds of meters in length) pose a severe challenge to the memory usage and computational efficiency of reconstruction algorithms;

[0006] The fusion of LiDAR point clouds and RGB images often only reaches the geometric registration level, lacking semantically guided multimodal geometry-photometric joint optimization, which leads to deviations in edge details and material properties in the reconstruction results;

[0007] Existing 3DGS-based methods are mainly geared towards small indoor scenes and have not been optimized for large-scale industrial scenes, transient disturbances (splashes, dust), and physical material modeling.

[0008] Therefore, there is an urgent need for a new technology that can achieve high-fidelity reconstruction of large-scale welding scenarios and generate multimodal data that can be used for digital twins. Summary of the Invention

[0009] This invention is made to solve the above problems, and aims to provide a high-fidelity reconstruction and data generation system and method for large-scale welding scenes. Through multimodal data perception, semantically guided adaptive optimization and physically based rendering, it can achieve efficient and high-precision 3D scene reconstruction and re-illuminated high-fidelity data output.

[0010] This invention provides a high-fidelity reconstruction and data generation system for large-scale welding scenes, used for constructing digital twin systems of welding production lines in multiple industries including automotive manufacturing, shipbuilding, and aerospace. It features: a multimodal data acquisition module, including an industrial camera and a LiDAR, for simultaneously acquiring RGB image sequences and LiDAR point cloud sequences of the welding scene; the LiDAR point cloud containing three-dimensional geometric information and potential reflection intensity information; a semantic guidance and analysis module for instance segmentation and semantic parsing of the RGB images, identifying and labeling welding robots, tooling fixtures, weldments, weld seams, spatter areas, and highly reflective components in the RGB images, generating corresponding semantic labels; and a multimodal data preprocessing module for processing the RGB images and LiDAR data. The system performs spatiotemporal synchronization and extrinsic parameter calibration on the point cloud and semantic tags to generate a dense color point cloud with semantic tags. A global hash semantic Gaussian map initialization module is used to construct a hash-indexed octree Gaussian map architecture: the dense color point cloud is divided into multiple leaf node voxels, each leaf node voxel stores a set of 3D Gaussian primitives, and each set of 3D Gaussian primitives contains semantic tags and physically based rendering material parameters, i.e., PBR material parameters. A reconstruction and optimization module is used to optimize the parameters of all 3D Gaussian primitives to obtain a globally consistent hash octree Gaussian map. A high-fidelity data generation module, based on the PBR material parameters of each 3D Gaussian primitive, generates high-fidelity reconstructed data using a differentiable renderer and a preset BRDF model, which serves as input data for the digital twin system.

[0011] The reconstruction and optimization module includes: a sliding window optimization submodule, which extracts discrete data from the hash octree Gaussian map into the GPU contiguous memory buffer of the data generation system only within a sliding window consisting of the current field of view and neighboring keyframes, and restricts the Gaussian primitives to be optimized within the sliding window, keeping the computational load at a constant level; a forward rendering and loss calculation submodule, which calculates photometric loss, multimodal material consistency loss, semantic guidance loss, normal alignment loss, and projection depth loss; a semantic gradient dynamic adjustment submodule, which obtains the semantic features of each frame of RGB image and maps the semantic features to Gaussian primitives through back projection, updating the semantic attributes of the Gaussian primitives; an adaptive density control submodule, which dynamically switches the preset density threshold and relaxation threshold based on the semantic category of the semantic label and the iteration stage, thereby achieving local dense encryption of key workstation areas and global sparse coverage of background areas; and a dynamic interference filtering submodule, which identifies and dynamically removes Gaussian primitives representing transient elements based on semantic labels and multi-frame temporal consistency analysis of RGB images.

[0012] The large-scale welding scene high-fidelity reconstruction and data generation system provided by this invention can also have the following features: In the global hash semantic Gaussian map initialization module, the hash index octree Gaussian map architecture uses a spatial hash function to achieve O(1) complexity voxel random access. The size of each leaf node voxel adapts to the depth of the octree in the hash index octree Gaussian map architecture, and memory is allocated only to leaf node voxels that actually have Gaussian elements.

[0013] The large-scale welding scene high-fidelity reconstruction and data generation system provided by this invention may also have the following features: In the adaptive density control submodule, the density threshold is less than the relaxation threshold. The preset density threshold and relaxation threshold are dynamically switched based on the semantic category and iteration stage of the semantic label. For key workstation areas with semantic labels such as "weld", "welding robot end", and "edge of highly reflective component", a density threshold is used. When the gradient of the view space position of the Gaussian primitive exceeds the density threshold, the hash octree is triggered to split downward at the local level, increasing the density of local child node primitives. For non-critical areas with semantic labels such as "background" and "tooling table", a relaxation threshold is used. Splitting is only allowed when the gradient exceeds the relaxation threshold; otherwise, merging or pruning operations are performed.

[0014] The high-fidelity reconstruction and data generation system for large-scale welding scenes provided by this invention may also have the following features: the multimodal data preprocessing module includes: a spatiotemporal synchronization unit, which uses hardware triggering or software interpolation to ensure that the RGB image and the LiDAR point cloud are aligned at the same timestamp; and an extrinsic parameter calibration and registration unit, which is used to initially register the LiDAR point cloud coordinate system with the industrial camera coordinate system to obtain the RGB color value corresponding to each LiDAR point cloud point, forming a dense color point cloud.

[0015] The large-scale welding scene high-fidelity reconstruction and data generation system provided by this invention may also have the following features: In the global hash semantic Gaussian map initialization module, the PBR material parameters include basic color parameters, metallicity parameters, and roughness parameters. For Gaussian elements identified by the semantic guidance and analysis module as highly reflective metal parts, the metallicity parameter is initialized to 0.9–1.0, and the roughness parameter is initialized to 0.2–0.4. For the weld area, the initial range of its basic color parameters and roughness parameters is dynamically adjusted according to the reflection intensity of the lidar and the texture complexity of the RGB image. For insulating plastic or non-metallic parts, the metallicity parameter is initialized to 0.0–0.1, and the roughness parameter is initialized to 0.6–0.8.

[0016] This invention also provides a method for high-fidelity reconstruction and data generation of large-scale welding scenes, which uses the above-mentioned system and has the following features:

[0017] The multimodal data acquisition process involves simultaneously acquiring RGB image sequences and LiDAR point cloud sequences of the welding scene.

[0018] The semantic guidance and analysis step is used to perform instance segmentation and semantic parsing on RGB images, identify and label welding robots, tooling fixtures, weldments, weld seams, spatter areas and highly reflective parts in RGB images, and generate semantic labels;

[0019] The multimodal data preprocessing step is used to perform spatiotemporal synchronization and extrinsic parameter calibration of RGB images, LiDAR point clouds and semantic labels to generate dense color point clouds with semantic labels;

[0020] The global hash semantic Gaussian map initialization step is used to construct the hash index octree Gaussian map architecture, which divides the dense color point cloud into multiple leaf node voxels. Each leaf node voxel stores a set of three-dimensional Gaussian primitives, and each set of three-dimensional Gaussian primitives contains semantic labels and PBR material parameters.

[0021] The reconstruction and optimization steps are used to optimize the parameters of all 3D Gaussian primitives. During the optimization process, the following sub-steps are executed in sequence: sliding window optimization, forward rendering and loss calculation, dynamic adjustment of semantic gradient, semantically guided adaptive density control, and dynamic interference filtering to obtain a globally consistent hash octree Gaussian map.

[0022] The high-fidelity data generation step involves generating high-fidelity reconstructed data based on the PBR material parameters of each 3D Gaussian primitive, using a differentiable renderer and a preset BRDF model, which serves as the input data for the digital twin system.

[0023] The high-fidelity reconstruction and data generation method for large-scale welding scenes provided by this invention may also have the following features: In the reconstruction and optimization steps, the sliding window optimization includes: maintaining a sliding window containing the most recent N keyframes; in each sub-step of the reconstruction and optimization steps, determining the hash voxel set to be activated based on the keyframes and neighborhood range within the current window; extracting the Gaussian metadata in the hash voxel set from the discrete hash table to a contiguous cache array of the GPU; updating only the parameters corresponding to the cache array; when the window slides, releasing the Gaussian metadata leaving the window and loading the newly entered Gaussian metadata.

[0024] The high-fidelity reconstruction and data generation method for large-scale welding scenes provided by this invention may also have the following features: In the global hash semantic Gaussian map initialization step, the initialization and optimization steps of PBR material parameters include: using high dynamic range RGB images captured from multiple angles, and employing an inverse rendering method to pre-estimate the illumination and reflection components of each material region; using the reflection intensity of the lidar to help distinguish between metal and non-metal regions, and setting differentiated initial values ​​of PBR material parameters for different semantic regions; and using a physically based BRDF model to calculate the gradient of PBR material parameters in optimized differentiable rendering.

[0025] Compared with the prior art, the functions and effects of the present invention include:

[0026] The high-fidelity reconstruction and data generation system for large-scale welding scenes involved in this invention achieves O(1) complexity voxel access through a global hash semantic Gaussian map initialization module and constant-level computational load through a sliding window optimization submodule. The combination of these two features can support real-time reconstruction of welding scenes at the hundred-meter level. The adaptive density control submodule performs local densification on key areas such as welds and highly reflective components and sparsely represents the background area, balancing accuracy and storage efficiency. The forward rendering and loss calculation submodule fuses the normal constraints of the lidar point cloud and the photometric loss of the RGB image, and the semantic gradient dynamic adjustment submodule introduces a projection depth weighting factor to suppress far-end noise, significantly improving the geometric accuracy of reconstruction. The dynamic interference filtering submodule automatically identifies and filters out Gaussian primitives that represent transient elements such as welding spatter and smoke, ensuring the purity and stability of the map. Through the high-fidelity data generation module, each Gaussian primitive is decoupled from the PBR material parameters, supporting relighting rendering based on the BRDF model, providing highly realistic data input for the digital twin system. Attached Figure Description

[0027] Figure 1 This is an overall structural block diagram of the high-fidelity reconstruction and data generation system for large-scale welding scenes in an embodiment of the present invention;

[0028] Figure 2 This is a flowchart of the reconstruction and optimization steps in the high-fidelity reconstruction and data generation method for large-scale welding scenes in an embodiment of the present invention.

[0029] In the figure, 1. Multimodal data acquisition module; 11. Industrial camera; 12. LiDAR; 2. Semantic guidance and analysis module; 3. Multimodal data preprocessing module; 4. Global hash semantic Gaussian map initialization module; 5. Reconstruction and optimization module; 6. High-fidelity data generation module; 10. Data generation system. Detailed Implementation

[0030] In the description of this application, it should be noted that, unless otherwise expressly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection, an electrical connection, or a connection that allows communication between them; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication between two components or the interaction between two components. Those skilled in the art can understand the specific meaning of the above terms in this application according to the specific circumstances.

[0031] To make the technical means, creative features, objectives and effects of the present invention easy to understand, the following embodiments, in conjunction with the accompanying drawings, provide a detailed description of the large-scale welding scene high-fidelity reconstruction and data generation system and method of the present invention.

[0032] Figure 1 This is an overall structural block diagram of the high-fidelity reconstruction and data generation system for large-scale welding scenes in an embodiment of the present invention.

[0033] This embodiment provides a high-fidelity reconstruction and data generation system 10 for a large-scale welding scene, hereinafter referred to as the data generation system 10. Specifically, it is used for the digital twin reconstruction of a large-scale automotive welding workshop, with workshop dimensions of 120m × 60m × 8m, including multiple welding robots, tooling fixtures, conveyor lines, and a large number of highly reflective metal weldments. For example... Figure 1 As shown: The data generation system 10 includes a multimodal data acquisition module 1, a semantic guidance and analysis module 2, a multimodal data preprocessing module 3, a global hash semantic Gaussian map initialization module 4, a reconstruction and optimization module 5, and a high-fidelity data generation module 6.

[0034] The multimodal data acquisition module 1 includes an industrial camera 11 and a lidar 12, used to simultaneously acquire RGB image sequences and lidar point cloud sequences of the welding scene. The lidar point cloud contains three-dimensional geometric information and potential reflection intensity information.

[0035] Specifically, in this embodiment, the acquisition devices used by the multimodal data acquisition module 1 are six high-resolution industrial cameras 11 (resolution 2048×2048) and three 64-line lidars 12, which are evenly arranged on the mobile inspection platform or fixed bracket. Hardware synchronous triggering ensures that the timestamps of the RGB images and lidar point cloud data are aligned, and the acquisition frequency is 10Hz.

[0036] The semantic guidance and analysis module 2 is used to perform instance segmentation and semantic parsing on the acquired RGB images, identify and label welding robots, tooling fixtures, welds, weld seams, spatter areas and highly reflective parts in the RGB images, and generate corresponding semantic tags.

[0037] In this embodiment, the semantic guidance and analysis module 2 uses the lightweight network Fast-SAM to perform instance segmentation on each frame of RGB image and generate pixel-level semantic tags. At the same time, it uses the reflection intensity of the LiDAR 12 to help distinguish between metals and non-metals.

[0038] The multimodal data preprocessing module 3 is used to perform spatiotemporal synchronization and extrinsic parameter calibration on the acquired RGB images, LiDAR point clouds and generated semantic tags to generate dense color point clouds with semantic tags.

[0039] Specifically, the multimodal data preprocessing module 3 obtains the extrinsic parameter matrix of the industrial camera 11 and the lidar 12 through a preset calibration board, performs initial registration using traditional ICP (a point set to point set registration method), projects the lidar point cloud onto the RGB image plane, assigns an RGB color value to each point, and forms a dense color point cloud.

[0040] The multimodal data preprocessing module 3 includes a spatiotemporal synchronization unit and an extrinsic parameter calibration and registration unit. The spatiotemporal synchronization unit uses hardware triggering or software interpolation to ensure that the RGB image and the LiDAR point cloud are aligned at the same timestamp. The extrinsic parameter calibration and registration unit is used to initially register the LiDAR point cloud coordinate system with the industrial camera 11 coordinate system to obtain the RGB color value corresponding to each LiDAR point cloud point, forming a dense color point cloud.

[0041] The global hash semantic Gaussian map initialization module 4 is used to construct a hash index octree Gaussian map architecture: the dense color point cloud is divided into multiple leaf node voxels, and each leaf node voxel stores a set of three-dimensional Gaussian primitives. Each set of three-dimensional Gaussian primitives contains position, covariance matrix, opacity, semantic label and physically based rendering (PBR) material parameters, i.e., PBR material parameters.

[0042] The PBR material parameters are represented by learnable parameters, including basic color parameters, metallicity parameters, and roughness parameters. The initialization and optimization process includes: using high dynamic range RGB images captured from multiple angles, and employing an inverse rendering method to pre-estimate the illumination and reflection components of each material region; using the reflection intensity of LiDAR 12 to help distinguish between metallic and non-metallic regions, and setting differentiated initial values ​​for PBR material parameters for different semantic regions; and using a physically based BRDF model to calculate the gradient of the PBR material parameters in optimized differentiable rendering.

[0043] Specifically, for Gaussian elements identified as highly reflective metal parts by the semantic guidance and analysis module 2, the metallicity parameter is initialized to 0.9–1.0, and the roughness parameter is initialized to 0.2–0.4; for weld areas, the initial ranges of their basic color parameters and roughness parameters are dynamically adjusted according to the reflection intensity of the lidar 12 and the texture complexity of the RGB image; for insulating plastic or non-metallic parts, the metallicity parameter is initialized to 0.0–0.1, and the roughness parameter is initialized to 0.6–0.8.

[0044] Furthermore, in this embodiment, the global hash semantic Gaussian map initialization module 4 constructs an 8-layer hash index octree, with the root node voxel having a side length of 5m and the deepest voxel having a side length of approximately 0.04m. A spatial hash function is used to achieve O(1) complexity for random access to voxels, and the hash table size is 2^24. Each leaf node stores a dynamic array containing several Gaussian elements. The size of each leaf node voxel adapts to the depth of the hash index octree, and memory is allocated only to leaf node voxels that actually contain Gaussian elements. Specifically, when initializing each Gaussian element, the position is taken as the point coordinates of the LiDAR point cloud, the covariance is set to isotropic (radius 0.02m), the semantic label is assigned through the nearest neighbor pixel label, and the PBR material parameters are initialized according to the semantic prior (the base color is taken as RGB color; metal parts: metallicity 0.95, roughness 0.3; plastic: metallicity 0.05, roughness 0.7; weld area: metallicity 1.0, roughness 0.2).

[0045] Reconstruction and Optimization Module 5 employs a joint optimization strategy in this embodiment, simultaneously optimizing the parameters (position, covariance, opacity, PBR material parameters, and semantic information) of all 3D Gaussian elements in one iteration. The total number of iterations is approximately 30,000, ultimately yielding a globally consistent hash octree Gaussian map. In each optimization iteration, the following sub-modules are executed: sliding window optimization sub-module, forward rendering and loss calculation sub-module, semantic gradient dynamic adjustment sub-module, adaptive density control sub-module, and dynamic interference filtering sub-module.

[0046] The sliding window optimization submodule is specifically manifested as follows: During the optimization process, in order to ensure the real-time performance of modeling, a sliding window strategy is introduced. Discrete data of the hash index octree Gaussian map is extracted to the GPU continuous memory buffer of the data generation system 10 only within the sliding window composed of the current field of view and the neighboring keyframes. This effectively overcomes the non-continuousness of hash storage and strictly limits the hash voxels to be optimized to the current field of view and the neighboring keyframes, ensuring that the computing load on the machine side remains at a constant level.

[0047] Specifically, in this embodiment, the sliding window optimization module performs the following process: maintaining a sliding window containing the 20 most recent keyframes; calculating all keyframes within the current window to obtain an active voxel set; and extracting Gaussian metadata from the set from a discrete hash table into a contiguous memory buffer on the GPU. The optimizer only updates the parameters in this buffer; Gaussian metadata outside the window is not updated. When the platform moves beyond a certain distance or angle, the sliding window advances forward, ensuring that the computational load at each step is constant.

[0048] The forward rendering and loss calculation submodule is used to calculate the physically realistic photosensitivity loss, multimodal material consistency loss, semantic guidance loss, normal alignment loss, and projection depth loss of the fused PBR material parameters.

[0049] Specifically, the forward rendering and loss calculation include: generating color images, depth maps, semantic segmentation maps, and normal maps for all keyframe views within the current window using a differentiable renderer. A total loss function is constructed, with the weights for each loss term preset experimentally. The expression for the total loss function is:

[0050] ;

[0051] in, , , , , These represent the physically realistic photosensitivity constraint, the multimodal material consistency constraint, the semantic guidance constraint, the normal alignment constraint, and the projection depth constraint, respectively. K1 is the physically realistic photosensitivity constraint. The weights constrain material parameters such as metallicity and roughness to conform to physical priors. K2 is a multimodal material consistency constraint. The weights. K3 is the semantic guidance constraint. The weights are used to make the rendered semantic segmentation map match the real semantic labels. Figure 1 To. K4 is the normal alignment constraint. The weights. K5 is the projection depth constraint. The weights are: k is the keyframe number (index) within the sliding window; W is the set of keyframe indices within the sliding window, or the set of keyframes participating in the current optimization; each keyframe contains an RGB image, a LiDAR point cloud, and a semantic label map; θ is the set of model parameters to be optimized, including the positions of all Gaussian elements, covariance matrix, opacity, PBR material parameters, and semantic features. * These are the optimal parameters obtained through optimization, which are the parameter values ​​that minimize the total loss.

[0052] Optimization stops when the total loss decreases below a preset threshold over multiple consecutive iterations.

[0053] The normal alignment process involves: for each point in the LiDAR point cloud, calculating its local neighborhood normal; and for each Gaussian element, obtaining the normal direction (the eigenvector corresponding to the smallest eigenvalue) from the currently optimized covariance matrix through eigenvalue decomposition. The normal alignment loss measures the difference between the two directions, and its gradient is backpropagated to update the covariance matrix, ensuring that the Gaussian ellipsoid orientation aligns with the LiDAR point cloud normal. During optimization, the normal of each point is estimated from the LiDAR point cloud (calculated through local neighborhood PCA). Experiments show that this loss effectively corrects geometric drift caused by lack of texture.

[0054] The semantic gradient dynamic adjustment submodule obtains the semantic features of each frame of RGB image through a lightweight network, and maps the semantic features to Gaussian elements through backpropagation, that is, updates the semantic attributes of Gaussian elements. During backpropagation, the gradient of the semantic features is multiplied by the projection depth weighting factor. This projection depth weighting factor is related to the projection distance from the 12 points of the LiDAR to the image plane and the reflection intensity of the LiDAR point cloud. This makes the nearby points with high reflection intensity contribute more gradients and suppress the noise of distant and low-quality observations, thereby achieving the effect of optimization dominated by nearby high-confidence observations and suppression of distant noise gradients.

[0055] The adaptive density control submodule dynamically switches between preset density and relaxation thresholds based on the semantic category of semantic tags and the iteration stage. When the density threshold is less than the relaxation threshold, the key work area is locally densely densified, while the background area is globally sparsely covered.

[0056] In this embodiment, density control is performed once every 1000 iterations.

[0057] Specifically, local dense encryption of key workstation areas includes: for key workstation areas with semantic labels such as "weld seam", "welding robot end effector", and "edge of highly reflective component", a density threshold is adopted. When the view space position gradient of the Gaussian primitive exceeds the density threshold, the hash octree is triggered to split downward at the local level to generate a higher density of child node primitive groups.

[0058] For transient areas such as splashes and smoke, a pruning strategy is adopted, and the opacity regularization coefficient is increased to make them decay and be removed quickly.

[0059] Global sparse coverage of the background region includes: applying a relaxed threshold to non-critical regions with semantic labels "background" and "tooling table". Splitting is only allowed when the gradient exceeds the relaxed threshold; otherwise, merging or pruning operations are performed.

[0060] Specifically, in this embodiment, in the V-shaped weld area of ​​a certain weldment, a relaxation threshold is used in the initial optimization stage, and the Gaussian density is kept at a low level. When the data generation system 10 identifies that this area belongs to a critical station, it switches to a dense threshold. When the gradient reaches the set value, an octree split is triggered, and sub-primitives are generated in the deep nodes, resulting in a significant increase in density. Finally, the weld surface exhibits millimeter-level groove details. Meanwhile, the background concrete column always uses a relaxation threshold and does not split, thereby achieving the purpose of saving memory.

[0061] After density control, the hash octree is dynamically adjusted as follows: when splitting causes voxels to overflow, the octree splits downward to generate child nodes; empty voxel memory is reclaimed after pruning.

[0062] The dynamic interference filtering submodule identifies and dynamically removes Gaussian elements that represent transient elements, including welding spatter and smoke, based on semantic tags and multi-frame temporal consistency analysis of RGB images.

[0063] Specifically, the dynamic interference filtering submodule calculates the lifetime confidence of each Gaussian element (based on opacity and semantic category) every 50 iterations, and deletes Gaussian elements with confidence below the threshold and semantic labels of splash or smoke, to ensure that transient interference does not contaminate the map.

[0064] The high-fidelity data generation module 6, based on the input light source parameters, uses the BRDF model and the PBR material parameters of each three-dimensional Gaussian primitive to calculate the reflected radiation brightness in real time through a differentiable renderer, generating high-fidelity reconstructed data with relighting capability, serving the rendering engine, simulation training, and state visualization of the digital twin system.

[0065] This embodiment also provides a method for high-fidelity reconstruction and data generation of large-scale welding scenes, which is carried out using the above-mentioned data generation system 10 and includes the following steps S1 to S6.

[0066] S1, Multimodal data acquisition step, synchronously acquires RGB image sequences and LiDAR point cloud sequences of the welding scene;

[0067] S2, the semantic guidance and analysis step, is used to perform instance segmentation and semantic parsing on RGB images, identify and label welding robots, tooling fixtures, weldments, weld seams, spatter areas and highly reflective parts in RGB images, and generate semantic labels;

[0068] S3, the multimodal data preprocessing step, is used to perform spatiotemporal synchronization and extrinsic parameter calibration of RGB images, LiDAR point clouds and semantic labels to generate dense color point clouds with semantic labels;

[0069] S4, Global Hash Semantic Gaussian Map Initialization Step, is used to construct the hash index octree Gaussian map architecture, dividing the dense color point cloud into multiple leaf node voxels. Each leaf node voxel stores a set of three-dimensional Gaussian primitives, and each set of three-dimensional Gaussian primitives contains semantic labels and PBR material parameters.

[0070] S5, the reconstruction and optimization step, is used to optimize the parameters of all 3D Gaussian primitives. Figure 2 This is a flowchart of the reconstruction and optimization steps in the high-fidelity reconstruction and data generation method for large-scale welding scenes in an embodiment of the present invention, as shown below. Figure 2 As shown, the following sub-steps are executed sequentially during the optimization process, including: S5-1, sliding window optimization; S5-2 forward rendering and loss calculation; S5-3 dynamic adjustment of semantic gradient; S5-4 semantically guided adaptive density control; S5-5 dynamic interference filtering, finally obtaining a globally consistent hash octree Gaussian map.

[0071] S6, the high-fidelity data generation step, generates high-fidelity reconstructed data based on the PBR material parameters of each 3D Gaussian primitive through a differentiable renderer and a preset BRDF model, which serves as the input data for the digital twin system.

[0072] The role and effect of the embodiments

[0073] The high-fidelity reconstruction and data generation system 10 for large-scale welding scenes in this embodiment achieves O(1) complexity voxel access through the global hash semantic Gaussian map initialization module 4, and achieves constant-level computational load through the sliding window optimization submodule. The combination of the two can support real-time reconstruction of welding scenes at the hundred-meter level. The adaptive density control submodule performs local densification on key areas such as welds and highly reflective components, and sparsely represents the background area, balancing accuracy and storage efficiency. The forward rendering and loss calculation submodule fuses the normal constraints of the lidar point cloud and the photometric loss of the RGB image, and introduces a projection depth weighting factor through the semantic gradient dynamic adjustment submodule to suppress far-end noise, significantly improving the geometric accuracy of reconstruction. The dynamic interference filtering submodule automatically identifies and filters out Gaussian elements that represent transient elements such as welding spatter and smoke, ensuring the purity and stability of the map. Through the high-fidelity data generation module 6, each Gaussian element is decoupled from the PBR material parameters, supporting relighting rendering based on the BRDF model, providing highly realistic data input for the digital twin system.

[0074] This invention can be directly deployed in scenarios such as automotive body-in-white welding lines and aerospace large structural component welding workshops for the construction of high-fidelity digital twin platforms. The generated data can be used for robot offline programming, process parameter optimization, equipment management, etc., resulting in significant economic benefits.

[0075] Those skilled in the art should understand that this invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to this invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the invention as claimed. The scope of protection of this invention is defined by the appended claims and their equivalents.

Claims

1. A high-fidelity reconstruction and data generation system for large-scale welding scenes, used for constructing digital twin systems of welding production lines in multiple industries including automobile manufacturing, shipbuilding, and aerospace, characterized in that... include: The multimodal data acquisition module includes an industrial camera and a lidar, used to simultaneously acquire RGB image sequences and lidar point cloud sequences of the welding scene. The lidar point cloud contains three-dimensional geometric information and potential reflection intensity information. The semantic guidance and analysis module is used to perform instance segmentation and semantic parsing on the RGB image, identify and label the welding robot, tooling fixture, weldment, weld seam, spatter area and highly reflective parts in the RGB image, and generate corresponding semantic tags; A multimodal data preprocessing module is used to perform spatiotemporal synchronization and extrinsic parameter calibration on the RGB image, the lidar point cloud, and the semantic tags to generate a dense color point cloud with the semantic tags. The global hash semantic Gaussian map initialization module is used to construct a hash index octree Gaussian map architecture: the dense color point cloud is divided into multiple leaf node voxels, each leaf node voxel stores a set of three-dimensional Gaussian primitives, and each set of three-dimensional Gaussian primitives contains the semantic labels and physically based rendering material parameters, i.e., PBR material parameters. The reconstruction and optimization module is used to optimize the parameters of all the three-dimensional Gaussian elements to obtain a globally consistent hash octree Gaussian map. The high-fidelity data generation module, based on the PBR material parameters of each 3D Gaussian primitive, generates high-fidelity reconstructed data using a differentiable renderer and a preset BRDF model, which serves as the input data for the digital twin system. The reconstruction and optimization module includes: The sliding window optimization submodule is used to extract discrete data from the hash octree Gaussian map into the system's GPU contiguous memory buffer only within a sliding window consisting of the current field of view and neighborhood keyframes, and to restrict the Gaussian primitives to be optimized within the sliding window, so that the computational load is kept at a constant level. The forward rendering and loss calculation submodule is used to calculate photometric loss, multimodal material consistency loss, semantic guidance loss, normal alignment loss, and projection depth loss. The semantic gradient dynamic adjustment submodule obtains the semantic features of each frame of the RGB image, maps the semantic features to the Gaussian primitives through back projection, and updates the semantic attributes of the Gaussian primitives. The adaptive density control submodule dynamically switches between preset density thresholds and relaxation thresholds based on the semantic category of the semantic tag and the iteration stage, thereby achieving local dense encryption of key workstation areas and global sparse coverage of background areas. The dynamic interference filtering submodule identifies and dynamically removes Gaussian elements that represent transient elements based on the semantic tags and multi-frame temporal consistency analysis of the RGB image.

2. The high-fidelity reconstruction and data generation system for large-scale welding scenes according to claim 1, characterized in that: In the global hash semantic Gaussian map initialization module, the hash index octree Gaussian map architecture uses a spatial hash function to achieve O(1) complexity voxel random access. The size of each leaf node voxel adapts to the depth of the octree in the hash index octree Gaussian map architecture, and memory is allocated only to leaf node voxels that actually have Gaussian elements.

3. The high-fidelity reconstruction and data generation system for large-scale welding scenes according to claim 1, characterized in that: In the adaptive density control submodule, the density threshold is less than the relaxation threshold. The preset density threshold and relaxation threshold are dynamically switched based on the semantic category of the semantic tag and the iteration stage, including: For key workstation areas with semantic labels such as "weld seam", "welding robot end effector", and "edge of highly reflective component", a density threshold is used. When the view space position gradient of the Gaussian primitive exceeds the density threshold, the hash octree is triggered to split downward at the local level, increasing the density of local child node primitives. The relaxation threshold is applied to non-critical regions with semantic labels "background" and "tooling table". Splitting is only allowed when the gradient exceeds the relaxation threshold; otherwise, merging or pruning operations are performed.

4. The high-fidelity reconstruction and data generation system for large-scale welding scenes according to claim 1, Its features are: The multimodal data preprocessing module includes: The spatiotemporal synchronization unit uses hardware triggering or software interpolation to ensure that the RGB image and the LiDAR point cloud are aligned at the same timestamp. The extrinsic parameter calibration and registration unit is used to initially register the lidar point cloud coordinate system with the industrial camera coordinate system to obtain the RGB color value corresponding to each lidar point cloud point, thereby forming the dense color point cloud.

5. The high-fidelity reconstruction and data generation system for large-scale welding scenes according to claim 1, characterized in that: In the global hash semantic Gaussian map initialization module, the PBR material parameters include basic color parameters, metallicity parameters, and roughness parameters. For the Gaussian elements identified by the semantic guidance and analysis module as highly reflective metal components, the metallicity parameter is initialized to 0.9–1.0, and the roughness parameter is initialized to 0.2–0.

4. For the weld area, the initial range of its basic color parameters and roughness parameters is dynamically adjusted according to the reflection intensity of the lidar and the texture complexity of the RGB image. For insulating plastic or non-metallic parts, the metallicity parameter is initialized to 0.0 to 0.1, and the roughness parameter is initialized to 0.6 to 0.

8.

6. A method for high-fidelity reconstruction and data generation of large-scale welding scenes, comprising using the high-fidelity reconstruction and data generation system for large-scale welding scenes as described in any one of claims 1-5, characterized in that, Includes the following steps: The multimodal data acquisition process involves simultaneously acquiring RGB image sequences and LiDAR point cloud sequences of the welding scene. The semantic guidance and analysis step is used to perform instance segmentation and semantic parsing on the RGB image, identify and label the welding robot, tooling fixture, weldment, weld seam, spatter area and highly reflective parts in the RGB image, and generate semantic tags; A multimodal data preprocessing step is used to perform spatiotemporal synchronization and extrinsic parameter calibration on the RGB image, the lidar point cloud, and the semantic label to generate a dense color point cloud with the semantic label; The global hash semantic Gaussian map initialization step is used to construct a hash index octree Gaussian map architecture, which divides the dense color point cloud into multiple leaf node voxels. Each leaf node voxel stores a set of three-dimensional Gaussian primitives, and each set of three-dimensional Gaussian primitives contains the semantic label and the PBR material parameters. The reconstruction and optimization steps are used to optimize the parameters of all the three-dimensional Gaussian primitives. During the optimization process, the following sub-steps are executed in sequence, including: sliding window optimization, forward rendering and loss calculation, dynamic adjustment of semantic gradient, adaptive density control, and dynamic interference filtering, to obtain a globally consistent hash octree Gaussian map. The high-fidelity data generation step involves generating high-fidelity reconstructed data based on the PBR material parameters of each three-dimensional Gaussian primitive, using a differentiable renderer and a preset BRDF model, as input data for the digital twin system.

7. The method for high-fidelity reconstruction and data generation of large-scale welding scenes according to claim 6, characterized in that: In the reconstruction and optimization steps, the sliding window optimization includes: Maintain a sliding window containing the N most recent keyframes; In each sub-step of the reconstruction and optimization step, the set of hash voxels to be activated is determined based on the keyframes and neighborhood range within the current window. Extract the Gaussian metadata from the hash voxel set from the discrete hash table into a contiguous cache array of the GPU; Only update the parameters corresponding to this cache array; When a window is slid, the Gaussian metadata that leaves the window is released, and the Gaussian metadata that newly enters the window is loaded.

8. The method for high-fidelity reconstruction and data generation of large-scale welding scenes according to claim 6, characterized in that: In the global hash semantic Gaussian map initialization step, the initialization and optimization steps of the PBR material parameters include: By using high dynamic range RGB images captured from multiple angles, the lighting and reflection components of each material area are estimated in advance using an inverse rendering method. Based on the reflection intensity of the lidar, the metal and non-metal regions are distinguished, and different initial values ​​of PBR material parameters are set for different semantic regions. The gradient of the PBR material parameters is calculated using a physically based BRDF model in optimized differentiable rendering.

Images