Real-time three-dimensional reconstruction method and system under cabin assembly scene

Abstract

This invention relates to the field of 3D reconstruction technology, and discloses a real-time 3D reconstruction method and system for cabin assembly scenarios to improve overall performance. The method includes: first, uniformly standardizing the input real-time image; then, calculating pixel-level visibility weights based on photometric residuals and geometric reprojection errors; and finally, selecting keyframes needed to construct a ray pool using the mean of the pixel-level visibility weights; performing volume rendering on the standardized real-time image based on a global coarse network, and then reconstructing the 3D model of the cabin based on the point cloud; comparing the reconstruction result with an offline teaching benchmark model to obtain a reconstruction quality assessment result; determining whether there are problem areas based on the reconstruction quality assessment result; if so, performing local volume rendering reprocessing on each problem area using a local fine network mapped to the problem type; and then reconstructing the 3D model of the cabin based on the point cloud after spatially fusing the local fine network volume field and the global coarse network volume field.

Description

Technical Field

[0001] This invention relates to the field of three-dimensional reconstruction technology, and in particular to a real-time three-dimensional reconstruction method and system for cabin assembly scenarios. Background Technology

[0002] 3D reconstruction is widely used in scenarios such as robot assembly and positioning, medical navigation, autonomous driving simulation, and industrial inspection. It generates point clouds, meshes, voxels, or implicit fields based on observed images and depth information to restore the 3D geometry and appearance of the scene or target, and then obtains measurable 3D information in a unified coordinate system.

[0003] Existing 3D reconstruction methods utilize multi-view / multi-time observations acquired by one or more sensors to estimate and generate a 3D representation (geometry and appearance) of the scene consistent with these observations. The goal is to recover the shape, position, and possible texture / reflection properties of objects / environment within a given coordinate system and scale, and to be used for rendering or measurement from any viewpoint.

[0004] Among them, the 3D reconstruction technology based on NeRF network (Neural Radiance Field, NeRF) can reproduce the appearance of complex materials and the visible view after occlusion well, but the training / rendering cost is large, explicit geometry requires threshold extraction, and the accuracy of boundary / thin wall is affected.

[0005] Meanwhile, in actual inspection / modeling scenarios, common problems such as severe occlusion, limited viewing angle, and small target scale often lead to incomplete reconstruction, loss of detail, and scale drift. Therefore, it is necessary to improve NeRF to complete real-time 3D reconstruction of small structures under confined spaces and occlusion conditions. This will provide real-time and reliable key object status data support for complex human-machine collaborative assembly scenarios, helping to improve assembly accuracy, efficiency, and operational safety. Summary of the Invention

[0006] The purpose of this invention is to disclose a real-time three-dimensional reconstruction method and system for cabin assembly scenarios, so as to improve overall performance.

[0007] To achieve the above objectives, this invention discloses a real-time three-dimensional reconstruction method for cabin assembly scenarios, comprising: Step S1: First, perform unified standardization processing on the input real-time image, then calculate the pixel-level visibility weight based on the photometric residual and geometric reprojection error, and then filter the key frames required to construct the ray pool by the mean of the pixel-level visibility weight. Step S2: First, perform volume rendering on the standardized real-time image based on a global coarse network, and then reconstruct the 3D model of the cabin based on the point cloud. Step S3: Compare the reconstruction results with the offline teaching benchmark model to obtain the reconstruction quality assessment results. Based on the reconstruction quality assessment results, determine whether there are problem areas. If so, identify the type of problem area and proceed to step S4; otherwise, proceed to step S5. Step S4: For each problem area, perform local volume rendering reprocessing of the cabin through the local fine network mapped to the problem type, and then merge the reprocessed local fine network volume field with the global coarse network volume field in space. Then, reconstruct the 3D model of the cabin based on the merged point cloud, and repeat the process to step S3. Step S5: Output the reconstructed 3D model; Among them, the global coarse network and the local fine network are processed differently using two NeRF networks with different rates. Both NeRF networks adopt an improved NeRF network that introduces position-encoded multi-resolution hashing and directional-encoded spherical harmonics. Compared with the global coarse network, each local fine network adds error-driven incremental sampling and has a higher hash level and spatial resolution in its network structure.

[0008] Preferably, the calculation of pixel-level visibility weights includes: ; in, For the first Coordinates in frame image The visibility weight of the pixel. For the first Coordinates in frame image The visibility weight of the pixel. For optical flow confidence, Rate the luminance. Scoring for geometry It is a constant, and , .

[0009] Preferably, the keyframes required to construct the ray pool are selected by means of pixel-level visibility weights, including: Image frames with contribution scores greater than a specified threshold are selected as keyframes. If the set of keyframes selected within the set time period is empty, the image frame with the highest contribution score is selected as the keyframe. Among them, the Frame image contribution The calculation formula is: ; For the first The statistical mean of the visibility weights of frame images; For the first Compare the view frustum of the frame image with the existing occupied grid; To measure the first Information diversity index of sampled rays in a frame image.

[0010] Preferably, the multi-resolution hashing of the global coarse network adopts a progressive update strategy.

[0011] Preferably, the update frequency of the local fine network is lower than that of the global coarse network, and it is triggered when the quality score of any local region after processing in step S4 is lower than a set threshold.

[0012] Preferably, the types of problems in the problem area include: reconstruction holes, artifacts, and deformed regions.

[0013] Preferably, when the problem area exceeds the coverage of the local fine network, the problem area is divided, and then the sub-regions are processed one by one in sequence; when the problem area is less than or equal to the coverage of the local fine network, the area is extracted according to the coverage of the local fine network.

[0014] Preferably, the process of outputting the reconstructed 3D model also includes: Noise and artifact filtering is performed to remove isolated points, flight points, artifacts that do not conform to the cabin structure, dynamic afterimages, and erroneous structures. Smoothing is applied to local high-noise areas and normal anomalies are corrected. The number of points is reduced by voxel simplification and cluster downsampling while preserving key configurations and details; if the output is a mesh, faceting, topology correction and texture coordinate finishing are performed. The final 3D model is unified to the preset coordinate system and unit system, layered according to the cabin structure and key teaching parts, and output according to the predetermined data format.

[0015] To achieve the above objectives, the present invention also discloses a real-time three-dimensional reconstruction system for a cabin assembly scenario, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-described method.

[0016] The present invention has the following beneficial effects: 1. A novel method for calculating pixel-level visibility weights based on photometric residuals and geometric reprojection errors has been disclosed, which can be completed in a very short time to meet the requirements of real-time image output; moreover, it can improve the quality of keyframes required for constructing ray pools by mean screening of pixel-level visibility weights.

[0017] 2. An offline teaching benchmark model is introduced. By comparing the reconstruction results with the offline teaching benchmark model, the reconstruction quality assessment results can be obtained, which can ensure the accuracy of the problem area location.

[0018] 3. When there are no problem areas, the reconstruction and output of the 3D model of the cabin can be quickly achieved through a global coarse network. When there are problem areas, the cabin's local volume rendering is reprocessed through a local fine network, and the reprocessed local fine network volume field is spatially fused with the global coarse network volume field. The two networks with different levels of fineness cooperate with each other, and in addition to the complementary relationship of coarse network base and fine network repair, a real-time image inspection, correction and output online closed loop is also realized.

[0019] 4. The mapping between local fine networks and problem types can accommodate many complex situations where the location of problem areas cannot be predicted in advance. When identifying problem areas, only the most important problem types need to be identified. Even if a problem area has two or more problem types under certain circumstances, it can be repaired one by one to meet the quality assessment standards and then output based on the iterative mechanism of this invention. Since the area to be repaired locally is usually relatively small, even if the hash level and spatial resolution of the fine network structure are higher than those of the coarse network, the time required to complete the repair is relatively controllable. The real-time performance of the two working together is also far superior to the resource and time consumption required to process the entire real-time image directly based on the fine network.

[0020] The present invention will now be described in further detail with reference to the accompanying drawings. Attached Figure Description

[0021] The accompanying drawings, which form part of this application, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an undue limitation of the invention. In the drawings: Figure 1 This is a flowchart of the real-time three-dimensional reconstruction method for cabin assembly scenarios disclosed in the embodiments of the present invention. Detailed Implementation

[0022] The embodiments of the present invention will be described in detail below with reference to the accompanying drawings, but the present invention can be implemented in many different ways as defined and covered by the claims.

[0023] Example 1 This embodiment is based on a confined cabin assembly scenario, specifically an assembly scenario inside a cabin, to address issues such as inaccurate recognition, inefficient reconstruction, and difficulty in perceiving the internal state caused by dynamic occlusion and limited viewing angle. A depth camera is placed approximately 1 meter away from the cabin. The input real-time image is configured to output only a portion of the cabin's image. The input real-time image undergoes online processing to remove dynamic occlusion, followed by 3D reconstruction based on an improved NeRF algorithm. Finally, the 3D reconstruction model is optimized to solve the problems of incomplete reconstruction, low efficiency, and difficulty in perceiving the internal state in confined spaces. This achieves accurate reconstruction of the cabin assembly scenario, providing operators with an intuitive model to understand the internal state and supporting subsequent collaborative assembly applications.

[0024] likeFigure 1 As shown, the method in this embodiment mainly includes the following steps.

[0025] Step S1: First, perform unified standardization processing on the input real-time image, then calculate the pixel-level visibility weight based on the photometric residual and geometric reprojection error, and then select the keyframes required to construct the ray pool by the mean of the pixel-level visibility weights.

[0026] Dynamic occlusion mainly refers to the changes in lighting and shadows caused by the movement of the robotic arm during assembly, as well as the movement of the operator and the robotic arm itself. If left unaddressed, this can lead to the reconstruction of the 3D model containing parts that are not actually present in the actual cabin. Therefore, this step primarily generates a pixel-level visibility weight map based on the photometric residuals and geometric reprojection errors of adjacent frames, used for online masking / reduced dynamic occlusion. The specific online processing logic is detailed below.

[0027] S11. The RGB images from the camera are standardized in both photometric and geometric dimensions to eliminate interference from brightness, color differences, and lens distortion, and to unify the images into a stable, distortion-free, and size-controlled standard image space; thereby providing a high-quality and comparable input data foundation for subsequent photometric residual calculation, geometric reprojection error estimation, and NeRF 3D reconstruction.

[0028] Distortion correction is performed on the depth image from the camera. Similar to color distortion correction for RGB images, depth maps are also affected by camera lens distortion, especially with wide-angle lenses. Therefore, the depth map needs to be distorted along with the RGB image to ensure the geometric accuracy of the depth information. The depth image is then normalized to ensure that the image resolution is aligned with the image, guaranteeing accurate depth information for each pixel.

[0029] Operating procedures: S111, Image Capture Settings: The output real-time image is calibrated to ensure that only the cabin portion of the image is output, while other parts unrelated to the cabin are deleted.

[0030] S112, Image Alignment, Color and Intensity Normalization: First, align the depth image and RGB image to ensure that each frame corresponds one-to-one, unify the dynamic range and color gamut of the image, and avoid instability in subsequent algorithms due to differences in brightness / exposure.

[0031] The specific approach can be as follows: treat the current frame image as sRGB; normalize the color gamut range of the pixels to [0,1], and calculate the luminance channel. To prepare for subsequent photometric residuals, R, G, and B represent the values ​​of different color channels.

[0032] S113, Distortion Correction: Used to correct distortions in depth and RGB images to ensure correct geometric relationships. To ensure real-time processing, an offline LUT (Lookup Table) can be trained in advance. That is, an offline LUT is constructed using distortion formulas to complete the mapping of distorted coordinates to ideal coordinates.

[0033] S114. Unified Resolution: Used to stabilize real-time frame rate, save video memory, and make the computational load of subsequent modules predictable. A common practice is to scale proportionally while maintaining the aspect ratio, with camera intrinsics also scaled proportionally.

[0034] S12, Photometric residual processing, utilizes the photometric changes between adjacent frames to measure pixel-by-pixel appearance consistency, highlights areas with significant texture changes that may be moving or occluded, suppresses photometrically stable static background areas, and provides photometric constraints for subsequent visibility weight estimation and keyframe quality assessment, thereby achieving early identification and weight reduction of dynamically occluded pixels.

[0035] The specific operating procedure is as follows: S121. To ensure lightweight design, the Lucas-Kanade pyramid method (a widely used differential method for optical flow estimation) is used to estimate the optical flow from the previous frame to the current frame. To ensure real-time performance, the estimation is first performed at a low resolution, and then upsampled back to the original size. Simultaneously, the optical flow confidence level is calculated. For subsequent judgment, optical flow confidence level The value ranges from [0,1]. A larger value indicates a more stable and reliable image frame. Represents the coordinates of a pixel.

[0036] S122. Use the motion field between two frames to geometrically resample the pixels of the previous frame into the coordinate system of the current frame, so that the static scene is aligned between the two frames, making it convenient to perform pixel-by-pixel subtraction. , This represents the brightness value of the previous frame image after color normalization and distortion correction. This represents the optical flow from the previous frame to the current frame. Let be the optical flow deformation function. This represents the result of geometrically resampling the pixels of the previous frame into the coordinate system of the current frame.

[0037] S123, Calculate pixel-level photometric residuals ;in, This represents the brightness value of the current image frame after color normalization and distortion correction.

[0038] S124. Perform local contrast normalization. , yes The 3×3 local standard deviation.

[0039] S125, to Perform median filtering or small morphological opening operations to remove isolated noise; combine with optical flow confidence: if A lower value increases the suspicion level of that pixel.

[0040] S13, Geometric reprojection error, is a cross-frame geometric prediction of feature points based on camera pose and scene geometry. It quantifies geometric consistency by comparing the deviation between the predicted position and the actual observed position. Pixels with small errors are regarded as static and reliable regions, while pixels with large errors are regarded as dynamic objects, occlusions, or inaccurately estimated regions. It provides a geometric confidence measure for constructing a pixel-level visibility weight map and improves robustness under complex motion and occlusion conditions.

[0041] The specific operating procedure is as follows: S131, Feature Detection and Matching, in and Feature extraction is performed using AKAZE (Accelerated-KAZE, a scale-invariant and rotation-invariant feature detection and description algorithm). The essential matrix E or homography matrix H is estimated based on the Random Sample Consensus Algorithm (RANSAC) through fast matching, outliers are removed, and interior points M are extracted.

[0042] S132. Geometric prediction of points: Based on camera motion and scene geometry, predict the position of a point from the previous frame in the current frame. Then, subtract the predicted position from the actual observed position to obtain the geometric reprojection error—this is the key basis for determining whether a pixel belongs to static consistency (small error) or dynamic / occluded / inaccurate estimation (large error). Specifically, it can be done by using RANSAC to fit a homography matrix H within small blocks (the image is divided into small blocks), and then reprojecting the pixel from the previous frame... Mapped to the current frame, .

[0043] S133. Calculate the sparsity error of feature points. .

[0044] S134, pixel-level expansion, using nearest neighbor and Gaussian smoothing to obtain a dense error field. .

[0045] S14, Merge into a pixel-level visibility weight map The specific steps can be divided into: S141, Luminosity Score , It is a constant whose value ranges from 0.1 to 0.2; The function is used to ensure that pixel values ​​are within the appropriate range.

[0046] S142, Geometric Scoring It is a constant whose value is in the range of 1 to 2. The value approaches 1, indicating a small error and a tendency towards a static state. Approaching 0 results in a large error and occlusion. Intermediate transformation parameters for ease of description (no real meaning).

[0047] S143. Visibility weight calculation: ; ; in, , .

[0048] In this step, visibility weight Multiple formulas are introduced to impose restrictions. For ease of description, intermediate transformation parameters (without real meaning) are used to improve... The accuracy and reliability. When This pixel can be classified as a static point. This pixel can be classified as a dynamic point, thus completing the annotation of dynamically occluded pixels.

[0049] In this step, keyframe selection involves automatically selecting frames with high clarity, rich information, and reasonable perspective changes from the continuous video stream as keyframes for 3D reconstruction. Blurry, abnormally exposed, or redundant frames that contribute little to the scene are removed. This process balances image quality, visibility, and new information increments, thereby reducing the size of NeRF training data, improving the utilization rate of effective samples, and ensuring the real-time performance and stability of 3D reconstruction.

[0050] The detailed logical processing steps can be as follows: S151. Quality gating can be set from multiple different dimensions: Dimension 1: Convert the current frame image to grayscale I, calculate the Laplacian response, take the variance, and determine whether the variance is greater than the first empirical threshold, which is tentatively set to [80, 120].

[0051] Dimension 2: Calculate the mean value and saturated pixel ratio of the current frame's grayscale image. If the above indicators are not within the second empirical threshold, the frame is considered too dark or too bright and is not selected as a keyframe.

[0052] Dimension 3: Calculate the rotation and translation changes between the current frame and the nearest keyframe. If the change is less than the set third threshold, the frame is considered to have only slight jitter or insufficient viewpoint change, and is not selected as a keyframe.

[0053] The above three dimensions are the core and conventional keyframe selection strategies in NeRF-based 3D reconstruction. They are selected from the three golden dimensions of image information richness, imaging quality, and motion baseline, respectively.

[0054] S152. Contribution Score (Select Highest Score): The specific scoring formula can be: ;in, The statistical mean of the visibility weight map for the current frame (the visibility weight of a single pixel is as described above). A higher value indicates less dynamic occlusion and more reliable pixels; The current frame's view frustum is compared with the existing occupied grid to measure how many "newly observed voxels / new parallaxes" this frame can bring; This is an index used to measure the information diversity of sampled rays in an image frame, and is used to evaluate whether the texture / edge and spatial distribution are balanced, so as to avoid selecting image frames that are "all smooth walls".

[0055] S153. Keyframe Determination: First, perform a coarse screening of keyframes based on three conventional dimensions (including: the current frame must have a sufficient pose difference relative to the previous keyframe); then select from the coarse screening results. Frames exceeding the specified threshold are considered keyframes. In special cases, even if the keyframes selected based on the aforementioned judgment rules are an empty set, at least one frame will be selected according to the contribution score within a set time period (e.g., every 0.6-1.0 seconds) to prevent a long period without keyframes.

[0056] Based on the above screening, keyframes suitable for 3D reconstruction have been successfully obtained. A ray pool can then be performed on these selected frames. Construct, in which, Indicates the starting point. Indicates direction; the camera and starting point are determined by the camera's intrinsic parameters and the current pose.

[0057] Step S2: The standardized real-time image is first rendered using a global coarse network, and then the 3D model of the cabin is reconstructed based on the point cloud.

[0058] In this embodiment, modifications are made to the original NeRF network by introducing position-encoded multi-resolution hashing and orientation-encoded spherical harmonics. A dual-rate NeRF structure of a global coarse network and local fine networks is adopted: multi-resolution hashing maps 3D spatial coordinates to multi-scale features, enabling even small networks to express fine textures and sharp geometry; spherical harmonic orientation encoding characterizes view-dependent appearances (such as specular highlights and anisotropic reflections). Based on this, the cabin space is divided into a global coarse-resolution implicit volume field and several local high-resolution implicit volume fields. The former ensures rapid updates to the overall cabin geometry each frame, while the latter refines key teaching parts and problem areas as needed, within the time budget, thus balancing real-time performance and local accuracy under limited computing power.

[0059] Among them, spherical harmonics are a set of orthogonal basis functions defined on a sphere, which are particularly suitable for modeling direction-dependent functions (such as illumination). Encoding the dependence of color on direction using a set of spherical harmonic coefficients is more mathematically intuitive and requires fewer parameters than letting the network learn directly; multi-resolution hashing is a commonly used technique for improving NERF 3D reconstruction, which will not be elaborated on here.

[0060] In this step, a global coarse network covers the entire cabin space, using a smaller MLP network and fewer levels of multi-resolution hash coding. The hash grid has a low resolution and is mainly used to quickly fit the overall geometric contour of the cabin.

[0061] Step S3: Compare the reconstruction results with the offline teaching benchmark model to obtain the reconstruction quality assessment results. Based on the reconstruction quality assessment results, determine whether there are problem areas. If so, identify the type of problem area and proceed to step S4; otherwise, proceed to step S5.

[0062] To facilitate this step, this embodiment introduces a mechanism that combines offline teaching and online self-testing. First, a standard cabin benchmark model and key component detection rules are constructed through offline teaching. Then, in the online stage, voids, artifacts, deformations, and deviations from the standard assembly state are automatically detected and corrected through error-driven incremental sampling and network fine-tuning. This ensures that the final 3D model meets the application requirements of confined cabin assembly scenarios in terms of integrity, accuracy, and stability.

[0063] Among them, offline teaching and benchmark model construction involves building a high-precision three-dimensional benchmark of a "standard cabin" offline during the system deployment or trial assembly phase, as well as the spatial position and allowable deviation of several key detection parts. This is used to perform alignment detection and key optimization on the real-time reconstruction results during the online phase, guiding subsequent incremental training to focus on key assembly areas.

[0064] The specific operating procedures include: S31. Baseline Data Acquisition: When the cabin is in a standard or qualified assembly state, use the same or equivalent cameras as those used in the operation phase to acquire multi-view images of the cabin interior, and combine them with the cabin's three-dimensional model as a geometric prior when necessary.

[0065] S32. Offline high-precision 3D reconstruction: Without being limited by real-time requirements, more thorough iteration and denser sampling are used to obtain a high-precision 3D reference model of the cabin (explicit representation of point cloud / mesh / Gaussian, etc. + corresponding NeRF implicit volume field).

[0066] S33. Teaching Key Parts and Detection Rules: Teach and mark key detection parts on the benchmark 3D model, including the inner surface of the bulkhead, the inner boss of the bulkhead, and the inner cable of the bulkhead, and record their spatial position, geometric range and category; set the allowable position / attitude / shape deviation threshold and the missing / interference judgment rules for each type of part.

[0067] S34. Teaching Parameter Storage: Package and store the baseline 3D model, the above-mentioned key teaching part information, and the detection threshold as teaching detection parameters for online use.

[0068] S35. Reconstruction quality assessment and problem area detection.

[0069] Under the constraints of the offline teaching benchmark, the current 3D reconstruction results are quantitatively evaluated and spatially located. On the one hand, the reconstruction holes, artifacts and deformed areas are identified, and on the other hand, key parts with excessive deviations from the standard assembly state are detected. This provides a spatial block-level basis for subsequent targeted incremental sampling and network optimization, avoids blind retraining or global reconstruction, and provides problem area inputs for the activation and updating of local fine networks.

[0070] The operation process can be as follows: using the feature points in the current point cloud / mesh, perform rigid body registration or feature registration between the current 3D model and the offline obtained reference 3D model to make them fall in the same coordinate system; according to the spatial position of each key part in the teaching parameters, extract the corresponding region in the current reconstructed model, calculate the deviation between it and the reference model in position, attitude and local geometry (flatness, hole position and diameter, etc.), mark the region with deviation exceeding the corresponding threshold as "problem region", and then determine the problem type corresponding to each region according to the problem classification algorithm.

[0071] Step S4: For each problem area, perform local volume rendering reprocessing of the cabin through a local fine network mapped to the problem type, and then merge the reprocessed local fine network volume field with the global coarse network volume field in space. Then, reconstruct the 3D model of the cabin based on the merged point cloud, and repeat the process to step S3.

[0072] In this step, the local fine network is used to define several local sub-regions around the key teaching parts, the current assembly station, and the problem areas obtained from subsequent self-inspection. Each sub-region corresponds to a local fine network or a local high-resolution hash sub-network, which uses more hash levels and higher spatial resolution to perform fine reconstruction of local details.

[0073] In this step, during the process of mapping the problem region to the local fine network, the size of the area that each local fine network can cover during the processing is fixed. When the problem region exceeds the coverage range of the local fine network, the problem region is divided, and then the sub-regions are processed one by one in sequence. When the problem region is smaller than or equal to the coverage range of the local fine network, the region is extracted according to the coverage range of the local fine network.

[0074] In this embodiment, the global coarse network and the local fine network are differentiated by two NeRF networks with different rates. Both NeRF networks are improved NeRF networks that introduce position-encoded multi-resolution hashing and direction-encoded spherical harmonics. Compared with the global coarse network, each local fine network adds error-driven incremental sampling and has a higher hash level and spatial resolution in its network structure.

[0075] The local fine network employs a strategy of focused sampling and short-iteration fine-tuning, while the global coarse network maintains lightweight updates or is partially frozen. This approach of "coarse network as a foundation + fine network for local refinement" particularly strengthens key assembly parts marked by offline teaching, achieving local refinement and globally stable online optimization, rather than costly global retraining from scratch.

[0076] Furthermore, the global coarse network and the local fine network share the same directional encoding (spherical harmonic basis), while the position encoding maintains its own independent hash table and occupied grid, so as to control the sampling density and update frequency at different scales.

[0077] In this step, the error-driven incremental sampling, training, and optimization iterations of the local fine network are detailed below.

[0078] S41. Selection of Closed-Loop Keyframes: For any type of problem, the local fine network to be optimized can be back-searched based on the location of the problem area it covers. Keyframes with high visibility weight (less dynamic occlusion), complementary viewpoints (large parallax), and high image quality (large Laplacian variance) are selected first to construct the local ray subset of the local fine network. For the region where the key teaching part is located, the selection ratio of relevant keyframes can be appropriately increased.

[0079] S42. Key Sampling in Problem Regions: Within each problem region covered by the local fine network, the sample point density is increased only in depth intervals determined as "possibly geometric" by the corresponding local OccGrid (Occupancy Grid, an intermediate data structure for data storage and representation). Simultaneously, the sampling probability is increased for depth intervals with high reprojection errors, high geometric discontinuities, or large teaching biases. Meanwhile, the sampling frequency is kept low or non-existent outside the local fine network to control the overall computational load. The global coarse network maintains only a small amount of global ray sampling in this stage to maintain global geometric stability and does not participate in key sampling.

[0080] S43. Incremental Training Update and Body Field Fusion: Adopting a small-batch, short-cycle incremental training strategy, only performing several rounds of forward / backward updates on the local ray subsets corresponding to each local fine network, controlling the learning rate and update steps, ensuring smooth optimization convergence, and not introducing new large-scale deformations.

[0081] After incremental training is completed, the updated local fine network volume field and the global coarse network volume field are spatially fused: within the coverage area of ​​the local fine network, the density and color prediction results of the local fine network take precedence, while in other areas, the prediction results are provided by the global coarse network, and are simultaneously fed back to the OccGrid and point cloud export module, realizing an online closed loop of "coarse network as the base and fine network as the repair" for simultaneous verification, correction and output, and prioritizing the improvement of reconstruction quality near key teaching parts.

[0082] Step S5: Output the reconstructed 3D model.

[0083] Preferably, during the model optimization and release process, noise and artifact filtering is first performed by combining point weights, visibility statistics, and local geometric consistency to remove isolated points, flying points, and artifacts that do not conform to the cabin structure. Dynamic afterimages and erroneous structures near key teaching areas are prioritized for removal. Next, local high-noise areas are lightly smoothed, and normal anomalies are corrected to ensure the continuity and geometric accuracy of the cabin surface, especially key areas such as equipment mounting surfaces and interfaces. Subsequently, data compression and simplification are performed, reducing the number of points through voxel simplification and cluster downsampling while retaining key configurations and details. If the output is a mesh, faceting, topology correction, and texture coordinate refining are performed to improve rendering and transmission efficiency. Finally, the final 3D model is unified to a preset coordinate system and unit system, layered according to the cabin structure and key teaching areas to facilitate assembly task decomposition and collaboration, and output according to a predetermined data format for subsequent assembly simulation, path planning, collision detection, and AR collaborative applications.

[0084] In summary, this embodiment, based on the elimination of dynamic occlusion through high-quality keyframe screening, utilizes a dual-rate improved NeRF network with multi-resolution hash position coding and spherical harmonic direction coding to perform hierarchical implicit modeling of the confined cabin space: a global coarse network maintains the real-time updated overall outline of the cabin, while a local fine network performs high-resolution refinement of key assembly areas and reconstruction problem areas. Combined with structures such as OccGrid, it realizes an explicit model of point cloud / mesh that is trained and exported simultaneously. This transforms the traditional incomplete and inefficient reconstruction process into a computationally controlled, real-time updated, and locally high-precision 3D reconstruction result, thereby providing intuitive and reliable 3D environment model support for internal state perception and collaborative operation during cabin assembly.

[0085] Example 2 This embodiment is a further optimization based on the above embodiment 1.

[0086] In this embodiment, on the one hand, within each training / update cycle, rays are extracted from the ray pool under a fixed total ray budget, and the main components are pre-allocated to the global coarse network. Rays are preferentially extracted from keyframes with high pixel visibility weights, dispersed coverage, and large viewpoint changes to ensure balanced constraints on the global cabin outline under multiple viewpoints. In the occupied grid corresponding to the global coarse network, layered sampling is only performed in depth intervals with a high probability of being occupied, skipping obviously empty intervals to avoid sampling in empty areas. For each sample point, the density and intermediate feature vector are obtained by multi-resolution hash position encoding of the global coarse network, and then the color is decoded by spherical harmonic direction encoding, and volume rendering is performed along the ray to obtain the rendered color. The color reconstruction loss (the error between the volume rendered color and the real color) is calculated and weighted according to the ray weight. At the same time, density sparsity loss is introduced to suppress invalid density, and the total loss is the sum of color reconstruction loss and density sparsity loss. A small batch, fixed-step training loop is used to perform forward and backward propagation, update the hash table parameters and spherical harmonic direction branch parameters of the global coarse network, so that the global coarse network continues to converge within a finite time.

[0087] To balance convergence speed and stability, the multi-resolution hashing of the global coarse network adopts a progressive update strategy: initially, only low-resolution hashing layers are enabled to quickly capture the overall geometry of the cabin. As iterations progress or the global loss decreases, higher-resolution hashing layers are gradually enabled. The parameters of the newly enabled layers are randomly initialized and a relatively small learning rate is set to avoid oscillations on the converged results.

[0088] On the other hand, under the premise of satisfying the global coarse network update, the remaining ray budget and remaining time budget are allocated to the local fine network for high-resolution refinement of key areas. Based on the teaching detection and self-inspection results, local sub-regions containing key teaching parts, under-observed areas or geometrically inconsistent areas are selected as the objects to be refined in this round. Within the ray range corresponding to these local regions, high-weight rays passing through the region are extracted from the ray pool to form local ray subsets.

[0089] In the local fine network corresponding to each local ray, high-density sampling is performed only in depth intervals with high occupancy probability. The sampling probability is increased for depth segments with high reprojection error, high geometric discontinuity, or large teaching deviation. Utilizing the high-resolution hash encoding and spherically harmonic direction branching of the local fine network, forward inference and volume rendering are performed on the sampled points. Color reconstruction loss and local geometric regularization loss are calculated, and these are given higher loss weights compared to the global coarse network. Preferably, this embodiment uses a small learning rate and a limited number of iterations to incrementally update the local fine network, updating only the parameters of the corresponding local region to avoid reverse interference to other regions.

[0090] The local fine network updates less frequently than the global coarse network, and is triggered when a local area's quality score is detected to be below a threshold. This allows for more precise detail completion of key assembly areas while ensuring overall real-time performance.

[0091] Furthermore, during the global-local fusion of the volumetric field: the densities of the global coarse network and the local fine network are queried separately on regular 3D grid points, and the geometric strength is calculated by combining information such as occupied grid and static weights. For grid points simultaneously covered by both global and local networks, the geometric strength of the local fine network is preferentially adopted, ensuring that key regions are dominated by the high-resolution volumetric field, while other regions are provided with geometry by the global coarse network. This is consistent with the steps described above. Corresponding to the calculated geometric score, during the derivation process, a moving average can be applied to the geometric intensity, and an adaptive threshold can be used to classify grid points with a value greater than the threshold as "having geometry" and the rest as air, in order to reduce flickering and artifacts in online training.

[0092] Preferably, in this embodiment, grid points with intensity greater than a threshold are selected in the fused geometric intensity field to generate a point cloud: their three-dimensional coordinates are extracted, and the normal is estimated by the neighborhood geometric intensity gradient; the points are assigned color by the volume rendering module, and the rendering results of the local fine network are used preferentially in key areas; at the same time, static weights are added to each point by combining visibility weights and multi-frame consistency to distinguish between static structural points of the cabin and potential dynamic points.

[0093] In the 3D model reconstruction process, preferably, point clouds with normals, colors, and static weights are input into conventional surface reconstruction algorithms (such as Poisson reconstruction or triangulation). Based on the overall geometry provided by the global coarse network, local fine networks supplement the details of key parts, generating a 3D mesh model of the cabin interior in real time. This model can be used for subsequent self-verification and optimization, and can also be directly used for assembly simulation, path planning, and collaborative operation demonstration.

[0094] Example 3 Based on the two embodiments described above, this embodiment discloses a real-time three-dimensional reconstruction system for a cabin assembly scenario, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the methods disclosed in the two embodiments described above.

[0095] In summary, the methods and systems disclosed in the above embodiments of the present invention have the following beneficial effects: 1. A novel method for calculating pixel-level visibility weights based on photometric residuals and geometric reprojection errors has been disclosed, which can be completed in a very short time to meet the requirements of real-time image output; moreover, it can improve the quality of keyframes required for constructing ray pools by mean screening of pixel-level visibility weights.

[0096] 2. An offline teaching benchmark model is introduced. By comparing the reconstruction results with the offline teaching benchmark model, the reconstruction quality assessment results can be obtained, which can ensure the accuracy of the problem area location.

[0097] 3. When there are no problem areas, the reconstruction and output of the 3D model of the cabin can be quickly achieved through a global coarse network. When there are problem areas, the cabin's local volume rendering is reprocessed through a local fine network, and the reprocessed local fine network volume field is spatially fused with the global coarse network volume field. The two networks with different levels of fineness cooperate with each other, and in addition to the complementary relationship of coarse network base and fine network repair, a real-time image inspection, correction and output online closed loop is also realized.

[0098] 4. The mapping between local fine networks and problem types can accommodate many complex situations where the location of problem areas cannot be predicted in advance. When identifying problem areas, only the most important problem types need to be identified. Even if a problem area has two or more problem types under certain circumstances, it can be repaired one by one to meet the quality assessment standards and then output based on the iterative mechanism of this invention. Since the area to be repaired locally is usually relatively small, even if the hash level and spatial resolution of the fine network structure are higher than those of the coarse network, the time required to complete the repair is relatively controllable. The real-time performance of the two working together is also far superior to the resource and time consumption required to process the entire real-time image directly based on the fine network.

[0099] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A real-time 3D reconstruction method for a cabin assembly scenario, characterized in that, include: Step S1: First, perform unified standardization processing on the input real-time image, then calculate the pixel-level visibility weight based on the photometric residual and geometric reprojection error, and then filter the key frames required to construct the ray pool by the mean of the pixel-level visibility weight. Step S2: First, perform volume rendering on the standardized real-time image based on a global coarse network, and then reconstruct the 3D model of the cabin based on the point cloud. Step S3: Compare the reconstruction results with the offline teaching benchmark model to obtain the reconstruction quality assessment results. Based on the reconstruction quality assessment results, determine whether there are problem areas. If so, identify the type of problem area and proceed to step S4; otherwise, proceed to step S5. Step S4: For each problem area, perform local volume rendering reprocessing of the cabin through the local fine network mapped to the problem type, and then merge the reprocessed local fine network volume field with the global coarse network volume field in space. Then, reconstruct the 3D model of the cabin based on the merged point cloud, and repeat the process to step S3. Step S5: Output the reconstructed 3D model; Among them, the global coarse network and the local fine network are processed differently using two NeRF networks with different rates. Both NeRF networks adopt an improved NeRF network that introduces position-encoded multi-resolution hashing and directional-encoded spherical harmonics. Compared with the global coarse network, each local fine network adds error-driven incremental sampling and has a higher hash level and spatial resolution in its network structure.

2. The real-time three-dimensional reconstruction method for cabin assembly scenarios according to claim 1, characterized in that, The calculation of pixel-level visibility weights includes: ； in, For the first Coordinates in frame image The visibility weight of the pixel. For the first Coordinates in frame image The visibility weight of the pixel. For optical flow confidence, Rate the luminance. Scoring for geometry It is a constant, and , .

3. The real-time three-dimensional reconstruction method for cabin assembly scenarios according to claim 2, characterized in that, The keyframes required to construct the ray pool by filtering the mean of pixel-level visibility weights include: Image frames with contribution scores greater than a specified threshold are selected as keyframes. If the set of keyframes selected within the set time period is empty, the image frame with the highest contribution score is selected as the keyframe. Among them, the Frame image contribution The calculation formula is: ; For the first The statistical mean of the visibility weights of frame images; For the first Compare the view frustum of the frame image with the existing occupied grid; To measure the first Information diversity index of sampled rays in a frame image.

4. The real-time three-dimensional reconstruction method for cabin assembly scenarios according to any one of claims 1 to 3, characterized in that, The multi-resolution hashing of the global coarse network adopts a progressive update strategy.

5. The real-time three-dimensional reconstruction method for cabin assembly scenarios according to any one of claims 1 to 3, characterized in that, The local fine network updates less frequently than the global coarse network, and is triggered when the quality score of any local region after processing in step S4 is found to be lower than a set threshold.

6. The real-time three-dimensional reconstruction method for cabin assembly scenarios according to any one of claims 1 to 3, characterized in that, Problem types in the problem area include: reconstruction holes, artifacts, and deformed regions.

7. The real-time three-dimensional reconstruction method for cabin assembly scenarios according to claim 6, characterized in that, When the problem area exceeds the coverage of the local fine network, the problem area is divided into sub-regions, and then each sub-region is processed in turn. When the problem area is less than or equal to the coverage of the local fine network, the region is extracted according to the coverage of the local fine network.

8. The real-time three-dimensional reconstruction method for cabin assembly scenarios according to claim 7, characterized in that, The process of outputting the reconstructed 3D model also includes: Noise and artifact filtering is performed to remove isolated points, flight points, artifacts that do not conform to the cabin structure, dynamic afterimages, and erroneous structures. Smoothing is applied to local high-noise areas and normal anomalies are corrected. The number of points is reduced by voxel simplification and cluster downsampling while preserving key configurations and details; if the output is a mesh, faceting, topology correction and texture coordinate finishing are performed. The final 3D model is unified to the preset coordinate system and unit system, layered according to the cabin structure and key teaching parts, and output according to the predetermined data format.

9. A real-time 3D reconstruction system for a cabin assembly scenario, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method described in any one of claims 1 to 8.

Images