Multi-scale scene reconstruction method based on multi-source data fusion

Abstract

The application discloses a multi-scale scene reconstruction method based on multi-source data fusion, relates to the technical field of three-dimensional scene reconstruction, and comprises the following steps: by utilizing a modulation angle marker of a row direction statistical signal to solve an image, an image frame sequence is divided into multiple modulation angle channels with consistent illumination states; for each channel, a coarse-level geometric segment representing a macroscopic topology of a scene and a fine-level geometric segment containing a specific illumination strip feature are independently constructed; by calculating a cross-channel geometric divergence between the fine-level geometric segments, a fine-level credibility weight for quantitatively distinguishing real details and strip artifacts is generated; based on a coarse-level basic model and by utilizing the weight, multi-channel fine-level geometries are selectively fused, and a multi-scale three-dimensional scene model is output. The application utilizes the difference between the space-time invariance of real textures and the phase dependence of artifacts, accurately removes rolling shutter geometric wave noise, and perfectly retains high-frequency texture details.

Description

Technical Field

[0001] This invention relates to the field of 3D scene reconstruction technology, specifically to a multi-scale scene reconstruction method based on multi-source data fusion. Background Technology

[0002] In the fields of industrial digital twins and facility operation and maintenance inspection, high-precision 3D scene reconstruction using multi-source acquisition devices mounted on mobile robots or handheld terminals has become the mainstream operating mode. To meet millimeter-level measurement requirements, such operations are typically carried out in industrial sites equipped with high-intensity AC lighting systems, and the algorithm needs to map the acquired multi-source image sequences and construct a 3D spatial geometric model.

[0003] However, the flickering characteristics of light sources commonly found in industrial environments are highly susceptible to asynchronous coupling with the rolling shutter mechanism of imaging equipment, resulting in horizontal stripes of alternating light and dark areas in the original image. In existing 3D reconstruction workflows, algorithms often struggle to distinguish between illumination variations in the radiation domain and textures in the geometric domain, easily misinterpreting these two-dimensional illumination stripes as normal undulations of the 3D surface, thus generating wavy artifacts on the model surface that are not actually there. This false geometric noise can mask real defects such as tiny cracks, severely impacting the reliability of automated measurements. While conventional smoothing filtering methods can suppress artifacts, they also erase crucial high-frequency details such as nameplate characters and screw hole edges, making it difficult to maintain high model fidelity while removing illumination interference. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention proposes a multi-scale scene reconstruction method based on multi-source data fusion. This method solves the problem that geometric stripe artifacts are generated in the reconstruction model due to the coupling of rolling shutter effect and ambient lighting modulation, and that it is difficult to retain real high-frequency details while removing artifacts.

[0005] To achieve the above objectives, the present invention provides the following technical solution:

[0006] The row direction statistical signal of each frame in the image frame sequence is extracted, and the modulation angle mark representing the relative phase position in the ambient lighting change cycle at the moment of image acquisition is calculated.

[0007] The image frame sequence is divided into multiple modulation angle channels based on modulation angle markers;

[0008] By using pose data, image frames in each modulation angle channel are unified to the global coordinate system, and for each modulation angle channel, coarse geometric segments representing the macroscopic topology of the scene and fine geometric segments containing lighting strip features are constructed.

[0009] For the corresponding position in the reconstructed space, after obtaining the geometric feature quantity of the fine geometric segment of each modulation angle channel at that position, the cross-channel divergence is calculated, and a fine confidence weight is generated to quantify and distinguish between real details and illumination strip artifacts.

[0010] Generate a coarse-level basic model based on coarse-level geometric fragments generated from modulation angle channels;

[0011] Fine-grained confidence weights are used to fuse the fine-grained geometric segments of all modulation angle channels, and combined with the coarse-grained basic model to output a multi-scale 3D scene model.

[0012] Compared with existing technologies, it has the following advantages:

[0013] This proposed method for multi-scale scene reconstruction based on multi-source data fusion discretizes continuously time-varying dynamic illumination interference into multiple internally stable channel subsets using modulation angle markers. By constructing a multi-scale representation of coarse skeleton + fine details, and leveraging the essential difference between the spatiotemporal invariance of real structures and the phase dependence of stripe artifacts, an unsupervised authenticity identification mechanism based on cross-channel geometric divergence is established. This mechanism can accurately locate and eliminate false geometric wave noise while perfectly preserving real high-frequency textures such as nameplate characters and micro-cracks. This overcomes the technical bottleneck of traditional smoothing and denoising algorithms that inevitably erase key details when suppressing stripe artifacts, providing a digital foundation for industrial facility operation and maintenance that combines geometric flatness and high-fidelity texture. Attached Figure Description

[0014] Figure 1 This is a schematic diagram of the method flow of the present invention. Detailed Implementation

[0015] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0016] Please see Figure 1 This application provides a multi-scale scene reconstruction method based on multi-source data fusion;

[0017] The method specifically includes the following steps:

[0018] The processor receives a multi-source image frame sequence containing all acquired data from the mobile robot and handheld devices. To eliminate interference from texture details and highlight the stripe features distributed along the row direction, the processor employs a row direction integration data processing method to perform dimensionality reduction on each frame. First, for the k-th frame in the sequence, with a width of W and a height of H, the one-dimensional row mean signal of that frame is calculated according to the following formula. :

[0019]

[0020] In the formula, k is the frame identifier, used to uniquely identify the sequence number of each image in the multi-source data sequence, and its value range is a positive integer sequence. is the pixel grayscale value, which is the single-channel brightness value of the k-th frame image after grayscale processing at the two-dimensional pixel coordinates (x,y). W is the image width, which is the total number of pixels contained in the image in the horizontal direction. The one-dimensional row mean signal is the arithmetic mean of the brightness of all pixels in the y-th row of the k-th frame image in the vertical direction, representing the overall illumination response intensity of that row after excluding horizontal texture interference.

[0021] Specifically, the image data is processed using the signal superposition enhancement principle. Since the rolling shutter strips caused by industrial lighting have geometric uniformity in the horizontal direction, while the texture of the actual scene usually exhibits random high-frequency changes, the ambient light signal and the scene texture signal can be effectively separated by the accumulation or averaging statistical operation in the row direction. This processing method amplifies the strip signal with horizontal uniformity, while smoothing and suppressing random texture noise without horizontal uniformity, thus providing a data foundation with a high signal-to-noise ratio for subsequent calculation steps.

[0022] Obtaining a one-dimensional row mean signal Then, the processor uses the periodicity of the signal to calculate the environmental state parameters. In order to separate the modulation angle marker from the signal, the processor first establishes the following model:

[0023]

[0024] In the formula, The amplitude of the strip is the peak intensity of the periodic brightness oscillation signal caused by illumination modulation in the k-th frame image (i.e., the absolute value of the difference between the brightest point and the average brightness, which is a scalar). y represents the spatial angular frequency, reflecting how quickly the bright and dark fringes change in the vertical direction of the image, and y represents the row coordinates of the image. The DC component represents the average background brightness of the image after removing periodic band fluctuations, i.e., the base background brightness. The modulation angle marker is a scalar angle parameter (with a value range of [0, 2π]), which represents the relative phase position of the ambient light intensity during its periodic change process at the moment of acquisition of the k-th frame image;

[0025] In detail, the periodic brightness oscillation signal caused by illumination modulation refers to the sinusoidal brightness fluctuation component presented in the image line mean signal after the high-frequency time-varying characteristics of industrial LED luminous flux are mapped to the spatial domain by the rolling shutter. This component has already deducted the inherent texture of the scene and the background light. The spatial angular frequency is a quantitative indicator describing the density of the distribution of light and dark stripes in the vertical direction of the image. Its value is determined by the ambient lighting modulation frequency and the camera line readout time. In the image, the larger the spatial angular frequency, the more periods of brightness change are contained in the unit number of lines, and the denser the stripes appear. Conversely, the lower the spatial angular frequency, the sparser the stripes appear. The DC component represents the average background brightness reference value of the image after removing periodic flicker fluctuations, reflecting the overall brightness of the environment.

[0026] Furthermore, since the illumination intensity changes periodically (e.g., completing a bright-dark-bright cycle every 1 / 100th of a second), the moment each frame begins exposure (i.e., the moment the first row of pixels is captured) must fall at a certain instant within this cycle. The modulation angle marker describes the offset of that instant relative to the starting point of the intensity change cycle (e.g., the intensity zero-crossing point or peak point), expressed in angles (0 to 2π). For example, if... =0 (assuming the light intensity crosses zero and rises), meaning this photo was taken when the light was just beginning to brighten. =π / 2 (corresponding to peak light intensity), meaning this photo was taken when the light was at its brightest. In a rolling shutter camera, the exposure times for different rows are staggered. It directly determines the initial position of the bright and dark stripes on the image (i.e., in which row the first dark stripe appears). If the modulation angle markers of two frames are the same, it means that they were acquired at the same phase point of the light intensity change cycle, so the position and shape of the bright and dark stripes on their images will be completely consistent.

[0027] To ensure that the modulation angle marker in the above model can be calculated in real time and stably on mobile terminals with limited computing resources. In this embodiment, the parameters are determined using a strategy of locating the dominant frequency in the frequency domain and solving linearly in the time domain. Specifically:

[0028] S101: For one-dimensional row mean signal Perform a Fast Fourier Transform to search for the frequency component corresponding to the maximum amplitude, and determine it as the spatial angular frequency. This value is objectively determined by the inherent modulation frequency of the ambient lighting equipment and the line readout time of the camera;

[0029] S102: Using the trigonometric function expansion formula, the above nonlinear model is transformed into the following linear calculation model:

[0030]

[0031] Among them, intermediate parameters , Due to spatial angular frequency Since this has already been determined in the frequency domain search, for each row of y, its corresponding sine and cosine values ​​are known constants. At this point, the problem is transformed into a problem concerning the unknowns. The processor constructs a matrix equation for a system of linear equations with multiple variables (where... The corresponding coefficients are constants (1) and can be directly obtained using the standard least squares method (such as the pseudo-inverse matrix method). The optimal solution;

[0032] S103: The processor is based on the two-parameter arctangent relationship. Uniquely identifies the modulation angle marker ;

[0033] Specifically, this step first defines a model describing lighting modulation, and then proposes a linearization transformation strategy to transform the originally complex nonlinear fitting problem into a simple matrix multiplication operation, avoiding the risk of infinite loops or convergence failures that may occur in iterative algorithms, improving computational efficiency, and enabling the originally asynchronously acquired multi-source data to have the possibility of alignment and classification based on lighting state consistency.

[0034] In another embodiment, a one-dimensional row mean signal In addition to using the arithmetic mean, the calculation method can also use median statistics to generate a one-dimensional row mean signal. The median of the gray values ​​of all pixels in the y-th row is used instead of the mean. When there are extremely bright or dark noise points (such as salt and pepper noise) or local strong reflective points in the image, median statistics can more effectively suppress the interference of these extreme outliers on the strip signal extraction than mean statistics, thus generating a more robust one-dimensional signal. Correspondingly, in addition to using the least squares method for sine fitting, the modulation angle mark can also use the peak detection method, that is, directly detect the row coordinate position of the peak in the one-dimensional signal, and directly deduce the modulation angle mark based on the relationship between the peak position and the spatial angular frequency, thereby reducing the computational complexity and adapting to the real-time processing needs of low-computing-power devices.

[0035] Based on modulation angle marker Frames with similar lighting conditions are grouped together to isolate data from different lighting conditions. First, the modulation angle marker corresponding to each frame image is obtained. To achieve data-driven adaptive grouping, the processor establishes an optimization objective model based on minimizing the circumferential dispersion within the channel, as follows:

[0036]

[0037] Where J is the total circumferential dispersion, which is equal to the sum of the circumferential dispersions within all channels. The internal dispersion of a single channel is calculated by traversing all frames contained within that channel and calculating the modulation angle marker for each frame. The corner center of the channel circumferential distance between Square the values ​​and sum them up;

[0038] Specifically, the total circumferential dispersion J is a scalar objective function that measures the tightness of all groupings, and N is the total number of modulation angle channels. This value can be adaptively determined by the data distribution characteristics or preset. For the c-th modulation angle channel, there is a set of image frame indices belonging to the c-th channel. This set stores the sequence number k of all original image frames classified into the c-th channel. It is used to quickly locate and extract specific frame subsets with similar illumination modulation states from the original multi-source image frame sequence during the computation process. The modulation angle marker for the k-th frame. Let be the angle center of the c-th modulation angle channel, which is the average position of all modulation angle markers within that channel on the unit circle. This is a circumferential distance function used to calculate the shortest arc distance between two angles on the unit circle. Its specific expression is: ;

[0039] It should be noted that the modulation angle channel is a set of images after discretizing and classifying a continuous image sequence based on the illumination modulation state. The modulation angle channel divides the continuous phase space from 0 to 2π into N intervals and groups all image frames falling into the same interval. For example, assuming the original sequence has 10 images {1,2,3,4,5,6,7,8,9,10}, after division, the first modulation angle channel... It may contain {1, 5, 9}, the second channel. It may contain {2, 6, 10}, where {1, 5, 9} is... The sequence number of the original image frame;

[0040] Since the modulation angle markers are periodic data defined in the interval [0, 2π), 0 radians and 2π radians represent exactly the same lighting state. Ordinary Euclidean distance cannot correctly handle this characteristic of being consecutive at the beginning and end. However, the circumferential distance function ensures that the length of the minor arc on the circumference is always taken when calculating the difference, avoiding artificial truncation errors at the boundary. Let be an angle scalar in the interval [0, 2π), representing the statistical mean of the modulation angle markers of all frames in the c-th channel on the unit circle. Since the illumination intensity changes periodically with time, the angle center... It maps a specific relative phase position (i.e., illumination state) in the ambient lighting cycle. This relative phase position has a clear visual correspondence: when the modulation angle mark is close to the center of the angle, it means that these frames were exposed at the same relative time point of light intensity fluctuation. Therefore, the bright and dark stripes in their images will show a highly consistent spatial distribution pattern (e.g., the dark stripes in all frames appear precisely in the 100th and 300th rows of the image), thus ensuring the radiation consistency of appearance features within the channel. The circumferential dispersion within the channel reflects the degree of consistency of the illumination state of the image frames grouped into the same channel. The smaller the dispersion within the channel, the closer the positions of the bright and dark stripes contained in the image frames within that channel are and the more similar the stripe patterns are, thus satisfying the clustering of similar states. The total circumferential dispersion J is the global optimization objective, representing the sum of the internal dispersion of all channels. By minimizing the value of J, the processor ensures the optimality of the overall grouping strategy, that is, it realizes the most reasonable allocation of all image frames to each illumination state subset, so that the difference within each group is minimized.

[0041] To efficiently solve the above optimization objective and complete the channel construction, this embodiment adopts the following iterative clustering strategy:

[0042] S201: The processor bases all modulation angle markings The distribution density on the circumference is initialized with N seed points as the initial corner centers;

[0043] S202: For each frame k, calculate its modulation angle marker. The circumferential distance from each current corner center is used to assign it to the nearest channel, forming an updated set of frame identifiers;

[0044] S203: Recalculate the angle center of each channel and update the angle center value using the circumferential averaging method. Specifically, map all angles in the channel to unit vectors and then calculate the direction angle of the vector sum.

[0045] S204: Repeat S202 and S203 until the total circumferential dispersion J converges or the number of iterations reaches the preset maximum value, then output the final modulation angle channel set. ;

[0046] Detailed modulation angle channel set It refers to an index container that forms N independent state subsets after dividing the original multi-source image frame sequence based on the illumination modulation angle marker, where c=1,...,N, and N represents the total number of modulation angle channels after discretizing the continuous illumination modulation period. This stores unique identifiers for all image frames belonging to the c-th channel. This set represents the discretized sampling classification of the dynamic ambient lighting process, grouping them into the same channel. The modulation angle marker of all image frames within. All of them fall within the specific angle range defined by the channel, which means that these frames were acquired at the same relative phase instant of the ambient light intensity fluctuation cycle. Therefore, the bright and dark stripes on their images show a high degree of consistency in spatial distribution and radiation gain, thereby transforming the complex radiation interference that originally changed continuously over time into a data subset with stable internal state, providing a data basis for subsequent steps to isolate lighting artifacts by independently modeling within the channel.

[0047] Specifically, since the strip radiation distortion caused by industrial lighting modulation changes continuously with the modulation angle, directly mixing image data with different modulation angles will cause the strip signals to be misaligned and superimposed in space, resulting in aliasing artifacts that are difficult to eliminate. Through the above optimization model, frames with similar modulation angles (i.e., the smallest circumferential distance) are grouped into the same channel. In fact, the continuously changing time modulation process is discretized into several quasi-static lighting state subsets. Within each subset (channel), the radiation distortion pattern of the image remains highly consistent. This consistency provides the necessary control experimental group basis for subsequent steps to use cross-channel differences to identify true and false details.

[0048] It should be noted that the preset number of channels N can be a fixed value (e.g., 4 or 8) or an adaptive strategy (e.g., automatically splitting a channel when the maximum dispersion within a channel exceeds a threshold). Furthermore, to ensure channel effectiveness, if a channel contains too few frames (e.g., less than 3 frames), insufficient to support subsequent geometric reconstruction, the processor can forcibly merge the frames in that channel into neighboring channels or mark them as outliers to exclude them from subsequent calculations. During the calculation process of the circular averaging method described above, it is important to use trigonometric functions to convert angles into two-dimensional vectors. The values ​​are accumulated and then converted back to angles using the arctangent function to avoid periodic errors caused when the arithmetic mean is directly applied to the angle.

[0049] In another embodiment, in addition to the optimization-based iterative clustering method described above, the modulation angle channel can also be constructed using a fixed sector partitioning method. The processor uniformly divides the circumferential space [0, 2π) into multiple sectors of fixed width (e.g., each sector is π / 4), and directly uses the modulation angle markers... The range of sectors falling into a given channel is assigned to its corresponding channel. When processor computing resources are extremely limited, or when the ambient lighting modulation frequency is extremely high, resulting in an approximately uniform modulation angle distribution, the fixed sector partitioning method can eliminate the iterative calculation process. It can quickly complete channel construction with a time complexity. Although the dispersion within its channels may be slightly higher than that of iterative clustering, it provides a better performance balance in scenarios with extremely high real-time requirements, thus adapting to the operating needs of low-power embedded systems.

[0050] Obtain the set of modulation angle channels Original collected geometric observation data (such as depth maps, point clouds, or LiDAR data) and pose transformation data for each frame. For each modulation angle channel c, the processor iterates through the list of channels. For all frames k in the channel, spatial unification processing is performed to map the local observation data to the global coordinate system. Specifically, for any k-th frame within the channel, the processor calculates its global geometric observation data according to the following formula. :

[0051]

[0052] Among them, the global geometric observation data of the k-th frame Equal to the pose transformation matrix of this frame Left-multiply the local geometric observation data of this frame Specifically, the pose transformation matrix It is a 4×4 homogeneous transformation matrix containing rotation and translation vectors, describing the rigid body pose relationship of the sensor coordinate system in the k-th frame relative to a uniform scene coordinate system (such as the robot's initial odometry coordinate system). It originates from the front-end SLAM or odometry system and local geometric observation data. The raw 3D data acquired by the sensor in the k-th frame can be in the form of spatial point vectors corresponding to pixel coordinates in the depth map or LiDAR point clouds, or global geometric observation data. It is a set of three-dimensional point clouds that has been uniformly aligned to the world coordinate system after rigid body transformation;

[0053] Specifically, this step is to establish a unified spatial benchmark. Although the multi-source data are collected from different time points and spatial locations (such as mobile robot end and handheld end) and are subject to different lighting modulation interference, the absolute spatial position of static objects (such as walls and equipment) in the scene is constant. By using a high-precision pose transformation matrix to unify all local observations to the same global coordinate system, it is ensured that the data from different channels are observations of the same object when performing geometric fusion in the future, providing the necessary spatial consistency basis for the alignment and comparison of multi-scale geometry.

[0054] After spatial unification is completed, the processor constructs a coarse geometric fragment for each channel c using global observation data from all frames within that channel. Specifically, a voxelized mesh fusion method is used, and coarse voxel resolution parameters are set. (For example, if the voxel side length is 5cm), the processor generates coarse geometry based on the following mapping relationship:

[0055]

[0056] Among them, coarse geometric segments It is achieved by using all global geometric observation data belonging to the c-th channel. Mapped to a coarse voxel resolution In a 3D voxel mesh, an isosurface model is extracted by spatial downsampling or weighted averaging of the point cloud within each voxel. Specifically, the coarse-level voxel resolution... It is a relatively large spatial scale parameter, and its value is usually set to be greater than the width of the ambient lighting strip, representing a coarse geometric segment. The output is a 3D mesh model, which represents the spatial topology of macroscopic structures in the scene (such as wall contours, ground flatness, and the outlines of large machinery) under this lighting channel condition. For the voxel mesh fusion function: First, establish a side length of The three-dimensional mesh space is then traversed, the input point cloud is traversed, the number of points falling into each mesh cell and the centroid coordinates are counted, and finally, effective voxels containing a sufficient number of points are retained, and the centroids of adjacent effective voxels are connected to generate a triangular mesh surface.

[0057] Specifically, large-scale voxel meshes are used as spatial low-pass filters. Since the strip radiation distortion caused by industrial lighting usually manifests as high-frequency geometric fluctuations in space (the virtual bumps caused by the lighting strips have small amplitudes and change rapidly, i.e., lighting strip artifacts), by using lower resolution for fusion, these tiny geometric noises caused by illumination modulation can be smoothed out or ignored, ensuring that the generated coarse geometry can accurately capture the rigid skeleton of the scene. Even if the detailed features of different channels differ greatly due to different illumination, the coarse geometry still maintains a high degree of consistency in the global topology, thereby ensuring the structural stability in the multi-scale reconstruction process.

[0058] After establishing the coarse-level structure, the processor further mines the high-frequency details of the scene, constructing higher-resolution fine-level geometric fragments for each channel c. The processor sets the fine-level voxel resolution parameters. (For example, the voxel side length is 0.5cm), and a high-precision fusion algorithm is used to generate fine-grained geometry:

[0059]

[0060] Among them, fine-level geometric segments This is achieved by mapping the data belonging to the c-th channel to a resolution of A fine 3D surface model reconstructed from a high-precision mesh while preserving high-frequency spatial features; specifically, a fine-level voxel resolution. It is a tiny spatial scale parameter, its value being smaller than or close to the width of the lighting strip, to ensure that subtle geometric undulations and fine geometric fragments can be distinguished. This represents the fine morphology of the scene surface under a specific lighting channel condition. It's particularly noteworthy that this model not only records realistic textures (such as screw holes and equipment nameplates) but also fully captures the specific lighting stripe features of that channel (i.e., the artificial geometric undulations caused by the lighting stripes). For fine-grained fusion functions: at each resolution of The small voxel center is calculated, the directed distance from the center to the nearest observation point is calculated, and the multi-frame observation data is fused using a weighted average strategy. Finally, the zero level set isosurface is extracted as a high-precision geometric surface using the moving cube algorithm.

[0061] Specifically, this step follows the principles of channel isolation and detail preservation, grouping items into the same channel. The image frames have consistent modulation angle markers, meaning they record static stripe patterns. Through high-resolution independent reconstruction, these stripes are treated by the algorithm as a fixed texture feature and are clearly preserved. Instead of aliasing or blurring due to mixing with data from other channels (different strip positions), N parallel geometric versions containing different strip fingerprints are generated, providing a unique differentiated data source for subsequent steps to accurately identify and remove illumination artifacts by comparing the fine geometric differences between different channels.

[0062] It should be noted that when performing the above geometric fusion, if the input geometric observation data... For depth maps, the truncated symbolic distance field algorithm is preferred for fusion because it can effectively handle observation noise and generate continuous isosurfaces. If the input is a LiDAR point cloud, the probabilistic occupancy grid or moving least squares surface reconstruction algorithm can be used. In addition, the resolution settings for coarse and fine levels should be adaptively adjusted according to the actual industrial lighting frequency and camera parameters. The general principle is that the coarse voxel resolution should be at least 5 to 10 times that of the fine voxel resolution to achieve effective frequency band separation.

[0063] In another embodiment, to accommodate mobile terminal devices with limited computing resources and storage space (such as embedded industrial control computers), the fine-level geometry construction method can employ sparse hash mapping technology: the processor does not pre-allocate contiguous voxel memory, but instead uses a hash table to store only fine-level voxel blocks containing valid observation data. Furthermore, a view frustum culling strategy can be used, culling only in coarse-level geometry. Within a certain normal distance range of the surface (e.g., ±3×) This method performs detailed voxel allocation and updates, avoiding unnecessary calculations and storage for large open areas in the scene, reducing memory usage from cubic to quadratic levels, and thus significantly improving the real-time performance of the algorithm on low-computing-power platforms.

[0064] By utilizing multiple independent, detailed geometric segments, a cross-channel consistency verification mechanism is established. Reconstruction results under different illumination modulation states are used as control experimental groups, and the geometric divergence at each point in space is calculated. This allows for precise identification and quantification of real details and illumination stripe artifacts. The specific process is as follows:

[0065] Obtain the set of fine-grained geometric fragments for all channels. To perform point-by-point comparison, the processor needs to discretize and sample these geometric fragments in a unified three-dimensional space: the processor traverses any position z in the reconstruction space (this position can be the center coordinates of the fine-level voxel mesh or the vertex coordinates of the target reconstruction mesh), and for each position z, the processor queries the geometric feature quantities of N fine-level geometric fragments at that location. :

[0066]

[0067] Wherein, the feature sampling set at position z It consists of N scalar elements, where the c-th element This represents the geometric property value of the fine-grained geometric segment of the c-th channel at position z;

[0068] Specifically, geometric attribute values The preferred method is to use the truncated symbolic distance value, that is, the directed distance from the query position z to the reconstructed surface of the c-th channel. If the c-th channel does not generate effective geometry at position z due to occlusion or field of view limitation, the sampled value of the channel is invalid, the channel is marked as an invalid channel and will not participate in subsequent calculations, while the valid channels are retained.

[0069] After acquiring the sample set, the processor calculates the cross-channel divergence at position z. This metric quantifies whether the geometry at a given location remains stable when external lighting conditions change. The processor calculates this using the following variance formula:

[0070]

[0071] Among them, the cross-channel divergence at position z It equals the statistical variance Var of all effective geometric features at that point, specifically: first, calculate the arithmetic mean of all effective channel sample values. Then calculate the sample value for each channel. The squares of the differences from the average value are summed and then divided by the number of effective channels. Specifically, cross-channel divergence It is a nonnegative scalar;

[0072] If cross-channel divergence Approaching 0 means that regardless of how the ambient lighting is modulated (i.e., regardless of which channel it belongs to), the reconstructed surface is in the same position. This indicates that the geometric undulations at that location are inherent to the scene's structure (such as the unevenness of walls or the edges of devices), because real objects do not move with the flickering of the lighting frequency.

[0073] If cross-channel divergence A large value means that the reconstructed surface position has shifted significantly under different lighting channels. This indicates that the undulation at this point is caused by the rolling shutter strip, because the position of the strip will shift in phase in space as the modulation angle changes (i.e., the false geometry is moving).

[0074] Based on the calculated cross-channel divergence This nonlinearly maps it to fine-grained credibility weights. To guide subsequent fusion, the mapping adopts a negative exponential Gaussian function model:

[0075]

[0076] Among them, the fine-grained confidence weight at position z It is equal to the power of the natural constant e, where the exponent is the cross-channel divergence degree. The negative value divided by twice the sensitivity parameter The square of;

[0077] Specifically, sensitivity parameters Used to control the suppression intensity of strip noise, its value is usually related to the average amplitude of the ambient lighting strip, and the fine-grained confidence weight. The value range is [0,1];

[0078] When cross-channel divergence When the value is extremely small (high consistency), the fine-grained credibility weight is applied. A value close to 1 indicates that the detail is highly credible and should be retained.

[0079] When cross-channel divergence When the value is large (low consistency), fine-grained credibility weight A rapid decay to near 0 indicates that the detail is a lighting artifact and should be suppressed or removed.

[0080] The final output of fine-grained credibility weights It is a three-dimensional scalar field with the same resolution as the fine-level voxel mesh, which labels each voxel point in space with a true or false label.

[0081] Specifically, in traditional single-frame reconstruction, it is impossible to distinguish whether the brightness variations in an image originate from object texture or lighting stripes. However, this invention utilizes the essential difference between the two: the geometric properties of the real structure are spatiotemporally invariant, while the geometric properties of stripe artifacts are modulation angle dependent. This is achieved by calculating the cross-channel variance. In fact, it detects the sensitivity of geometric features to lighting conditions, thereby automatically and unsupervisedly identifying strip noise hidden in complex textures and generating pixel-level weight masks. This allows for the accurate removal of false strip ripples without losing real high-frequency details (such as screw textures), which is superior to traditional smoothing filtering algorithms.

[0082] It should be noted that in spatial sampling, if the number of effective channels at a certain location z is... Too few (e.g., less than 3) are insufficient to support statistical significance, and the processor will remove the weights at that point. Forced to 0 or a lower default value to avoid misjudgment due to insufficient data. In addition, to improve computational efficiency, the bifurcation degree calculation can be performed only in a narrow band region near the coarse geometry surface.

[0083] In another embodiment, to further improve sensitivity to minute surface normal perturbations, geometric features are... Besides distance values, it can also be surface normal vectors. The processor calculates the normal vector of each channel at position z, and the corresponding bifurcation degree. The calculation is the normal vector dispersion (e.g., 1 minus the average normal vector magnitude). Some high-frequency stripes may have small deviations in distance, but they may have drastic reversals in local surface orientation. The weight field based on the normal vector can more sensitively capture and eliminate this kind of shallow relief strip noise.

[0084] Get the set of coarse geometric fragments for all channels and set of fine-level geometric fragments To perform multi-channel fusion, the processor first establishes a unified target voxel mesh that surrounds the entire scene. For any voxel position z in this mesh space, the processor needs to convert the input explicit mesh into an implicit numerical representation. Specifically, the processor performs a signed distance lookup operation for each channel c:

[0085]

[0086]

[0087] Wherein, the coarse symbol distance value of the c-th channel at position z Equal to the distance from coordinate point z to the coarse geometric segment mesh. The directed Euclidean distance to the nearest surface; similarly, the fine-grained signed distance value. Equal to the coordinate point z to the fine-level geometric segment mesh The directed Euclidean distance to the nearest surface;

[0088] Specifically, the core of this step lies in data mode conversion: since the geometric mesh is composed of discrete triangular patches, it is difficult to directly perform voxel-level weighted operations. Through the above distance query operation, the geometric shape is quantified into a continuously changing distance field value in space, thus providing a unified basis for subsequent algebraic averaging and weighted fusion.

[0089] After obtaining the coarse-level symbolic distance values ​​for all channels, since coarse geometry mainly reflects the macroscopic rigid skeleton of the scene (such as walls and ground), and its structure does not change with illumination modulation, all channels have high consistency at the coarse scale. The processor uses a full-channel arithmetic averaging strategy to generate a globally unified coarse-level fundamental distance field, specifically calculating the final coarse-level distance values ​​according to the following formula. :

[0090]

[0091] In the formula, the symbolic distance value of the final coarse-level basic model Equals the coarse sign distance value of all N channels at position z The summation divided by the total number of channels N, specifically, by using an arithmetic mean to fuse the macroscopic structural information of all channels and multi-channel averaging, can effectively offset the random measurement noise in a single observation, generating a coarse-level basic model with a smooth surface, closed structure, and extremely high robustness. (Depend on (Definition), providing a reliable geometric attachment surface for subsequent detail blending;

[0092] After establishing the basic framework, fine-grained credibility weights are used. The processor performs weight-driven selective fusion of the fine-level symbol distance values ​​for each channel to distinguish between true details and stripe artifacts. For any position z, the processor calculates the final fine-level symbol distance value according to the following formula. :

[0093]

[0094] Among them, the symbolic distance value of the final refinement model Equal to the fine-grained sign distance value of all channels And the fine-grained credibility weight at this position The sum of the products is then divided by the sum of all channel weights;

[0095] Furthermore, to prevent geometric holes from appearing in regions where all channel weights are extremely low, this embodiment employs a coarse-level backoff strategy: if the total weight at a certain position... If the data is below a preset confidence threshold (e.g., 0.1), it indicates that all fine-grained geometry at that location represents high-risk stripe artifacts or invalid data. In this case, the processor will directly... That is, to force the use of coarse geometric numerical substitution without artifacts;

[0096] Specifically, the symbolic distance value of the final refinement model The generation implements an intelligent fusion mechanism for on-demand writing. In real detail areas (such as device nameplates and texture bumps), due to the geometric consistency of the reconstruction results of each channel, the calculated weights... The high quality of the fusion results fully preserves these high-frequency features, achieving detail enhancement. However, in areas with stripe artifacts, phase deviations exist in the reconstruction results of each channel. Extremely low, the geometric contribution of this channel is strongly suppressed, thus smoothing out the false fluctuations caused by the rolling shutter and solving the problem that traditional methods are difficult to remove stroboscopic stripes;

[0097] The processor uses isosurface extraction algorithms (such as the moving cube algorithm) to extract the fine-grained symbolic distance field from the final calculated value. Extract the zero-level set surface to generate the final multi-scale 3D scene model. The final output model By integrating the skeletal stability of the coarse-level model with the detail realism of the fine-level model, this invention successfully eliminates the strip-like geometric artifacts caused by the coupling of rolling shutter and industrial lighting modulation while retaining sub-millimeter-level industrial details through the above-mentioned fusion logic, and outputs a high-fidelity digital twin model.

[0098] The above embodiments are only used to illustrate the technical methods of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical methods of the present invention without departing from the spirit and scope of the technical methods of the present invention.

Claims

1. A multi-scale scene reconstruction method based on multi-source data fusion, characterized in that, include: The row direction statistical signal of each frame in the image frame sequence is extracted, and the modulation angle mark representing the relative phase position in the ambient lighting change cycle at the moment of image acquisition is calculated. The image frame sequence is divided into multiple modulation angle channels based on modulation angle markers; By using pose data, image frames in each modulation angle channel are unified to the global coordinate system, and for each modulation angle channel, coarse geometric segments representing the macroscopic topology of the scene and fine geometric segments containing lighting strip features are constructed. For the corresponding position in the reconstructed space, after obtaining the geometric feature quantity of the fine geometric segment of each modulation angle channel at that position, the cross-channel divergence is calculated, and a fine confidence weight is generated to quantify and distinguish between real details and illumination strip artifacts. Generate a coarse-level basic model based on coarse-level geometric fragments generated from modulation angle channels; Fine-grained confidence weights are used to fuse the fine-grained geometric segments of all modulation angle channels, and combined with the coarse-grained basic model to output a multi-scale 3D scene model.

2. The multi-scale scene reconstruction method based on multi-source data fusion according to claim 1, characterized in that, For each frame in the image frame sequence, calculate the arithmetic mean of the gray values ​​of all pixels in each row of the frame to generate a one-dimensional row mean signal.

3. The multi-scale scene reconstruction method based on multi-source data fusion according to claim 2, characterized in that, include: Frequency domain analysis is performed on a one-dimensional row mean signal to determine the spatial angular frequency corresponding to the maximum spectral response, and a linear regression model containing cosine and sine terms is constructed. The least squares method is used to fit the one-dimensional row mean signal, and the first coefficient of the cosine term and the second coefficient of the sine term are solved. The phase angle determined by the first and second coefficients is calculated using a two-parameter arctangent function, and this phase angle is used as the modulation angle marker.

4. The multi-scale scene reconstruction method based on multi-source data fusion according to claim 1, characterized in that, include: Initialize a preset number of channel corner centers and perform iterative clustering steps, as follows: Calculate the circumferential distance between each modulation angle mark and the center of each channel angle, assign each modulation angle mark to the channel corresponding to the channel angle center with the smallest circumferential distance, calculate the average angle of all modulation angle marks in each channel, and use it as the updated channel angle center; Repeat the iterative clustering steps until the preset convergence condition is met to obtain multiple modulation angle channels after partitioning.

5. The multi-scale scene reconstruction method based on multi-source data fusion according to claim 1, characterized in that, include: Set the coarse voxel resolution parameters and establish a global 3D mesh space based on the coarse voxel resolution parameters; For each modulation angle channel, all image frames in that channel are traversed, and the global geometric observation data of each frame is calculated using the pose data. The global geometric observation data is a set of three-dimensional point clouds that are uniformly aligned to the world coordinate system after rigid body transformation. The voxelized mesh fusion method is used to map all global geometric observation data belonging to the channel to a three-dimensional mesh space, count the number of points falling into each mesh cell and the centroid coordinates, retain the effective voxels containing a sufficient number of points and connect the centroids of adjacent effective voxels to generate the coarse geometric segment.

6. The multi-scale scene reconstruction method based on multi-source data fusion according to claim 5, characterized in that, include: Set the fine voxel resolution parameter, the value of which is smaller than the width of the ambient lighting strip; For each modulation angle channel, a high-precision mesh based on fine-level voxel resolution parameters is constructed; Calculate the directed distance from the center of each tiny voxel in the high-precision grid to the nearest observation point, and fuse multiple frames of observation data using a weighted averaging strategy; The zero-level set isosurface is extracted using the moving cube algorithm to generate fine-grained geometric segments containing illumination strip features.

7. The multi-scale scene reconstruction method based on multi-source data fusion according to claim 6, characterized in that, include: For each location in the reconstruction space, the truncated symbol distance value of the fine geometric segment of each modulation angle channel at that location is obtained. The truncated symbol distance value is the directed distance from that location to the nearest surface of the fine geometric segment. Using the truncated symbol distance value as the geometric feature quantity, calculate the statistical variance of the geometric feature quantity of each modulation angle channel at the position, and use the statistical variance as the cross-channel divergence degree at that position.

8. The multi-scale scene reconstruction method based on multi-source data fusion according to claim 7, characterized in that, include: For each location in the reconstructed space, the exponent of the natural constant is calculated, where the exponent is the negative value of the cross-channel divergence at that location divided by twice the square of the preset sensitivity parameter, to obtain the fine-grained confidence weight at that location, and the value of the fine-grained confidence weight ranges from 0 to 1.

9. The multi-scale scene reconstruction method based on multi-source data fusion according to claim 8, characterized in that, include: For each location in the reconstruction space, obtain the coarse truncation symbol distance value of the coarse geometric segment of each modulation angle channel at that location; A full-channel arithmetic mean strategy is adopted to calculate the arithmetic mean of the coarse-level truncated symbol distance values ​​of all modulation angle channels at the specified position, and the arithmetic mean is used as the coarse-level base distance value at that position. A globally unified coarse-level distance field is constructed from the coarse-level distance values ​​at all locations in the reconstructed space, and this globally unified coarse-level distance field is used as the coarse-level model.

10. The multi-scale scene reconstruction method based on multi-source data fusion according to claim 9, characterized in that, include: For each location in the reconstruction space, calculate the sum of the fine-grained confidence weights of all modulation angle channels at that location; Determine whether the sum is lower than a preset confidence threshold; If the sum is lower than the confidence threshold, the coarse-level foundation distance value of the coarse-level foundation model at that position is directly used as the final symbol distance value at that position. If the sum is greater than or equal to the confidence threshold, then the truncated symbol distance value of each modulation angle channel at that position is weighted and averaged using the fine-level confidence weight, and the resulting weighted average value is used as the final symbol distance value at that position. The zero-level set surface is extracted from the final distance field, which is composed of the final symbolic distance values ​​of all locations in the reconstructed space, using the isosurface extraction algorithm to generate a multi-scale 3D scene model.

Images