Raster 3D reconstruction system and method based on multi-view shadow segmentation

Abstract

The present invention discloses a grating three-dimensional reconstruction system and method based on multi-view shadow segmentation, which belongs to the field of optical three-dimensional reconstruction. The present invention proposes a multi-scale self-attention segmentation network structure, focuses on the use of the self-attention mechanism, introduces a multi-scale structure, and accurately and quickly completes shadow segmentation; the multi-view shadow segmentation three-dimensional reconstruction method of the present invention can achieve the fusion of high-precision phase data, and solves the influence of shadows on the reconstruction results in the existing grating projection three-dimensional reconstruction method. Compared with the existing solutions, the present invention has a strong interference suppression ability and high precision, can accurately identify shadow areas, and remove phase noise caused by shadows, generate high-precision multi-view fusion phase, can meet the factory's accuracy requirements for three-dimensional defect detection of PCB parts, and play a certain role in improving the level of production automation.

Description

Technical Field

[0001] The invention relates to a grating three-dimensional reconstruction system and method based on multi-view shadow segmentation, and belongs to the field of optical three-dimensional reconstruction. Background Art

[0002] Structured light measurement technology, an active optical measurement method, uses a grating to project coded stripes, producing a structured light image of the object's surface, modulated by its height, to calculate three-dimensional topography data. It has been widely studied in fields such as reverse engineering, digital modeling, and computational measurement. However, for objects with complex topography, single-view measurement faces the problem of shadows caused by both the height and the grating's projection angle, making it impossible to accurately obtain complete three-dimensional information about the object. Therefore, 3D reconstruction based on multiple viewing angles is the correct solution.

[0003] Currently, there are numerous grating projection 3D reconstruction methods. In 1983, Japanese scholar Takeda proposed Fourier profilometry (Mitsuo Takeda, Kazuhiro Mutoh. Fourier transform profilometry for the automatic measurement of 3-D object shapes [J]. Applied Optics, 1983, Vol. 22: 3977-3982.). He first used the Fourier transform for phase demodulation and successfully applied it to 3D measurement. The same year, the sine fitting method was proposed. Because it does not require frequency-domain filtering or spectral shifting, it is more efficient than Fourier profilometry. In 1984, V. Srinivasan et al. first proposed the phase-shift method (V. Srinivasan, H.C. Liu, M. Halioua. Automated phase-measuring profilometry of 3-D diffuse objects [J]. Applied Optics, 1984, Vol. 23: 3105-3108). Since then, the phase-shift method has been widely used in grating measurement due to its high precision, fast speed, and simple principle. However, these traditional methods can only achieve 3D reconstruction and cannot solve the shadow problem caused by grating projection. The reconstructed point cloud in the shadow area is scattered, equivalent to noise, resulting in poor reconstruction results.

[0004] To solve the shadow problem, semantic segmentation algorithms are usually used to segment the shadows. Common semantic segmentation algorithms are based on deep learning models driven by big data. The basic idea is to use the backbone network as an encoder to extract high-level semantic information from the image, and gradually restore the feature map to its original resolution in the decoder, thereby completing pixel-level classification through gradient descent. In 2015, Jonathan Long first proposed the use of fully convolutional networks (FCNs) to predict the category of each pixel in "Long J, Shelhamer E, Darrell T. Fully convolutional networks for semantic segmentation [J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2015, 39(4): 640-651." This successfully implemented the application of convolutional neural networks in the field of image segmentation. To solve the task of cell segmentation in the biomedical field, "RONNEBERGER O, FISCHER P, BROX TU-net: Convolutional networks for biomedical image segmentation [C]. Munich: Springer, 2015: 234-241." U-net uses symmetrical downsampling and upsampling components combined with skip-layer connections to fuse information from previous and next layers, constructing a classic and concise shadow segmentation network. "Chen, Liang-Chieh, et al. Encoder-decoder with atrous separable convolution for semantic image segmentation [C]. European Conference on Computer Vision. Berlin, Germany: Springer, 2018: 801-818." The DeepLab series of image segmentation networks uses dilated convolutions with different dilation rates to increase the field of view and proposes a spatial pyramid pooling structure to fuse semantic information from different levels, achieving excellent segmentation results.

[0005] Zhang Likun (Zhang Likun, Research on multi-resolution measurement point cloud fusion method [D]. Xidian University, 2018.), Cui Haihua (Cui Haihua, Jiang Tao, Du Kunpeng, Guo Ronghui, Zhao Anan. Multi-view structured light three-dimensional imaging method based on deep learning pose estimation [J]. Acta Optica Sinica, 2021, 41(17): 72-81.), Ma Suodong (Ma Suodong. Research and application of three-dimensional morphology composite channel measurement technology based on phase recovery [D]. Nanjing University of Science and Technology, 2012.) and others have improved the multi-view camera combined with grating scheme from the aspects of improving multi-resolution point cloud fusion, multi-view point cloud registration and stitching, and other point cloud processing. Figure 3 However, using grating projection to generate the point cloud of the object to be measured at each perspective and then performing registration and fusion will superimpose and amplify the reconstruction errors of each perspective, further resulting in low reconstruction accuracy. Summary of the Invention

[0006] In order to solve the existing reconstruction blind area problem and the error superposition problem caused by the multi-view solution, the present invention provides a grating 3D reconstruction system and method based on multi-view shadow segmentation. The technical solution is as follows:

[0007] The first object of the present invention is to provide a grating 3D reconstruction system based on multi-view shadow segmentation, the system comprising: a grating projection module, an image acquisition module, a shadow segmentation module and a grating 3D reconstruction module;

[0008] The grating projection module includes a plurality of grating projectors, and the grating projectors project grating stripes onto the surface of the object to be measured;

[0009] The image acquisition module is used to acquire a grating image of the surface of the object to be measured, and input the grating image into the shadow segmentation module;

[0010] The shadow segmentation module uses the MobileViT multi-scale self-attention segmentation network to obtain the shadow mask of the grating image at each viewpoint;

[0011] The grating 3D reconstruction module obtains the absolute phase map of each viewing angle through phase shift profilometry and multi-frequency heterodyne method combined with the shadow mask; then, the high-precision absolute phase map is obtained by fusion by solving the phase Gaussian weight, and the complete point cloud data is calculated according to the system calibration parameters.

[0012] Optionally, the MobileViT multi-scale self-attention segmentation network includes: an encoder part and a decoder part;

[0013] The downsampling process of the encoder part includes: first, the input raster image tensor enters the stem part of the network to obtain a 1 / 2 downsampled feature map; then it continues to pass through two MobileNet modules, during which the feature map is downsampled by changing the convolution step size to obtain a 1 / 4 downsampled feature map; at this time, the network has a parallel structure, forming a downsampling branch and a downsampling main path;

[0014] The 1 / 4 downsampled feature map is passed to the decoder part through the downsampling branch to provide contour boundary information for the decoding process; on the other hand, features are continuously extracted through multiple serially connected MobileNet modules on the downsampling main path to achieve 1 / 8 downsampling; each subsequent downsampling uses the parallel structure to pass the downsampled feature map into the decoder part through the downsampling branch, and at the same time, the MobileViT module on the downsampling main path extracts the downsampled feature map that takes into account both local feature information and global attention until the required downsampling result is obtained and input into the decoder part;

[0015] The decoder part is used to restore the deep semantic feature information extracted by the encoder part back to the original input resolution in spatial scale. The decoder part restores the feature maps of different scales output by the downsampling branch of the encoder part during the upsampling process.

[0016] Optionally, the downsampling branch further includes: an efficient dilated pyramid attention segmentation module;

[0017] The process of processing feature maps of the efficient dilated pyramid attention segmentation module includes:

[0018] First, after the separation and splicing operation, the dilated convolution and pooling operations with different dilation rates in the DeepLabV2 network are used to perform convolution and pooling on the different sub-channels separated from the feature map. While not changing the convolution kernel parameters, different receptive fields are observed to obtain multi-scale feature map information.

[0019] Next, in order to increase the exchange of feature information between sub-channels and prevent the excessive learning of a single feature, the feature map is subjected to the channel mixing Channel Shuffle operation in ShuffleNet.

[0020] The SE module is then used to extract the attention weights of different sub-channels in the multi-scale feature map.

[0021] Optionally, the phase fusion method of the grating 3D reconstruction module includes:

[0022] Step 1: Perform bilateral filtering on the absolute phase image of each view to eliminate phase noise while retaining the normal edge of the object change;

[0023] Step 2: Determine the basic phase weight of each viewpoint based on the calibrated reprojection error, and arrange and splice the basic phase weights in descending order on the channel dimension to convert the two-dimensional phase map into a three-dimensional phase tensor. The channel dimension of the three-dimensional phase tensor is the view dimension of the multi-view projection;

[0024] Step 3: For each filtering plane of the three-dimensional phase tensor, select the filter kernel center at the view angle with the smallest reprojection error. For the phase center position of the phase map at that view angle and the phases at other positions within the upper and lower ranges of the filter window size, calculate the joint Gaussian weights according to the phase space proximity and phase value similarity, and process all view angles in turn to obtain the fused phase value.

[0025] Step 4: Move the filtering plane along the width direction of the three-dimensional phase tensor, filter and fuse the phase information between different perspectives in all width dimensions, and thus obtain the final single-channel absolute phase map;

[0026] Step 5: Based on the single-channel absolute phase image and combined with the system calibration parameters, high-precision point cloud information of the object to be measured is calculated.

[0027] Optionally, the calculation process of the fused result phase value includes:

[0028]

[0029]

[0030]

[0031]

[0032] Where g(i) is the output phase value of the i-th row of the filter plane fusion of the four perspective dimensions; m is the row where the filter kernel center is located; c is the sequence number of the current filter channel, and all channels are weighted and summed so that the current phase result is affected by the phase of this perspective and partially modulated by the phases of other perspectives; f(i) is the original phase value of the i-th row; dist(i,m) is the phase space proximity factor, which decreases with the increase of the Euclidean distance between the phase point and the center point; phase(i,m) is the phase value similarity factor, which decreases with the increase of the phase change value; σ p Controls the phase space proximity attenuation factor, σ d Control the phase similarity factor; the weighting coefficient w(i,m) is the product of the phase space proximity factor and the phase value similarity factor, so that the phase result is modulated by both the phase difference value and the spatial position.

[0033] Optionally, the grating projection module includes: 4 grating projectors.

[0034] Optionally, the MobileNetV3 module is adopted in the MobileViT.

[0035] A second object of the present invention is to provide a grating 3D reconstruction method based on multi-view shadow segmentation, which is implemented using the above-mentioned grating 3D reconstruction system based on multi-view shadow segmentation, comprising:

[0036] Step 1: The grating projection module projects the grating stripes onto the surface of the object to be measured;

[0037] Step 2: The image acquisition module acquires grating images of the surface of the object to be measured projected by different grating projectors;

[0038] Step 3: Use the shadow segmentation module to segment the multi-view grating image obtained in step 2 to obtain the shadow mask of the grating image at each viewing angle;

[0039] Step 4: Use the raster 3D reconstruction module to calculate and obtain high-precision point cloud data of the object to be measured.

[0040] The beneficial effects of the present invention are:

[0041] To address the shadow problem, this paper proposes a multi-scale self-attention segmentation network architecture. This architecture leverages the self-attention mechanism and introduces a multi-scale structure to achieve precise and rapid shadow segmentation. Furthermore, using a multi-grating multi-view architecture, a joint Gaussian weighting algorithm based on channel filtering is designed to achieve high-precision phase data fusion.

[0042] To address the issue of reconstruction blind spots and the limitations of the aforementioned multi-view solutions, the present invention's multi-view shadow segmentation 3D reconstruction method achieves high-precision phase data fusion, resolving the impact of shadows on the reconstruction results in existing grating projection 3D reconstruction methods. Furthermore, compared to existing solutions that directly utilize Gaussian models for phase fusion, which suffer from poor interference suppression and insufficient accuracy, the present invention accurately identifies shadow areas and removes phase noise caused by shadows, generating highly accurate multi-view fusion phase data.

[0043] The present invention has a wide range of applications, especially in solving the shadow problem during the three-dimensional reconstruction of target components on PCBs. It can reconstruct part of the shadow area through a four-projector structure, thereby improving the overall three-dimensional reconstruction accuracy. It has good robustness and can meet the basic requirements of factories for partial three-dimensional defect detection of PCBs, playing a certain role in improving the level of production automation. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0045] Figure 1 3D reconstruction strategy diagram of a multi-view grating according to an embodiment of the present invention.

[0046] Figure 2 This is a shadow segmentation network model diagram of the present invention.

[0047] Figure 3 This is a diagram of the efficient hollow pyramid attention segmentation module of the present invention.

[0048] Figure 4 Schematic diagram of phase fusion of the present invention.

[0049] Figure 5 It is a phase fusion processing flow chart of the present invention.

[0050] Figure 6 It is a schematic diagram of the viewing angle dimension of the present invention.

[0051] Figure 7 It is a schematic diagram of the phase fusion processing of the present invention.

[0052] Figure 8 It is the shadow segmentation result diagram of the present invention. DETAILED DESCRIPTION

[0053] To make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.

[0054] Example 1:

[0055] This embodiment provides a grating 3D reconstruction system based on multi-view shadow segmentation. Figure 1 ,The system includes: a grating projection module, an image acquisition module, a shadow segmentation module and a grating 3D reconstruction module;

[0056] The grating projection module includes a plurality of grating projectors, which project grating stripes onto the surface of the object to be measured;

[0057] The image acquisition module is used to acquire the grating image of the surface of the object to be measured and input the grating image into the shadow segmentation module;

[0058] The shadow segmentation module uses the MobileViT multi-scale self-attention segmentation network to obtain the shadow mask of the grating image at each viewpoint;

[0059] The grating 3D reconstruction module uses phase-shift profilometry and multi-frequency heterodyning combined with shadow masking to obtain the absolute phase map of each viewing angle. It then completes the fusion by solving the phase Gaussian weight to obtain a high-precision absolute phase map, and calculates the complete point cloud data based on the system calibration parameters.

[0060] Example 2:

[0061] This embodiment provides a grating 3D reconstruction system and method based on multi-view shadow segmentation.

[0062] To solve the shadow problem caused by the height of the object to be measured and the grating projection angle, the present invention proposes a multi-view grating 3D reconstruction system. The specific system structure diagram is as follows: Figure 1 shown.

[0063] The grating 3D reconstruction system based on multi-view shadow segmentation consists of two main components: the first is the shadow segmentation module. This measurement system consists of four grating projection modules in conjunction with a central camera, each projecting a pre-loaded sinusoidal stripe. The central camera captures the digital image and feeds it into the improved MobileViT multi-scale self-attention segmentation network in this embodiment to obtain shadow masks for each viewpoint. The second is the grating 3D reconstruction module. This module uses phase-shift profilometry and multi-frequency heterodyning to combine the mask to obtain an absolute phase map for each viewpoint. This map is then fused by solving the phase Gaussian weights to obtain a high-precision absolute phase map, and the complete point cloud data is calculated based on the system calibration parameters.

[0064] Before performing grating projection on the object to be measured, the grating projector and camera must be calibrated first. The camera is used to collect images of the checkerboard calibration plate in different poses for analysis and calculation to obtain the intrinsic parameters of the camera, the intrinsic parameters of the projector, and the spatial relationship parameters of the camera and projector (which is a homogeneous transformation matrix). The reprojection error is a parameter to measure the calibration effect. The smaller the error, the higher the calibration accuracy.

[0065] After the system calibration is completed, the camera is used to collect images of the surface of the object to be measured under different grating projections to obtain grating images under different views, and then these grating images are shadow segmented.

[0066] A robust shadow segmentation network model of this embodiment has a specific structure as follows Figure 2 As shown, it includes two parts: encoder and decoder:

[0067] For the encoder, this embodiment no longer uses traditional convolutional neural networks such as VGG, ResNet, and DenseNet as the backbone network. Instead, it adopts a self-attention-based visual transformer. This extracts local image features while also observing global information, thereby better extracting deep semantic features. Furthermore, considering that the model should be as lightweight as possible to balance speed and accuracy, this embodiment uses MobileViT as the backbone network.

[0068] First, the input raster image tensor enters the stem part of the network to obtain a 1 / 2 downsampled feature map. Then it continues to pass through two MobileNet modules. In this process, the feature map is downsampled by changing the convolution step size to obtain a 1 / 4 downsampled feature map. At this time, the network has a parallel structure. One part is passed to the decoder to provide clear contour boundary information; the other part continues to extract features through multiple serially connected MobileNet modules and achieve 1 / 8 downsampling. Each subsequent downsampling uses a parallel structure, and the main path passes through the MobileViT module to extract the downsampled feature map that takes into account local feature information and global attention until 1 / 32 downsampling is completed and finally enters the decoder. Unlike the original MobileViT, this embodiment replaces all MobileNetV2 modules with MobileNetV3 modules. By adding the SE channel attention module, the model effect is improved while the amount of computation and parameters remain basically unchanged.

[0069] The decoder part is often the focus of improvement in the segmentation network. The decoder's responsibility is to restore the deep semantic feature information extracted by the encoder back to the original input resolution in spatial scale, so as to calculate the loss function with the segmentation label; but multiple downsampling and then upsampling will cause the fine size and position information of the feature map to be lost. The shadow segmentation network of this embodiment adopts a network parallel branch structure at 1 / 4, 1 / 8, 1 / 16 and 1 / 32 downsampling respectively. The main channel continuously extracts feature information through convolution and Transformer modules, while the branches output feature maps of different scales respectively to pass fine size and position information to the decoder part.

[0070] If these feature maps are directly upsampled to the same size and then spliced ​​between channels, although the shallow size and position information can be partially utilized, the convolution operation can also waste the relevant properties of increasing and decreasing feature dimensions and extracting features of different sizes through different receptive fields. Therefore, in order to better utilize deep semantic information and shallow fine boundary information and integrate multi-scale features, this embodiment proposes an efficient Atrous Pyramid Split Attention module (APSA). The module structure is as follows: Figure 3 shown.

[0071] This module first undergoes a Split and Concat (SPC) operation, and uses the dilated convolution and pooling operations with different dilation rates in the DeepLabV2 network to perform convolution and pooling on the different sub-channels separated from the feature map, observing different receptive fields while not changing the convolution kernel parameters to obtain multi-scale feature map information. Next, in order to increase the exchange of feature information between sub-channels and prevent the transitional learning of a single feature, the feature map is subjected to the channel mixing Channel Shuffle operation in ShuffleNet. The SE module is then used to extract the attention weights of different sub-channels in the multi-scale feature map, which not only allows the network to pay more attention to key channel information, but also establishes longer-range channel dependencies by changing the sub-channel dimensions. In this embodiment, the branch outputs of the 1 / 8, 1 / 16 and 1 / 32 downsampling scales, as well as the main output of the 1 / 32 downsampling module through the MobileViT module are all connected to the APSA module to improve the network's reuse of multi-scale feature information.

[0072] The multi-view phase fusion method of this embodiment specifically includes:

[0073] like Figure 4 As shown in the figure, for each pixel position of the phase image, different weights are assigned according to the calibrated reprojection errors of each perspective. Models with large reprojection errors are given lower weights, and vice versa. The weighted summation by position directly obtains the fused single-channel phase result image. Figure 5 This is a specific flow chart of the phase fusion method of the present invention.

[0074] First, the absolute phase images of each perspective have phase noise caused by light interference or random errors in optical hardware. At the same time, it must be taken into account that the drastic changes in phase values ​​in some modes are actually normal results obtained from the highly modulated grating stripes of the object to be measured and need to be retained. Therefore, bilateral filtering is performed on the absolute phase images of each perspective to eliminate phase noise while retaining the normal edges of object changes. Then, the basic phase weights of each perspective are determined according to the calibrated reprojection error, and the basic phase weights are arranged in descending order on the channel dimension to form a spliced ​​image, thereby converting the two-dimensional phase image into a three-dimensional phase tensor. The channel dimension of the three-dimensional tensor in this embodiment is the perspective dimension of the multi-view projection. The schematic diagram is as follows Figure 6 shown.

[0075] By observing the three-dimensional absolute phase tensor, it is found that for the phase value of a pixel position in a certain channel, not only the phase values ​​of other positions in the viewing angle dimension will affect this position; the phase values ​​of different pixel positions in other viewing angles will also have a considerable impact on this. If there are large fluctuations in the phase of the surrounding positions under this viewing angle, the phase deviation will be introduced into the phase value of this position after filtering. At this time, the more accurate phase values ​​of the surrounding positions in other viewing angle dimensions can be used to reduce the degree of phase deviation at this position. Therefore, this embodiment simultaneously considers the influence of the distance and phase change value of the viewing angle dimension on the weight, and finally fuses them to obtain the final phase result image.

[0076] like Figure 7 This is a 3D phase tensor view of the view filtering plane. The light-colored block in the leftmost diagram represents the center of the filter kernel at the view with the smallest reprojection error. For the phase map at this view, the phases of the filter center and other positions within the filter window are calculated using joint Gaussian weights (Formula 3) based on phase space proximity and phase value similarity (Formulas 1 and 2), respectively. All views are processed sequentially to obtain the fused phase value (Formula 4). Finally, the filter plane is shifted along the width of the 3D tensor, and the phase information between different views (channels) across all width dimensions is filtered and fused, resulting in a final high-precision single-channel absolute phase map.

[0077] Based on the above phase results and combined with the system calibration parameters, high-precision point cloud information of the object to be measured can be calculated.

[0078]

[0079]

[0080]

[0081]

[0082]

[0083]

[0084]

[0085]

[0086]

[0087]

[0088] Where g(i) is the output phase value of the i-th row of the filter plane fusion of the four perspective dimensions; m is the row where the filter kernel center is located; c is the sequence number of the current filter channel, and all channels are weighted and summed so that the current phase result is affected by the phase of this perspective and partially modulated by the phases of other perspectives; f(i) is the original phase value of the i-th row; dist(i,m) is the phase space proximity factor, which decreases with the increase of the Euclidean distance between the phase point and the center point; phase(i,m) is the phase value similarity factor, which decreases with the increase of the phase change value; σ p Controls the phase space proximity attenuation factor, σ d Control the phase similarity factor; the weighting coefficient w(i,m) is the product of the phase space proximity factor and the phase value similarity factor, so that the phase result is modulated by both the phase difference value and the spatial position. c Indicates the reprojection error of the camera calibration in each channel, A c is the camera internal parameter, R c is the rotation matrix in the external parameter; A p is the camera internal parameter, R p is the rotation matrix in the external parameter, and T is the translation matrix in the external parameter. c is the horizontal coordinate in the camera pixel coordinate system, v c is the vertical coordinate in the camera pixel coordinate system, u p is the horizontal coordinate in the projector pixel coordinate system. x, y and z are the coordinates of each point in the point cloud.

[0089] This example addresses the shadow problem by proposing a multi-scale self-attention segmentation network architecture. This architecture leverages the self-attention mechanism and introduces a multi-scale structure to achieve precise and rapid shadow segmentation. Next, using a monocular four-grating multi-view architecture, a joint Gaussian weighting algorithm based on channel filtering is designed to achieve high-precision phase data fusion.

[0090] In order to further demonstrate the beneficial effects of the present invention, an experiment was conducted to test the effectiveness of the algorithm based on a self-built data set according to the 3D reconstruction process designed by the present invention.

[0091] The experimental environment is: CPU: Intel i9-12900K, 32GB of RAM, GPU: NVIDIA RTX3090 with 24GB of VRAM. Operating system: Windows 10 Professional. Programming language: Python. Development platform: VSCode. Convolutional neural network framework platform: PaddlePaddle 2.0.

[0092] Dataset: The shadow segmentation problem studied in the present invention belongs to a specific scenario application, so a self-built segmentation dataset is used for training and testing. The hardware system built by the present invention is used to collect PCB images and produce a segmentation dataset. The samples in the dataset are divided into two segmentation categories: shadows and components. The dataset contains factors such as different ambient light, different grating projection angles, and different PCB boards. A total of 400 samples are collected. In order to increase the diversity of the training set samples, a total of 1,000 segmentation training samples and 200 verification samples are obtained through data enhancement processing such as random flipping, rotation, and translation within a certain range. At the same time, in order to verify the generalization performance of the network model, the present invention again collects 100 samples to form a separate test set. Finally, a total of 1,000 training set samples, 200 verification set samples, and 100 test set samples are obtained.

[0093] like Figure 8 As shown in the figure, a specific effect diagram of the test based on the shadow segmentation network of the present invention is given. According to the shadow segmentation effect diagram, it can be considered that the shadow segmentation network proposed in the present invention can effectively and accurately segment the shadow area caused by height, facilitating multi-view grating 3D reconstruction.

[0094] The effect diagram intuitively shows the shadow segmentation effect of the network proposed in the present invention. In order to more accurately evaluate the segmentation effect, the present invention uses UNet, SegFormer, and DeepLabV3 networks for training on the same training set, among which the SegFormer backbone network uses native MobileViT, and the DeepLabV3 backbone network uses ResNet34; all segmentation networks are tested using the same test set, and the comparative test results are shown in Table 1 below. mIou is the mean intersection-over-union ratio, which represents the ratio of the intersection and union of the two sets of true values ​​and predicted values, and is the most common semantic segmentation evaluation indicator; Acc segmentation pixel accuracy represents the proportion of correctly labeled pixels to the total pixels; Flops represents the computational amount of the neural network; Params represents the number of parameters of the network structure. According to the test data, it can be found that the proposed segmentation network uses less computational effort and maintains smaller network parameters while ensuring segmentation accuracy, and has excellent comprehensive segmentation effect.

[0095] Table 1 Test results comparison test

[0096]

[0097] To verify the reconstruction accuracy of the present 3D reconstruction method, the present invention performed 3D reconstruction on standard components on a PCB board, measured the height data of the standard components, and calculated the error data of the 3D reconstruction system. To obtain more accurate measurement data, the experiment used a method of averaging multiple measurements, performing eight repeated measurements on the component under test. Based on the experimental data, the present method achieved an absolute measurement accuracy of approximately 0.04 mm, demonstrating high 3D measurement accuracy and reconstruction quality.

[0098] Table 2 Measurement data results

[0099]

[0100] Some steps in the embodiments of the present invention may be implemented using software, and the corresponding software program may be stored in a readable storage medium, such as a CD or a hard disk.

[0101] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A grating 3D reconstruction system based on multi-view shadow segmentation, characterized in that: The system includes: a grating projection module, an image acquisition module, a shadow segmentation module and a grating three-dimensional reconstruction module; The grating projection module includes a plurality of grating projectors, and the grating projectors project grating stripes onto the surface of the object to be measured; The image acquisition module is used to acquire a grating image of the surface of the object to be measured, and input the grating image into the shadow segmentation module; The shadow segmentation module uses the MobileViT multi-scale self-attention segmentation network to obtain the shadow mask of the grating image at each viewpoint; The grating 3D reconstruction module uses phase-shift profilometry and multi-frequency heterodyning combined with the shadow mask to obtain absolute phase maps for each viewing angle. It then fuses the images by solving the phase Gaussian weights to obtain high-precision absolute phase maps, and calculates complete point cloud data based on system calibration parameters. The MobileViT multi-scale self-attention segmentation network includes: an encoder part and a decoder part; The downsampling process of the encoder part includes: first, the input raster image tensor enters the stem part of the network to obtain a 1 / 2 downsampled feature map; then it continues to pass through two MobileNet modules, during which the feature map is downsampled by changing the convolution step size to obtain a 1 / 4 downsampled feature map; at this time, the network has a parallel structure, forming a downsampling branch and a downsampling main path; The 1 / 4 downsampled feature map is passed to the decoder part through the downsampling branch to provide contour boundary information for the decoding process; on the other hand, features are continuously extracted through multiple serially connected MobileNet modules on the downsampling main path to achieve 1 / 8 downsampling; each subsequent downsampling uses the parallel structure to pass the downsampled feature map into the decoder part through the downsampling branch, and at the same time, the MobileViT module on the downsampling main path extracts the downsampled feature map that takes into account both local feature information and global attention until the required downsampling result is obtained and input into the decoder part; The decoder part is used to restore the deep semantic feature information extracted by the encoder part back to the original input resolution in spatial scale. The decoder part restores the feature maps of different scales output by the downsampling branch of the encoder part during the upsampling process.

2. The grating 3D reconstruction system based on multi-view shadow segmentation according to claim 1, characterized in that: The downsampling branch also includes: an efficient hole pyramid attention segmentation module; The process of processing feature maps of the efficient dilated pyramid attention segmentation module includes: First, after the separation and splicing operation, the dilated convolution and pooling operations with different dilation rates in the DeepLabV2 network are used to perform convolution and pooling on the different sub-channels separated from the feature map. While not changing the convolution kernel parameters, different receptive fields are observed to obtain multi-scale feature map information. Next, in order to increase the exchange of feature information between sub-channels and prevent the excessive learning of a single feature, the feature map is subjected to the channel mixing Channel Shuffle operation in ShuffleNet. The SE module is then used to extract the attention weights of different sub-channels in the multi-scale feature map.

3. The grating 3D reconstruction system based on multi-view shadow segmentation according to claim 1, characterized in that: The phase fusion method of the grating 3D reconstruction module includes: Step 1: Perform bilateral filtering on the absolute phase image of each view to eliminate phase noise while retaining the normal edge of the object change; Step 2: Determine the basic phase weight of each viewpoint based on the calibrated reprojection error, and arrange and splice the basic phase weights in descending order on the channel dimension to convert the two-dimensional phase map into a three-dimensional phase tensor. The channel dimension of the three-dimensional phase tensor is the view dimension of the multi-view projection; Step 3: For each filtering plane of the three-dimensional phase tensor, select the filter kernel center at the view angle with the smallest reprojection error. For the phase center position of the phase map at that view angle and the phases at other positions within the upper and lower ranges of the filter window size, calculate the joint Gaussian weights according to the phase space proximity and phase value similarity, and process all view angles in turn to obtain the fused phase value. Step 4: Move the filtering plane along the width direction of the three-dimensional phase tensor, filter and fuse the phase information between different perspectives in all width dimensions, and thus obtain the final single-channel absolute phase map; Step 5: Based on the single-channel absolute phase image and combined with the system calibration parameters, high-precision point cloud information of the object to be measured is calculated.

4. The grating 3D reconstruction system based on multi-view shadow segmentation according to claim 3, characterized in that: The calculation process of the fused result phase value includes: Where g(i) is the output phase value of the i-th row of the filter plane fusion of the four viewing dimensions; m is the row where the filter kernel center is located; c is the sequence number of the current filter channel; f(i) is the original phase value of the i-th row; dist(i,m) is the phase space proximity factor, which decreases as the Euclidean distance between the phase point and the center point increases; phase(i,m) is the phase value similarity factor, which decreases as the phase change value increases; σ p Controls the phase space proximity attenuation factor, σ d Control phase similarity factor; weighting coefficient w(i,m) is the product of phase space proximity factor and phase value similarity factor, α c Indicates the reprojection error of the camera calibration in channel c.

5. The grating 3D reconstruction system based on multi-view shadow segmentation according to claim 1, characterized in that: The grating projection module includes: 4 grating projectors.

6. The grating 3D reconstruction system based on multi-view shadow segmentation according to claim 1, characterized in that: The MobileViT uses the MobileNetV3 module.

7. A grating 3D reconstruction method based on multi-view shadow segmentation, characterized in that: The method is implemented by using the grating 3D reconstruction system based on multi-view shadow segmentation according to any one of claims 1 to 6, comprising: Step 1: The grating projection module projects the grating stripes onto the surface of the object to be measured; Step 2: The image acquisition module acquires grating images of the surface of the object to be measured projected by different grating projectors; Step 3: Use the shadow segmentation module to segment the multi-view grating image obtained in step 2 to obtain the shadow mask of the grating image at each viewing angle; Step 4: Use the raster 3D reconstruction module to calculate and obtain high-precision point cloud data of the object to be measured.

Images