High-reflectivity-based three-dimensional point cloud generation method, device, equipment and medium

Abstract

The application relates to a high-reflectivity-based three-dimensional point cloud generation method, device, equipment and medium. The method comprises the following steps: acquiring an initial image of a to-be-measured object under a full-white-field diagram projected by a projection device through an image acquisition device; processing the initial image to obtain a pure high-light mask, a normal diffuse reflection mask and a target brightness attenuation coefficient matrix; modulating a reference image sequence of the projection device based on the target brightness attenuation coefficient matrix to obtain a target projection image; projecting the target projection image through the projection device, and acquiring a target image of the to-be-measured object under the target projection image projected by the projection device through the image acquisition device; and generating a three-dimensional point cloud of the to-be-measured object based on the target image, the pure high-light mask and the normal diffuse reflection mask. The method can ensure efficiency and improve model generation accuracy.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to a method, apparatus, device and medium for generating three-dimensional point clouds based on high reflectivity. Background Technology

[0002] In 3D topography measurement technology based on structured light, objects with high dynamic range (HDR) reflectivity, such as metals and machined parts, are prone to severe overexposure of local areas on the camera target surface, leading to fringe phase truncation or loss, making 3D reconstruction impossible in that area. To address this high-reflectivity overexposure problem, the most relevant existing technologies currently fall into two main categories:

[0003] Multiple exposure fusion: This method involves capturing multiple sequences of stripe patterns with different exposure intensities by varying the camera's exposure time or the projector's global brightness, and then effectively stitching and fusing them pixel by pixel. Essentially, this method trades time for dynamic range, requiring a significant increase in the number of projected and captured images. In practical measurements, this results in an extremely long overall scanning time, failing to meet stringent efficiency requirements.

[0004] The deep learning-based direct overexposure stripe restoration method directly inputs overexposure stripe images with truncated or missing highlights into a neural network (such as a GAN-based image-to-image network), attempting to directly predict or restore the missing phase information and stripe patterns through a generative model. This end-to-end processing approach is a typical black-box model, severely lacking interpretability based on optical physics mechanisms. The stripes restored by the network are essentially generated based on the distribution of training data, rather than physical reflections. Therefore, when the system faces workpieces with unknown materials and complex, unseen geometries, generalization cannot be guaranteed. It is highly prone to generating false stripe textures in overexposure areas and introducing unpredictable depth artifacts, completely undermining the absolute accuracy requirements of industrial measurement. Summary of the Invention

[0005] Therefore, it is necessary to provide a method, apparatus, device, and medium for generating 3D point clouds based on high reflectivity that can improve model generation accuracy while ensuring efficiency, in order to address the above-mentioned technical problems.

[0006] In a first aspect, this application provides a method for generating three-dimensional point clouds based on high reflectivity, the method comprising:

[0007] The initial image of the object under test is acquired by the image acquisition device under the full white field image projected by the projection device.

[0008] The initial image is processed to obtain a pure specular mask, a normal diffuse mask, and a target brightness attenuation coefficient matrix;

[0009] Based on the target brightness attenuation coefficient matrix, the reference image sequence of the projection device will be modulated to obtain the target projection image;

[0010] The target projection image is projected by the projection device, and the target image of the object under test under the target projection image projected by the projection device is acquired by the image acquisition device.

[0011] A 3D point cloud of the object under test is generated based on the target image, the pure specular mask, and the normal diffuse mask.

[0012] In some optional embodiments, processing the initial image to obtain a pure specular mask, a normal diffuse mask, and a target brightness attenuation coefficient matrix includes:

[0013] The initial image is processed by a multi-layered cascaded inverse residual feature extraction network of a pre-trained segmentation model to obtain a high-dimensional feature map.

[0014] The high-dimensional feature map is segmented using a pre-trained segmentation model's segmentation network to obtain a pure specular mask and a normal diffuse mask.

[0015] The high-dimensional feature map is processed using the pre-trained segmentation model's luminance regression network based on the pure specular mask to obtain the target luminance attenuation coefficient matrix.

[0016] In some optional embodiments, the process of obtaining a high-dimensional feature map from the initial image through a multi-layered cascaded inverse residual feature extraction network of a pre-trained segmentation model includes:

[0017] The initial image is dimensionally expanded by the dimension-up expansion layer of the multi-layer cascaded inverse residual feature extraction network.

[0018] The target feature map is obtained by extracting features from the initial image after dimensional expansion through the lightweight spatial filtering layer of the multi-layer cascaded inverse residual feature extraction network.

[0019] The target feature map is dimensionality reduced by the linear dimensionality reduction projection layer of the multi-layered cascaded inverse residual feature extraction network, and then fused with the initial image output by the residual skip connection layer of the multi-layered cascaded inverse residual feature extraction network to obtain a high-dimensional feature map.

[0020] In some optional embodiments, the high-dimensional feature map is processed based on the pure specular mask using a pre-trained segmentation model's luminance regression network to obtain a target luminance attenuation coefficient matrix, including:

[0021] The high-dimensional feature map is processed by the brightness regression network of the pre-trained segmentation model to obtain the global initial brightness attenuation coefficient matrix.

[0022] The target brightness attenuation coefficient matrix is ​​obtained by processing the global initial brightness attenuation coefficient matrix based on the pure specular mask.

[0023] In some optional embodiments, the processing of the initial image to obtain the pure specular mask, normal diffuse mask, and target brightness attenuation coefficient matrix is ​​achieved through a pre-trained segmentation model; the training method of the segmentation model includes:

[0024] The image acquisition device acquires reference images of the full white field image projected onto the sample object under test by the projection device with multiple global brightness coefficients that decrease in gradient.

[0025] Based on each of the reference images, when the gray value of any pixel in each of the reference images under the field of view of the image acquisition device is at the upper limit of the optimal linear response range of the image acquisition device, the global brightness coefficient of the projection device is determined.

[0026] If a pixel in the target reference image is truncated due to grayscale saturation or the global luminance coefficient of the pixel is less than the target value, a pure specular mask is generated based on the pixel, and a normal diffuse mask is generated based on the pure specular mask; the target reference image is the reference image corresponding to the projection of the full white field image with maximum luminance.

[0027] A segmentation model is obtained by training the model based on the reference images, the pure specular mask of the sample, the normal diffuse mask of the sample, and the global luminance coefficient of the sample.

[0028] In some optional embodiments, the step of training a segmentation model based on each of the reference images, the sample pure specular mask, the sample normal diffuse mask, and the sample global luminance coefficients includes:

[0029] The reference image is processed by a segmentation model to obtain a predicted pure specular mask, a predicted normal diffuse mask, and a predicted global luminance coefficient.

[0030] A mask segmentation loss function is generated based on the predicted pure specular mask and the sample pure specular mask;

[0031] A basic loss function is generated based on the predicted global brightness coefficient and the sample global brightness coefficient;

[0032] Based on the predicted global brightness coefficient and the full white field image, the physical space light modulation process is simulated to generate a modulated image. Based on the modulated image and the ideal linear response target image, under the supervision of the sample pure specular mask, a physical response consistency loss function is generated.

[0033] Morphological dilation is performed based on the sample pure specular mask to obtain a transition mask, and a boundary smoothing penalty loss function is generated under the supervision of the transition mask based on the spatial gradient of the predicted global brightness coefficient.

[0034] A target loss function is generated based on the mask segmentation loss function, the basic loss function, the physical response consistency loss function, and the boundary smoothing penalty term loss function;

[0035] Gradient backpropagation is performed based on the target loss function to obtain the trained segmentation model.

[0036] In some optional embodiments, before modulating the reference image sequence of the projection device based on the target brightness attenuation coefficient matrix to obtain the target projected image, the method further includes:

[0037] A second speckle image of the sample object under test is obtained by an image acquisition device under a first speckle image projected by a projection device;

[0038] Based on the first speckle image and the second speckle image, a mapping relationship is established between the field of view of the image acquisition device and the field of view of the projection device;

[0039] Based on the mapping relationship, the target brightness attenuation coefficient matrix is ​​processed to obtain the attenuation coefficient map corresponding to the pure specular mask under the field of view of the projection device;

[0040] The reference image sequence of the projection device is modulated based on the target brightness attenuation coefficient matrix to obtain the target projected image, including:

[0041] Based on the attenuation coefficient diagram, the reference image sequence of the projection device will be modulated to obtain the target projection image.

[0042] Secondly, this application also provides a three-dimensional point cloud generation device based on high reflectivity, the device comprising:

[0043] The acquisition module is used to acquire the initial image of the object under test under the full white field image projected by the projection device through the image acquisition device;

[0044] The segmentation module is used to process the initial image to obtain a pure specular mask, a normal diffuse mask, and a target brightness attenuation coefficient matrix;

[0045] The projection image generation module is used to modulate the reference image sequence of the projection device based on the target brightness attenuation coefficient matrix to obtain the target projection image;

[0046] The projection module is used to project the target projection image through the projection device and to acquire the target image of the object under test under the target projection image projected by the projection device through the image acquisition device.

[0047] The model generation module is used to generate a 3D point cloud of the object under test based on the target image, the pure specular mask, and the normal diffuse mask.

[0048] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the method described in any of the above embodiments.

[0049] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the method described in any of the above embodiments.

[0050] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the method described in any of the above embodiments.

[0051] The aforementioned method, apparatus, device, and medium for generating 3D point clouds based on high reflectivity acquires an initial image of the object under test under a full-white field image projected by a projection device using an image acquisition device. The initial image is processed to obtain a pure specular mask, a normal diffuse mask, and a target brightness attenuation coefficient matrix. Based on the target brightness attenuation coefficient matrix, a reference image sequence of the projection device is modulated to obtain a target projected image. The target projected image is projected through the projection device, and a target image of the object under test under the target projected image is acquired by the image acquisition device. A 3D point cloud of the object under test is generated based on the target image, the pure specular mask, and the normal diffuse mask. This approach, combining single-pass full-white field physical detection with a multi-task deep learning model, clearly decouples the tasks of region segmentation and brightness adjustment. While maintaining extremely high interpretability and absolute measurement accuracy of the optical physical process, it does not add an additional projection phase shift sequence, achieving adaptive attenuation modulation of local luminous intensity. This ensures that overexposed specular areas fall back to the camera's optimal linear response range, fundamentally balancing high measurement efficiency and strong algorithm robustness. Attached Figure Description

[0052] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0053] Figure 1 This is a schematic diagram of a structured light-based 3D point cloud generation system in one embodiment;

[0054] Figure 2 This is a schematic diagram of the framework of a thermal management subsystem integrating zoned active temperature control and air ducts in one embodiment;

[0055] Figure 3 This is a schematic diagram of a heat dissipation module in one embodiment;

[0056] Figure 4 This is a flowchart illustrating a method for generating 3D point clouds based on high reflectivity in one embodiment;

[0057] Figure 5 This is a schematic diagram of the segmentation model in one embodiment;

[0058] Figure 6 This is a structural diagram of a multi-layered cascaded inverse residual feature extraction network in one embodiment;

[0059] Figure 7 Here is a structural diagram of a semantic segmentation model in one embodiment;

[0060] Figure 8 This is a schematic diagram of the sub-region prediction model in one embodiment;

[0061] Figure 9 Here is a block diagram of the attenuation coefficient prediction model in one embodiment;

[0062] Figure 10a This is a schematic diagram of the phase optimization network in one embodiment;

[0063] Figure 10b Here is a diagram of the phase optimization network in another embodiment;

[0064] Figure 10c Here is a structural diagram of the attention weight generation network in one embodiment;

[0065] Figure 11 This is a flowchart illustrating a structured light-based 3D point cloud generation method in another embodiment.

[0066] Figure 12 This is a structural block diagram of a three-dimensional point cloud generation device based on high reflectivity in one embodiment;

[0067] Figure 13 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0068] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0069] It should be noted that the terms "first," "second," etc., used in this application can be used to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from the second element. The terms "comprising" and "having," and any variations thereof, used in this application, are intended to cover non-exclusive inclusion. The term "multiple" used in this application refers to two or more. The term "and / or" used in this application refers to one of the embodiments, or any combination of multiple embodiments.

[0070] The 3D point cloud generation method based on mutual reflection provided in this application can be applied to, for example... Figure 1 The illustrated 3D point cloud generation system based on structured light includes a projection device, an image acquisition device, and a controller. The projection device can be an optical engine, such as a high-precision DMD-based optical engine (in other embodiments, the number of optical engines can also be different, without specific limitations). The projection device projects a target image and K projected images onto the object under test. The image acquisition device can be a camera, such as a binocular camera, such as two high dynamic range industrial cameras (in other embodiments, the number of cameras included in the image acquisition device can also be different). Optionally, the image acquisition device and the projection device maintain microsecond-level hardware synchronization. The image acquisition device is used to acquire an initial image of the object under test projected by the target image, and the target image under the K projected images. The controller can coordinate and schedule hardware timing and execute deep learning model inference. Specifically, the controller is used to execute the 3D point cloud generation method based on mutual reflection of this application.

[0071] The optical engine and camera generate significant heat during continuous operation, causing thermal expansion of the internal lens assembly and CMOS sensor (i.e., "temperature drift"), resulting in micrometer-level shifts in calibration parameters. This not only increases the dark current noise of the sensor but also directly destroys the pixel-level mapping accuracy of the aforementioned algorithm. Therefore, this system not only includes the aforementioned projection device, image acquisition device, and controller but also incorporates a thermal balance management and protection system based on zoned active temperature control and air duct integration. Extreme physical stability ensures the ultimate accuracy at the algorithm level.

[0072] Optionally, the controller also integrates a PID closed-loop temperature control algorithm. In some optional embodiments, combined with Figure 2 As shown, Figure 2 This is a schematic diagram of the framework of a thermal management subsystem integrating zoned active temperature control and air duct in one embodiment. Each projection device and image acquisition device is separately configured with a heat dissipation module and a measurement and coding component. The controller is used to collect the temperature of each projection device and image acquisition device in real time through each measurement and coding component, and control the operation of each heat dissipation module based on each temperature.

[0073] This thermal management subsystem completely solves the temperature drift problem of the optical engine and camera by decoupling on-demand precise temperature control from sealed isolation protection. The specific structure includes an independent closed-loop active temperature control module for each component; that is, each projection device and image acquisition device is equipped with a separate heat dissipation module and measurement / editing components. Specifically, this embodiment eliminates the traditional blind spot in overall heat dissipation. The system constructs physically isolated dedicated heat dissipation ducts for one projection optical engine and two industrial cameras (other numbers may also exist in other embodiments). Each duct consists of a thermally conductive interface material and tightly fitted heat dissipation fins. The controller collects independent temperature sensor data from these three core components in real time and dynamically and independently adjusts the speed of the high-speed DC brushless fan on the corresponding duct based on a PID (proportional-integral-derivative) control algorithm. This mechanism ensures that the optical engine and camera can quickly reach and remain stable within the optimal operating temperature range (without fluctuations) after startup. This eliminates thermally induced accuracy drift from its physical origin.

[0074] In some alternative embodiments, combined with Figure 3 As shown, the heat dissipation module includes a main cavity and an air duct; the test assembly is sealed in the main cavity, one end of the air duct is connected to the main cavity, and the other end is connected to the heat dissipation fins outside the shell. The air duct is used to conduct the heat inside the main cavity to the heat dissipation fins outside the shell, and is cooled by the fan outside the shell.

[0075] To meet the high protection requirements of harsh industrial environments (such as highly reflective metal processing workshops with dust and moisture interference), this system adopts a dual-layer cavity-air duct architecture. The measurement and editing components are completely enclosed within a main cavity with an IP65 protection rating. The heat dissipation air duct acts as a heat-conducting bridge, penetrating the sealed cavity wall and transferring heat from the internal components to the heat dissipation fins located on the outside of the outer shell, where it is cooled by forced convection airflow from an external waterproof fan. This embodiment achieves heat conduction without any exchange of internal and external air, perfectly ensuring the cleanliness of the core optical components while achieving a much higher heat dissipation efficiency than traditional passive cooling.

[0076] In one exemplary embodiment, such as Figure 4As shown, a method for generating 3D point clouds based on high reflectivity is provided, and this method is applied to... Figure 1 Taking the controller in the example, the explanation includes the following steps 402 to 410. Wherein:

[0077] S402: Acquire the initial image of the object under test under a full white field image projected by the projection device using an image acquisition device.

[0078] The object under test can be a metal processing part, such as a right-angle bent painted part, the inner wall of a box, the surface of a battery box, or the concave surface of a car door panel, etc., which are industrial components with complex reflective properties. In other embodiments, the object under test can also be other types of processing parts with high light reflection or deep cavity structures. In this embodiment, the controller controls the projection device to project a full white field image onto the object under test, so that the digital micromirror array of the projection device is in a fully bright state, and the image acquisition device simultaneously performs a single exposure to obtain an initial image, which can be a full white field detection image. The full white field detection map is used to provide energy feedback of the surface of the object under test under the physical limit of luminescence, showing the spatial gradient distribution of the overexposed area of ​​a single specular reflection and the energy spillover.

[0079] S404: Process the initial image to obtain the pure specular mask, the normal diffuse mask, and the target brightness attenuation coefficient matrix.

[0080] In this embodiment, the initial image is processed to obtain a pure specular mask, a normal diffuse mask, and a target brightness attenuation coefficient matrix corresponding to the initial image.

[0081] The normal diffuse reflection region (corresponding to category 0 below) is the region other than the interreflection region and the pure specular region. From a physical optics perspective, the pixels in this region correspond to the area where the projected light enters the camera after one reflection from the surface of the object under test, without optical crosstalk. This region exhibits stable contrast and phase information that conforms to the expected linear distribution.

[0082] Pure specular areas refer to regions on an object's surface where supersaturated specular reflection occurs. From a physical optics perspective, the pixels in this region correspond to areas where, when projected light is reflected from the object's surface, the surface's microscopic normal direction forms a strict specular angle with the incident light (i.e., the angle of incidence equals the angle of reflection). This causes the reflected light intensity to concentrate sharply and exceed the upper limit of the camera sensor's dynamic range, resulting in areas where the pixel grayscale value reaches saturation (such as 255 in an 8-bit image) and phase information calculation fails (singularities or periodic jumps appear).

[0083] In this embodiment, the full white field detection map will be used. The input is fed into a pre-trained segmentation model for forward inference computation. This multi-task deep learning network includes a multi-layered cascaded inverse residual feature extraction network and parallel mask segmentation and brightness regression networks. The multi-layered cascaded inverse residual feature extraction network is used to extract features from the full white field detection map. The parallel mask segmentation network is used to segment the data based on the features extracted by the multi-layered cascaded inverse residual feature extraction network to obtain pure specular masks and normal diffuse masks. Finally, supervision is performed based on the pure specular masks and normal diffuse masks, and the target brightness attenuation coefficient matrix is ​​obtained based on the features extracted by the multi-layered cascaded inverse residual feature extraction network.

[0084] In other embodiments, the initial image can be partitioned to obtain a mutual reflection mask, a pure specular mask, and a normal diffuse reflection mask. The target pattern can be a high-frequency sinusoidal phase-shifting fringe pattern, which not only contains the basic texture modulation information of the object, but also can explicitly amplify and capture the local crosstalk of mutual reflection optical paths and the specular saturation features of single specular reflection.

[0085] In this embodiment, the controller controls the projection device to project the target image onto the object to be measured, and the image acquisition device simultaneously captures an initial image, which can be an initial stripe image. The initial image contains optical feature information such as stripe distortion and contrast changes required for subsequent partitioning.

[0086] In this embodiment, the initial image is partitioned to obtain a mutual reflection mask, a pure specular mask, and a normal diffuse reflection mask. For example, the initial image is partitioned using a pre-trained semantic segmentation model.

[0087] The mutual reflection region is the area where the contrast is significantly reduced, the phase is distorted, or a multi-peak response occurs due to the multiple bounces of light paths between concave cavities or different surface elements on the object surface (i.e., after the light is emitted from the projector, it is reflected by the object surface A to the object surface B, and then reflected into the camera).

[0088] Based on the mutual reflection mask and the initial image, the mutual reflection mask is divided into K sub-regions that do not reflect each other, where K is a positive integer greater than or equal to 1.

[0089] In other embodiments, a pure specular mask, a normal diffuse mask, and K non-reflective sub-regions can be directly obtained by processing the initial image; no specific limitations are imposed here.

[0090] S406: The reference image sequence of the projection device will be modulated based on the target brightness attenuation coefficient matrix to obtain the target projection image.

[0091] The controller generates a reference image for three-dimensional phase calculation based on a phase-shifting algorithm. This reference image can be a standard. A phase-shifted sinusoidal fringe sequence is generated. This standard sinusoidal fringe sequence is modulated pixel-by-pixel by multiplying with the target brightness attenuation coefficient matrix under the projection viewpoint to obtain the target projected image. Under this modulation mechanism, the projection device adaptively reduces the luminous intensity at the physical pixel positions corresponding to the overexposed highlight areas of the object under test, while maintaining full-scale luminous intensity at the pixel positions corresponding to the normal diffuse reflection areas.

[0092] In other embodiments, K projection images are generated based on K non-interfering sub-regions, and one of the projection images is used to simultaneously project the region corresponding to the pure specular mask after attenuation by the attenuation coefficient map, the region corresponding to the normal diffuse mask, and one of the sub-regions, while the other K-1 projection images are used to project the remaining sub-regions respectively.

[0093] This embodiment aims to overcome the bottleneck of the exponentially increased number of frames caused by serial projection in traditional anti-spectral and anti-reflection methods. Based on the decoupled physical mechanism of reflection properties, the normal diffuse reflection area... and pure highlight area The commonality between these two types of non-mutually reflecting regions is that their reflected light rays are directly directed towards the camera target surface without secondary or multiple bounces on the object surface. Therefore, even if these two types of non-mutually reflecting regions are illuminated, they will not become crosstalk light sources that interfere with the fringe imaging of the mutually reflecting regions. Utilizing the orthogonality of optical space, this embodiment constructs a spatiotemporal joint modulation mechanism that integrates continuous local brightness modulation with discrete partitioned spatial isolation within the same projection time sequence.

[0094] The three-dimensional measurement system is designed to adopt... The phase-shifting method, the physical pixel coordinates of the projection device are Traditional separate measurement strategies require separate measurements of the normal diffuse reflectance region. Pure highlight area as well as The total number of images required for independent projection of each mutually reflective sub-region is: open.

[0095] To achieve parallel optimization, a projected time series index is established. In the first Within each projection time slot, construct the global spatial fusion modulation matrix corresponding to each projection time slot. In this way, each projection time slice corresponds to a projection image.

[0096] Among them, due to the normal diffuse reflection area Pure highlight area Unaffected by mutual reflection interference, the solution requires only one complete phase-shifted projection over any given sub-region. To minimize the total number of frames, they can be uniformly scheduled in the first projection time slice ( (inner and mutually reflective sub-regions) Parallel projection; for subsequent time slices ( If so, only the remaining reflexive subregions will be projected. In other embodiments, the normal diffuse reflection area may also be used. Pure highlight area It is scheduled to be projected in parallel with other interreflective subregions in other projection time slices.

[0097] The formula for the global spatial fusion modulation matrix is ​​defined as follows:

[0098]

[0099] In the formula: For the first Each sub-region mask is a non-reflective sub-region with a value of 0 or 1. This is a normal diffuse mask, with a value of 0 or 1; This is a pure specular mask, with a value of 0 or 1; This is a pure specular local brightness attenuation mask that is predicted and transformed to the projection device coordinate system, and its value range is... ; It is an indicator function if and only if the projected time series When the value is 1, The value is 0.

[0100] S408: Project a target image using a projection device, and acquire a target image of the object under test under the target image projected by the projection device using an image acquisition device.

[0101] The projection device projects a set of target projection images that have undergone local downexposure modulation onto the object under test. The target projection images can be a stripe sequence, and the image acquisition device simultaneously acquires images to obtain the target images.

[0102] Optionally, K projection images are projected sequentially by a projection device, and the target image of the object under test under each projection image projected by the projection device is acquired by an image acquisition device. In this embodiment, after generating K projection images, the projection device is controlled to project the corresponding projection image of the object under test within the corresponding projection time slice, and then the target image under each projection image is acquired synchronously by the image acquisition device.

[0103] S410: Generates a 3D point cloud of the object under test based on the target image, pure specular mask, and normal diffuse mask.

[0104] Because the modulated projection beam suppresses the overexposure energy of the pure highlight region from the physical optics source, the pixel response of the highlight region re-acquired by the image acquisition device falls back to the optimal linear dynamic range, avoiding the phenomenon of fringe phase truncation. Based on the acquired high-quality modulated fringe pattern sequence, the phase-shifting algorithm is used to extract the absolute phase of the entire field, and spatial triangulation is performed in combination with the system calibration parameters to output high-precision three-dimensional point cloud data with no flying points and no distortion throughout the field.

[0105] Furthermore, after acquiring each target image, local phase information obtained through different modulation strategies can be calculated separately; specifically, the phase maps acquired and calculated by the system in different projection time slots are spatially integrated. For each pixel in the image... Based on its corresponding reflection attribute mask, extract the corresponding effective phase value. :like Take the solution phase in the conventional projection mode; if Take the phase without overexposure after adaptive brightness adjustment; if According to its partition The corresponding parallel partition projection phase is calculated. Guided by a mask, a phase map covering the entire target field is synthesized. Finally, a 3D point cloud of the object under test is generated based on the target phase map.

[0106] The aforementioned method for generating 3D point clouds based on high reflectivity combines a single full-white field physical detection with a multi-task deep learning model to clearly decouple the tasks of region segmentation and brightness adjustment. While preserving the extremely high interpretability and absolute measurement accuracy of the optical physical process, it achieves adaptive attenuation modulation of local luminous intensity without adding an additional projection phase shift sequence. This ensures that overexposed areas of highlights fall back to the camera's optimal linear response range, fundamentally balancing high measurement efficiency and strong algorithm robustness.

[0107] In some alternative embodiments, combined with Figure 5 As shown, Figure 5 This is a schematic diagram of the segmentation model in one embodiment. The initial image is processed to obtain a pure specular mask, a normal diffuse mask, and a target brightness attenuation coefficient matrix, including:

[0108] A1: The initial image is processed to obtain a high-dimensional feature map through a multi-layered cascaded inverse residual feature extraction network of a pre-trained segmentation model.

[0109] A full-white field detection image refers to the original image of a surface containing highly reflective materials (such as metals or mirrors) acquired under uniform, strong light source illumination. Its physical significance lies in the fact that the brightness values ​​of overexposed areas in this image exceed the linear response range of the sensor, truly reflecting the continuous high-gradient change in surface reflectivity.

[0110] High-dimensional feature maps refer to feature representations that have significantly increased channel dimensions (e.g., 64 dimensions or more) after extraction by deep neural networks. Their physical significance lies in mapping the original pixel brightness values ​​to a high-dimensional continuous space, thereby decoupling and characterizing subtle, continuous physical changes in material reflectivity.

[0111] Among them, the multi-layer cascaded inverse residual feature extraction network performs high-dimensional continuous feature extraction on the full white field detection map. Specifically, the multi-layer cascaded inverse residual feature extraction network is composed of N inverse residual modules (N≥2) cascaded sequentially, which is used to perform high-dimensional continuous feature extraction on the full white field detection map.

[0112] A2: The high-dimensional feature map is segmented using a pre-trained segmentation model's segmentation network to obtain pure specular masks and normal diffuse masks.

[0113] A pure specular mask is a binary matrix (pixel values ​​of 0 or 1), where pixels with a value of 1 mark overexposed areas of pure specular highlights. Physically, it clearly defines the physical boundary where surface reflectivity exceeds the camera's dynamic range; in other words, the pure specular mask is used to clearly mark the boundaries of physical areas requiring underexposure processing. A normal diffuse mask is a binary matrix, where pixels with a value of 1 mark normal diffuse areas (physically mutually exclusive with the pure specular mask). Physically, it marks areas that are not overexposed and whose reflectivity falls within the linear response region.

[0114] In this embodiment, the high-dimensional feature map is input to the mask segmentation branch (segmentation network). This branch sequentially includes: one or more convolutional layers for further feature mapping, followed by a Softmax activation layer that classifies each pixel into either "highlight" or "normal" based on probability, and finally an Argmax layer that takes the class with the highest probability and outputs a binary mask. Specifically: Output channel 1 is a pure highlight mask (physical mutual exclusion: M_HL=1 represents a pure highlight region). Output channel 2 is a normal diffuse reflection mask (physical mutual exclusion: M_Norm=1 represents a normal diffuse reflection region, and M_HL+M_Norm≤1).

[0115] A3: The target brightness attenuation coefficient matrix is ​​obtained by processing the high-dimensional feature map based on the brightness regression network of the pre-trained segmentation model and using pure specular mask.

[0116] The target brightness attenuation coefficient matrix is ​​a continuous value matrix with the same dimensions as the input image. Each element represents the required brightness reduction ratio for the corresponding pixel location. Physically, it represents the pixel-level continuous energy attenuation prior for pure highlight areas from the camera's perspective, while the value is 0 for normal diffuse areas.

[0117] In some of these alternative embodiments, combined with Figure 6 As shown, Figure 6Here is a structural diagram of a multi-layer cascaded inverse residual feature extraction network in one embodiment. The network processes the initial image using a pre-trained segmentation model to obtain a high-dimensional feature map, including:

[0118] B1: The initial image is expanded in dimension by using a multi-layered cascaded inverse residual feature extraction network.

[0119] The dimension-up expansion layer includes a 1×1 convolution and a ReLU6 activation function. Physically, this means that while preserving the spatial structure, the signal is projected into an extremely high-dimensional space, providing sufficient capacity to characterize complex continuous reflectivity gradients.

[0120] Among them, the inverse residual feature extraction network is extended to a higher dimension layer (1×1Conv+ReLU6) to achieve this. Convolutional layers combined with the ReLU6 activation function expand the input feature map (the initial image is used as the input for the first layer) to increase its dimensionality. Let the number of input channels be C_in, and the expansion factor be t (t>1, typically 6), then the number of output channels becomes C_mid = t × C_in. The ReLU6 activation function provides non-linearity but maintains upper bound stability. This step unfolds the data in a very high-dimensional feature space, enabling the network to fully represent the complex and continuous physical reflectivity gradient of highly reflective metallic surfaces.

[0121] B2: The target feature map is obtained by extracting features from the initial image after dimensional expansion through a lightweight spatial filtering layer of a multi-layered cascaded inverse residual feature extraction network.

[0122] The lightweight spatial filtering layer consists of a 3×3 depthwise separable convolution and a ReLU6 activation function. Physically, this means independently modeling the diffusion trend of local overexposure energy in each high-dimensional channel, achieving computationally lightweight spatial feature extraction.

[0123] Specifically, a lightweight spatial filtering layer (3×3DWConv+ReLU6) is used to process the upscaled feature map. Depthwise separable convolution performs independent 3×3 spatial convolutions on each channel, with only 1 / C_mid of the parameters required for standard convolution. ReLU6 introduces non-linearity again to extract spatial features independently for each channel, capturing the physical diffusion trend of local overexposure intensity.

[0124] B3: After the target feature map is reduced in dimension by the linear dimension reduction projection layer of the multi-layer cascaded inverse residual feature extraction network, it is fused with the initial image output by the residual skip connection layer of the multi-layer cascaded inverse residual feature extraction network to obtain a high-dimensional feature map.

[0125] The linear dimensionality reduction projection layer consists of a 1×1 convolution and a linear activation function (i.e., no activation function). The linear activation function is used to prevent nonlinear information loss in the continuous energy decay physical features of highly reflective materials during feature dimensionality reduction. Physically, it compresses high-dimensional features back to the original dimension. At the same time, linear activation ensures that the continuous energy decay physical quantities in pure specular regions do not suffer information loss due to nonlinear transformation.

[0126] Residual skip connections are used to directly add the module input to the output of the linear dimensionality reduction projection layer. Physically, this preserves the small specular spillover gradients in the original image without loss, mitigating gradient vanishing in deep networks and improving the fidelity of spatial mapping.

[0127] First, a linear dimension reduction projection layer (1×1 Conv+Linear) is used to compress the number of channels from C_mid back to the original input dimension C_in. A linear activation function (Linear) is employed here, abandoning nonlinear operations such as ReLU. The physical significance of this is to prevent numerical truncation or nonlinear distortion of high-dimensional continuous features (representing the physical quantity of energy attenuation in pure specular regions) during the dimension reduction process, thereby ensuring lossless information transmission.

[0128] Then, through residual skip connections, the initial input feature map of this module is added element-wise to the output of the linear dimensionality reduction layer to obtain the output feature map of the current module.

[0129] Specifically, the residual skip connection extracts the initial input feature map of this module across layers and adds it element-wise to the output of the linear dimensionality reduction layer. This skip connection not only alleviates the gradient vanishing problem when the network deepens from the bottom layer, but also ensures that the tiny specular overflow gradient features in the initial image can be preserved across layers without loss, greatly improving the fidelity of pixel-level spatial mapping.

[0130] After cascading processing by multiple inverse residual modules, the final output feature map is the high-dimensional feature map.

[0131] In some optional embodiments, the target brightness attenuation coefficient matrix is ​​obtained by processing the high-dimensional feature map based on a pure specular mask using a pre-trained brightness regression network of the segmentation model. This includes: processing the high-dimensional feature map using a pre-trained brightness regression network of the segmentation model to obtain a global initial brightness attenuation coefficient matrix; and processing the global initial brightness attenuation coefficient matrix based on a pure specular mask to obtain the target brightness attenuation coefficient matrix.

[0132] The high-dimensional feature map is synchronously input to the luminance regression branch (luminance regression network). This branch consists of one or more convolutional layers followed by a global sigmoid activation function, outputting a global initial luminance attenuation coefficient matrix K_init(x,y) with the same height and width as the input image, and its values ​​range from [0,1]. This matrix provides the network's initial attenuation estimate for all regions of the entire image.

[0133] A spatial masking guidance mechanism is introduced, using the pure specular mask M_HL as a gated prior signal. This prior signal is multiplied pixel-wise with the global initial attenuation tensor K_init. This forces the attenuation coefficient of normal diffuse regions marked as 0 to be 0 (no attenuation), while only pure specular masks marked as 1 retain the attenuation prediction value in K_init. After this physical constraint, the output is a pixel-level continuous brightness attenuation coefficient matrix that precisely corresponds to the camera's viewpoint. .

[0134] In practical applications, the high-dimensional feature map is synchronously input to the brightness regression branch. This branch outputs a global initial decay tensor through convolutional layers and sigmoid activation layers. Subsequently, a spatial mask guidance mechanism is introduced to transform the pure specular mask output from the mask segmentation branch. As a gated prior signal, it is multiplied pixel-by-pixel with the global initial attenuation tensor. After this physical constraint of multiplication, the network outputs a pixel-level continuous brightness attenuation coefficient matrix that precisely corresponds to the camera's viewpoint. This multiplication mechanism mathematically forces the attenuation coefficient of the normal diffuse reflection region to remain at a non-attenuation baseline, thus ensuring that the predicted de-exposure coefficient is strictly limited to within the overexposure boundary.

[0135] Through the above embodiments, the pure specular mask output by the network accurately defines the overexposure boundary, while the attenuation coefficient matrix only acts inside the boundary, and its value is a continuous physical quantity, avoiding nonlinear information loss, thereby providing high-fidelity prior information for subsequent image underexposure or HDR fusion.

[0136] Furthermore, the above embodiments construct a multi-task deep learning architecture with strict physical and mathematical constraints, achieving extremely accurate localization and continuous control of the underexposure region. This application integrates the spatial hard segmentation of the overexposed region and the continuous soft regression of the attenuation intensity into a unified single-path feature backbone, significantly reducing the computational overhead of forward inference. More importantly, the network innovatively introduces a spatial mask guidance mechanism, using the output pure specular mask to enforce pixel-by-pixel gating constraints on the brightness attenuation tensor, supplemented by the linear activation function of the network's dimensionality reduction layer to prevent the continuous attenuation feature from breaking. This design mathematically guarantees that normal diffuse reflection regions will not be disturbed by erroneous underexposure, ensuring the physical rigor and smoothness of local adaptive underexposure modulation.

[0137] In some optional embodiments, the processing of the initial image to obtain the pure specular mask, normal diffuse mask, and target brightness attenuation coefficient matrix is ​​achieved through a pre-trained segmentation model; the training method of the segmentation model includes:

[0138] C1: The reference images obtained by the image acquisition device and the projection device are projected onto the sample test object with multiple global brightness coefficients that decrease in gradient.

[0139] In the offline training phase, in order to obtain the dual-supervision labels required for the multi-task network, namely the ground truth of the specular mask... With the true value of the local attenuation coefficient Data acquisition was performed using a multiple exposure (HDR) projection method. The controller controlled the projection device to project a full white field image onto the sample test object with a gradient-decreasing global brightness coefficient (e.g., 100%, 90%, 80%, ..., 10%), while simultaneously acquiring reference images. The sample test object included highly reflective objects with various complex geometries.

[0140] C2: Based on each reference image, determine the global brightness coefficient of the projection device when the gray value of any pixel in each reference image under the field of view of the image acquisition device is at the upper limit of the optimal linear response range of the image acquisition device.

[0141] For any pixel in each reference image within the field of view of the image acquisition device In a multiple exposure sequence, the projected brightness coefficient is found when the acquired grayscale value is exactly at the upper limit of the camera's optimal linear response range (e.g., grayscale value between 220-240, without saturation truncation). This coefficient is then normalized and used as the sample global brightness coefficient for that pixel. .

[0142] C3: If a pixel in the target reference image has undergone grayscale saturation truncation or the global brightness coefficient of the pixel is less than the target value, generate a pure specular mask based on the pixel, and generate a normal diffuse mask based on the pure specular mask; the target reference image is the reference image corresponding to the projection of the full white field image with maximum brightness.

[0143] C4: A segmentation model is obtained by training the model based on each reference image, the pure specular mask of the sample, the normal diffuse mask of the sample, and the global brightness coefficient of the sample.

[0144] The full-white field detection image acquired at 100% full-scale brightness is used as the discrimination benchmark. Spatial consistency judgment logic is established: if pixel... Gray-scale saturation truncation occurred under full-scale projection, or its corresponding global brightness coefficient of the sample. If the value is less than the target value, which can be 1.0 (meaning that this point must undergo physical downexposure to return to the linear region), then in the truth mask... Pixels that are marked as pure specular regions are labeled as 1; conversely, pixels that maintain a linear response without requiring brightness adjustment are labeled as normal diffuse regions (labeled 0). After generating the labels, a segmentation model can be trained based on these labels.

[0145] The above embodiments eliminate the uncontrollability of end-to-end black-box generation, ensuring absolute physical accuracy and strong scene generalization ability of 3D reconstruction. Existing deep learning de-exposure networks tend to directly repair or predict truncated stripe images, which are prone to fabricating false textures and introducing depth artifacts when faced with unknown complex metal geometry. This application strictly limits deep learning to the scope of solving physical control parameters, and the basic measurement information still comes 100% from real physical optical phase-shift echoes. This method solves the problem of camera target surface overexposure from the optical source, perfectly preserving the sinusoidality of the original stripes and the high-frequency morphological details of the real workpiece surface, effectively avoiding secondary errors introduced by the algorithm. Furthermore, by using a single frame of full white field image as the detection source and introducing deep learning, full-field high dynamic range measurement is achieved without increasing the number of additional phase-shift projection rounds, breaking the technical bottleneck of the incompatibility between efficiency and quality in traditional high-reflectivity measurement.

[0146] In some optional embodiments, a segmentation model is trained based on reference images, sample pure specular masks, sample normal diffuse masks, and sample global luminance coefficients. This includes: processing the reference images using the segmentation model to obtain predicted pure specular masks, predicted normal diffuse masks, and predicted global luminance coefficients; generating a mask segmentation loss function based on the predicted pure specular masks and sample pure specular masks; generating a base loss function based on the predicted global luminance coefficients and sample global luminance coefficients; simulating the physical space light modulation process based on the predicted global luminance coefficients and a full white field image to generate a modulated image, and generating a physical response consistency loss function under the supervision of the sample pure specular masks based on the modulated image and an ideal linear response target image; performing morphological dilation based on the sample pure specular masks to obtain a transition mask, and generating a boundary smoothing penalty loss function under the supervision of the transition mask based on the spatial gradient of the predicted global luminance coefficients; generating a target loss function based on the mask segmentation loss function, the base loss function, the physical response consistency loss function, and the boundary smoothing penalty loss function; and performing gradient backpropagation based on the target loss function to obtain the trained segmentation model.

[0147] To achieve collaborative optimization of the mask segmentation task and the local attenuation coefficient regression task, this invention constructs a joint loss function that integrates physical optics priors. During the offline training phase, this loss function effectively addresses the severe imbalance between positive and negative samples in industrial high-reflectivity measurements and mathematically enforces optical continuity constraints. Its complete mathematical expression is as follows:

[0148]

[0149] in, The hyperparameter weights are set to balance the gradient contributions of various loss functions. The joint loss function includes: mask segmentation loss function. Basic loss function Physical response consistency loss function and the loss function with boundary smoothing penalty term .

[0150] In industrial measurement scenarios, the area of ​​severely overexposed highlight regions (positive samples) is typically much smaller than that of normal diffuse reflection regions (negative samples). To prevent the network from being dominated by a large number of normal background pixels during training, the segmentation branch is supervised using FocalLoss, the expression of which is:

[0151]

[0152] in, Predict pixels for the network The probability of belonging to a highlight area. True value of specular mask , To balance the adjustment factors for positive and negative sample weights, denoted by the difficulty level of the sample, where N is the number of pixels. This design forces the network to focus on identifying blurred edges in highlight areas, improving the accuracy of overexposure boundary segmentation.

[0153] Predicting the attenuation coefficient using smoothed L1 loss measurement With truth value The fundamental numerical deviation between them. Compared to mean squared error (MSE), it is more robust to potentially introduced extreme noise or singular outliers.

[0154] The predicted attenuation matrix is ​​then applied inversely to the original full-white field probe map to simulate the real physical space light modulation process:

[0155]

[0156] in, This is a full-field detection map. For the camera's maximum range, The target surface grayscale value is the optimal linear response value for the camera (e.g., 220). This is the Hadamard product. This term is mandatory at the physical closed-loop level, after... The overexposed highlights after modulation must ultimately have their image grayscale accurately reduced to the ideal linear range of unsaturation, where K is the mask. The total number of valid pixels within the region (i.e., the highlight region) is used to average and normalize the calculated loss value.

[0157] To prevent flying points in phase calculation due to extreme steps in luminescence intensity during projected fringes, a total variation spatial smoothing constraint is introduced:

[0158]

[0159] in, For the mask The transition mask obtained after morphological dilation. and These represent the first-order spatial gradient operators along the horizontal (x-axis) and vertical (y-axis) directions in the spatial coordinate system, respectively, used to calculate the difference in brightness attenuation coefficients between adjacent pixels. This term mandates that the predicted attenuation coefficients exhibit a smooth transition in spatial gradient at the boundary between the normal diffuse and specular regions, strictly ensuring the physical sinusoidality and continuity of the modulated structured light.

[0160] In some optional embodiments, before modulating the reference image sequence of the projection device based on the target brightness attenuation coefficient matrix to obtain the target projected image, the method further includes: acquiring a second speckle image of the sample object under the first speckle image projected by the projection device using an image acquisition device; establishing a mapping relationship between the field of view of the image acquisition device and the field of view of the projection device based on the first speckle image and the second speckle image; processing the target brightness attenuation coefficient matrix based on the mapping relationship to obtain an attenuation coefficient map corresponding to the pure specular mask under the field of view of the projection device; and modulating the reference image sequence of the projection device based on the target brightness attenuation coefficient matrix to obtain the target projected image, including: modulating the reference image sequence of the projection device based on the attenuation coefficient map to obtain the target projected image.

[0161] Because pure specular areas exhibit severe pixel saturation and phase cutoff under phase-shifted fringe projection, traditional phase-based point-to-point matching methods fail in this region. Therefore, the controller directs the projection device to project a high-contrast first speckle image onto the object under test, while the image acquisition device simultaneously captures a second speckle image. Speckle pattern plays a crucial role in rapid spatial localization and cross-device pixel matching. Utilizing the randomness and uniqueness of pixel intensity within a local window of the speckle map, and combining digital image correlation (DIC) algorithms or feature point extraction algorithms, a rapid search is performed between the overexposed highlight areas captured by the camera and the projection plane of the projector's DMD (Digital Micromirror Array), establishing a mapping relationship between the field of view of the image acquisition device and the field of view of the projection device. Furthermore, based on this mapping relationship, the field of view of the image acquisition device is then... Marked overexposed pixel coordinates Fast and accurate mapping to physical pixel coordinates within the field of view of the projection device A specular attenuation mask is constructed for the projector's field of view.

[0162] Therefore, during the online measurement phase, the camera view brightness attenuation coefficient map predicted in real time by the segmentation network is used. Using the obtained speckle space mapping relationship A coordinate system transformation is performed to accurately map and interpolate the coordinates to the projector coordinate system, generating a spatial brightness attenuation mask under the projection viewpoint. .

[0163] Therefore, the spatial brightness attenuation mask based on this projection viewpoint is subsequently used. The reference image sequence of the projection device is modulated to obtain the target projected image, thus ensuring accuracy.

[0164] In some of these alternative embodiments, combined with Figure 7 As shown, Figure 7 Here is a structural diagram of a semantic segmentation model in one embodiment. The initial image is partitioned to obtain a cross-reflection mask, a pure specular mask, and a normal diffuse mask, including:

[0165] D1: Multi-level feature maps are obtained by performing multi-level feature extraction on the initial image through the encoding feature extraction network of the pre-trained semantic segmentation model.

[0166] The pre-trained semantic segmentation model can be a fully convolutional neural network with dilated spatial pyramid pooling based on an encoder-decoder architecture. It can include an encoding feature extraction network, a dilated spatial pyramid pooling network, a decoding network, and a mask output network. This pre-trained semantic segmentation model can perform semantic segmentation on the initial image to obtain interreflection masks, pure specular masks, and normal diffuse masks.

[0167] The feature extraction network is the backbone of the network; taking the ResNet-50 residual network as an example, the initial input image... ResNet-50 extracts multi-level feature maps. The residual structure can effectively avoid the gradient vanishing problem when the network is deepened, while extracting high-frequency edge information from shallow layers (such as the texture boundary of stripes) to high-dimensional semantic features from deep layers (such as global context features of large-area highlights or groove mutual reflection).

[0168] D2: Through multiple parallel branches in the hollow spatial pyramid pooling network of the pre-trained semantic segmentation model, different receptive field features are extracted from the high-level feature maps in the multi-level feature maps, and the features extracted by multiple parallel branches are fused to obtain the target high-level feature map.

[0169] This hollow space pyramid pooling network comprises multiple parallel branches, each with a different receptive field. Taking mutual reflection as an example, optical path coupling can occur in locally adjacent regions (such as the bottom and sidewalls of a pit) or in globally separated regions (such as the two opposite sidewalls of a deep cavity). To capture the correlation at both scales simultaneously, the network requires multiple branches with different receptive fields to process in parallel.

[0170] In some optional embodiments, the parallel branches include a primary receptive field feature extraction branch, a global receptive field feature extraction branch, and a feature extraction branch located between the primary and global receptive fields. The primary receptive field feature extraction branch is a 1×1 convolutional branch used to capture local texture features. The global receptive field feature extraction branch is a global image pooling branch used to obtain contextual information at the global image level and capture long-distance cross-reflection correlations. The feature extraction branch located between the primary and global receptive fields includes three 3×3 dilated convolutional branches with different dilation rates (6, 12, and 18) used to multiply the receptive field without increasing the number of parameters and capture long-distance cross-reflection correlation features. In other embodiments, the number of feature extraction branches between the primary and global receptive fields can also be other numbers, only using different dilation rates.

[0171] The physical meaning of dilated convolution is that by inserting "holes" between the elements of the convolution kernel, the receptive field can be multiplied without increasing the number of parameters. Different dilation rates correspond to receptive fields of different scales, thereby capturing mutual reflection coupling relationships within different distance ranges.

[0172] Finally, the feature maps extracted from multiple parallel branches are fused to obtain the target high-level feature map. For example, the output feature maps of the above five parallel branches are spliced ​​and aggregated along the channel dimension, and multi-scale information is fused using 1 × 1 convolution to output the target high-level feature map with macroscopic optical interference sensing capability.

[0173] Combination Figure 7As shown, to address the issue of varying specular and interreflective region scales caused by machined parts with different curvatures and sizes, the feature maps output from the deep layers of ResNet-50 are fed into a dilated spatial pyramid pooling network. The dilated spatial pyramid pooling network contains five parallel branches: one... Convolutional layers; three The dilated convolutional layers are set with dilation rates of 6, 12, and 18, respectively. Dilated convolutions exponentially expand the network's receptive field without increasing computation or reducing spatial resolution, effectively capturing a wide range of optical path interference contextual information. A global average pooling layer is used to obtain global image-level features. Subsequently, the output feature maps from the five branches are concatenated along the channel dimension, and then processed through a... Convolutional layers perform feature dimensionality reduction and integration, outputting a high-level feature map of the target containing rich multi-scale semantic information.

[0174] D3: The low-level feature maps in the multi-level feature maps are dimensionality reduced by the decoding network of the pre-trained semantic segmentation model and then concatenated with the target high-level feature maps after spatial resolution adjustment in the channel dimension. The concatenated feature maps are then optimized and expanded to the same spatial resolution as the initial image.

[0175] The low-level feature map is used to prevent it from overshadowing the high-level semantic features during fusion. To achieve this, a low-level feature map containing rich spatial details (such as precise fringe phase truncation edges) is extracted from the shallow layers of the encoding feature extraction network and compressed using a 1×1 convolutional layer. The target high-level feature map output from the dilated spatial pyramid pooling network is then bilinearly upsampled by a factor of 4 (or other factors in other embodiments) to align its spatial resolution with the processed low-level feature map. Subsequently, the two are concatenated along the channel dimension. The concatenated feature map is then processed by a... The convolutional layer performs feature refinement and aliasing removal, and finally undergoes a 4x (or other factors in other embodiments) bilinear upsampling to restore the spatial resolution of the feature map to the same level as the initial image. (i.e., camera resolution) equal to or the same size.

[0176] D4: The feature map output by the decoding network of the semantic segmentation model is processed through the mask output network of the pre-trained semantic segmentation model to obtain the mutual reflection mask, pure specular mask and normal diffuse reflection mask.

[0177] The network's final stage is a mask output network. This network uses a Softmax activation function to transform the feature map into a probability distribution of each pixel belonging to different categories, and then uses an Argmax operation to extract the category label corresponding to the highest probability for each pixel. Finally, it outputs three binary spatial masks of the same resolution as the image acquisition device: a reflection mask, and a cross-reflection mask. Marking areas of mutual reflection; a pixel value of 1 indicates the presence of mutual reflection, and 0 indicates its absence. Pure specular mask. Marks pure highlight areas; a pixel value of 1 represents simple overexposure, and 0 represents none. Normal diffuse mask. Mark the diffuse reflection area without the above interference. For areas other than the two mentioned above, the pixel value is 1.

[0178] Furthermore, it should be noted that for any valid pixel within the field of view of the image acquisition device... After normalization and binarization, the output of the neural network satisfies the mutual exclusion and completeness constraints:

[0179]

[0180] Furthermore, it should be noted that the semantic segmentation model can be trained based on samples with pixel-level ground truth labels. The ground truth for the mutual reflection region is obtained by extracting the set of pixels with optical path conflicts using the traditional parallel partitioning projection method (or independent emission point projection method), accurately identifying the real mutual reflection regions where multiple reflections occur. The ground truth for the pure specular highlight region is obtained by using multiple exposure methods or global brightness thresholding to extract overexposed areas where a single specular reflection causes pixel saturation truncation and no optical path crosstalk occurs. The ground truth for the normal region is the remaining effective stripe coverage area after removing the aforementioned mutual reflection and pure specular highlight regions; this is the normal diffuse reflection region. Using these three types of real physical response boundaries, a large-scale dataset containing specular and mutual reflection features is constructed to train the semantic segmentation model.

[0181] In the above embodiments, by processing the initial image, a mutual reflection mask, a pure specular mask, and a normal diffuse reflection mask are obtained, laying the foundation for subsequent partitioned projection.

[0182] In some optional embodiments, based on the mutual reflection mask and the initial image, the mutual reflection mask is divided into K non-mutually reflecting sub-regions, including:

[0183] E1: Through the input network of the sub-region prediction model, the inter-reflection mask and the initial image are concatenated in the channel dimension, and the concatenated feature map is subjected to shallow feature mapping to obtain a shallow feature map.

[0184] This embodiment mainly focuses on the mutual reflection mask obtained by decoupling mentioned above. The region is further predicted by a deep learning model to predict the conflict properties between pixels, and then refined into a set of sub-regions that do not reflect each other, so as to facilitate the subsequent implementation of a partitioned projection strategy to completely eliminate the phase deviation caused by multiple reflections.

[0185] The sub-region prediction model comprises an input network, a shallow spatial attention mechanism network, a feature extraction network, and a prediction network. This sub-region prediction model is an end-to-end pixel-level classification network based on a fully convolutional network (FCN) architecture, used to predict the sub-region to which interreflective pixels belong in real time.

[0186] This input network is used to process the acquired initial image. With the predicted mutual reflection mask Concatenation is performed along the channel dimension. The concatenated feature maps are then input into a standard... Shallow feature maps are obtained by performing shallow feature mapping in the convolutional layer. This layer aims to initially fuse cross-modal information and fully preserve the high-frequency physical spatial details caused by mutual reflection without reducing spatial resolution.

[0187] E2: Through the shallow spatial attention mechanism network of the sub-region prediction model, different types of features are extracted from the shallow feature map, and a weight mask is generated based on the extracted different types of feature maps. An adaptive feature map is obtained based on the weight mask and the shallow feature map.

[0188] Different types of feature extraction networks include max pooling networks and average pooling networks. These networks perform max pooling and average pooling operations along the channel dimension of the shallow feature map, respectively, to capture significant local highlights and the overall background distribution. The physical meanings of these two pooling operations differ: max pooling captures significant local highlights (corresponding to spots and overexposed areas caused by mutual reflection and high reflection), while average pooling reflects the overall background distribution (corresponding to striped backgrounds in diffuse areas). The results of these two pooling operations are then concatenated channel-wise to obtain a stitched feature map. In other embodiments, the different types of feature extraction networks can also be used; no specific limitation is made here.

[0189] Subsequently, weight masks are generated based on the extracted feature maps of different types. Adaptive feature maps are obtained based on the weight masks and shallow feature maps. For example, the two-dimensional spatial attention weight mask is multiplied element-wise with the shallow feature map.

[0190] In some alternative embodiments, combined with Figure 8 As shown, Figure 8This is a schematic diagram of the sub-region prediction model in one embodiment. The model uses a shallow spatial attention mechanism network to extract different types of features from the shallow feature map, and generates weight masks based on these extracted feature maps. An adaptive feature map is then obtained based on the weight masks and the shallow feature map. This process includes: extracting different types of features from the shallow feature map using different feature extraction networks within the shallow spatial attention mechanism network; concatenating these different feature maps to obtain a concatenated feature map; performing feature extraction and nonlinear mapping on the concatenated feature map using convolutional layers and nonlinear activation networks within the shallow spatial attention mechanism network to obtain a weight mask; and adaptively weighting the weight mask and the shallow feature map using an adaptive weighting network within the shallow spatial attention mechanism network to obtain an adaptive feature map.

[0191] Specifically, a bypass branch is derived from the shallow feature map output by the aforementioned 3×3 convolutional layer. Global max pooling and global average pooling are performed along the channel dimension to compress channel information. Subsequently, the two are concatenated and passed through a 7×7 convolutional layer with a large receptive field. Finally, a sigmoid activation function is applied to output a two-dimensional spatial attention weight mask (SpatialAttentionMap) that retains extremely high physical resolution.

[0192] Then, the two-dimensional spatial attention weight mask is multiplied element-wise with the shallow feature map. This mechanism accurately assigns higher response weights to pixels with severe phase distortion before the shallow feature map enters the deep downsampling, thus filtering out redundant interference from normal diffuse reflection stripes at the source.

[0193] E3: The global context feature map is obtained by extracting features from the adaptive feature map through the feature extraction network of the sub-region prediction model.

[0194] In this embodiment, the attention-weighted adaptive feature map is input into the subsequent feature extraction network. To significantly reduce inference latency, the hidden layers of the feature extraction network abandon standard convolutions and adopt a multi-layered, cascaded lightweight feature extraction module. This module contains depthwise separable convolutions (i.e., a combination of channel-wise and pointwise convolutions). By continuously stacking this lightweight module, the feature extraction network effectively expands the receptive field with extremely low computational parameters, further extracting global contextual features caused by large-scale mutual reflection optical paths.

[0195] E4: The global context feature map is processed through the prediction network of the sub-region prediction model to obtain K non-reflective sub-regions.

[0196] The terminal of the sub-region prediction model processes the global context feature map through a prediction network to obtain K mutually non-reflective sub-regions. Specifically, the terminal passes through a... The convolutional layers are used for channel dimensionality reduction, directly compressing the number of channels in the global context feature map to [number missing]. (That is, the total number of non-interfering partitions). Then, a Softmax activation layer is connected, outputting a single image with [number of channels]. A multi-channel partition probability map is generated. Finally, the partition prediction result for each pixel is obtained through the Argmax operation. .

[0197] The prediction network learns from the initial image. Due to the decrease in fringe contrast, phase distortion, and global geometric context features caused by interreflection, a nonlinear mapping from visual distortion features to graph theory partition numbers is established. Compared to traditional offline scanning and complex graph coloring calculations, this network can achieve refined partitioning of interreflective regions in milliseconds.

[0198] In some optional embodiments, the above method further includes: a training step for a sub-region prediction model, which includes: obtaining the optical path coupling strength between any two pixels within the sample mutual reflection mask by high-frequency stripe scanning or independent pixel scanning, and determining two pixels with mutual reflection influence as a conflict pair based on the optical path coupling strength; generating a node in the conflict graph for each pixel within the sample mutual reflection mask, and generating edges between nodes based on the conflict pairs; processing the conflict graph so that, with the minimum number of sub-regions, two nodes connected by an edge are assigned to different sub-regions; determining the partition of each pixel within the sample mutual reflection mask based on the processed conflict graph; and training the sub-region prediction model based on the partition of each pixel within the sample mutual reflection mask, the sample mutual reflection mask, and the initial sample image.

[0199] In the offline ground truth construction stage, ground truth needs to be constructed. For typical complex reflection scenarios, physical optics models and graph coloring algorithms are used to decouple image pixels into several non-interfering mutually reflective sub-regions, generating multi-state ground truth labels containing physical prior constraints.

[0200] First, the optical coupling strength between any two pixels within the sample mutual reflection mask is obtained through high-frequency stripe scanning or independent pixel scanning. For example, a high-frequency phase-shifted stripe sequence (i.e., the sample target pattern) is projected onto the sample to be tested, such as the surface of a standard sample, and the stripe images are simultaneously acquired by an image acquisition device, i.e., the initial images of each sample. The contrast or modulation variation of each initial image is calculated, and macroscopic spatial decoupling is performed based on the differences in reflectance properties.

[0201] The second objective requirement includes a significant decrease in contrast, phase distortion, or multi-peak response caused by multiple bounces of the optical path. Therefore, in this application, pixels with a significant decrease in contrast, phase distortion, or multi-peak response caused by multiple bounces of the optical path are identified as complex interference regions, which serve as sample interreflection regions for subsequent microscopic image coloring processing.

[0202] In this embodiment, conflict processing can be performed on the mutual reflection regions of each sample to obtain K sample sub-regions that do not form mutual reflections by means of mutual reflection coupling strength quantification and conflict edge extraction.

[0203] In some optional embodiments, conflict processing is performed on each sample mutual reflection region to obtain K sample sub-regions that do not form mutual reflections. This includes: dividing the sample mutual reflection region into several regions to be processed; obtaining the optical path coupling strength between any two pixels in the sample mutual reflection region by activating the regions to be processed one by one, and determining two pixels with mutual reflection influence as a conflict pair based on the optical path coupling strength; generating a node in the conflict graph for each pixel in the sample mutual reflection region, and generating edges between nodes based on the conflict pairs; processing the conflict graph so that, with the minimum number of sample sub-regions, two nodes connected by an edge are assigned to different sample sub-regions; and determining the sample sub-region where each pixel in the sample mutual reflection region is located based on the processed conflict graph.

[0204] For the pixels in the aforementioned mutual reflection region, it is necessary to quantify the energy coupling relationship between each pixel in order to identify physical-level optical path interference. In this embodiment, independent illumination analysis is adopted, that is, small area block projection or multi-mode illumination scheme is used to activate specific areas within the interference region one by one.

[0205] Among them, obtaining the optical path coupling strength between any two pixels within the sample mutual reflection region includes: if in the projected pixel... At that time, camera pixels not directly illuminated If a significant light intensity response is received, then the pixel is determined. and There is mutual reflection coupling between camera pixels, and they are not directly illuminated. The received light intensity corresponds to the projected pixel. Camera pixels that are not directly illuminated The optical coupling strength between them.

[0206] Then, the interference of all pixel pairs within the mutual reflection region is statistically analyzed. If there is mutual reflection coupling between two pixels, these two pixels are defined as a pair of conflicting nodes, and a conflicting edge is constructed between them.

[0207] Then, a mathematical model is constructed, namely, a mutual reflection conflict graph is built. The conflict graph consists of nodes and edges, with each pixel within the mutual reflection region representing a node. All pairs of conflicting nodes exhibiting physical optical interference are connected to form edges in the conflict graph, thus creating a conflict graph characterizing the physical interference relationships of mutual reflection.

[0208] Finally, based on the conflict graph, the sample sub-region of each node is determined. Optionally, the conflict graph is processed to minimize the number of sub-regions, so that two nodes connected by an edge are assigned to different sub-regions. This includes: obtaining each node in the conflict graph and sorting each node in descending order of the number of edges connected to each node; obtaining the unassigned current node from the sorted nodes and determining the midpoint of the first conflict node that is connected to the current node and has been assigned, as well as the first sample sub-region corresponding to the first conflict node; assigning a second sample sub-region to the current node that is different from each of the first sample sub-regions; determining the unassigned second conflict nodes that are connected to the current node and assigning each second conflict node to a different third sample sub-region, which is different from the first and second sample sub-regions.

[0209] For ease of understanding, suppose we have a conflict graph containing five nodes: V1, V2, V3, V4, and V5. The edges between them represent areas where the same sample subregion cannot be assigned. First, we count the number of edges (degree) of each node. V1 connects V2, V3, and V4, with a degree of 3; V3 connects V1, V2, and V4, also with a degree of 3; V4 connects V1, V3, and V5, also with a degree of 3; V2 connects V1 and V3, with a degree of 2; and V5 connects only V4, with a degree of 1. We sort these edges in descending order of degree, resulting in the sequence V1, V3, V4, V2, and V5. During assignment, we first take the first unassigned node, V1. Since it has no assigned adjacent nodes (because none are assigned), we directly assign it a sample subregion, such as A. Next, we process the next node, V3. Its assigned adjacent node is V1 (assigned to A), so V3 cannot choose A and is assigned a different subregion, B. Then we process V4. Its assigned neighbor is V1 (A), but not V3 (because V3 was just assigned but hasn't been queried as the "first conflicting node among the assigned nodes" yet). Note that in the original algorithm description, "first conflicting node" refers to the current node's neighbors among the assigned nodes. Therefore, V4's only assigned neighbor is V1, so we avoid A, for example, choosing C. Then we process V2. Its assigned neighbors are V1 (A) and V3 (B), so V2 cannot choose A or B, so we choose C (note that C does not conflict with V1 or V3, but may conflict with V4? Here we only consider the already assigned neighbors). V2 and V4 have an edge, but V4 hasn't been assigned yet, so we ignore it for now. Therefore, V2 can choose C. Finally, we process V5. Its only assigned neighbor is V4 (C), so V5 cannot choose C. It chooses D, which is different from A, B, and C. The final assignment result is: V1 is assigned A, V3 is assigned B, V4 is assigned C, V2 is assigned C, and V5 is assigned D. Note that although V2 and V4 both chose C, there is no edge between them (in fact, V2 and V4 have no direct edge in this example), so this is allowed. V2 is different from V1 and V3, and V4 is different from V1, V3, and V5, satisfying all conflict constraints.

[0210] In this embodiment, a graph coloring algorithm can be used to color the conflict graph. The purpose of processing is to... The nodes are assigned to as few non-conflicting subsets as possible, meaning the nodes are placed in different sample sub-regions. This assignment is simplified by coloring, with coloring criteria including: nodes connected by edges... Two adjacent nodes connected (i.e., two pixels that interfere with each other through reflection) must be assigned different color numbers. The optimization objective is to find the minimum total number of colors. This ensures that all nodes in the image are colored without conflict. A heuristic greedy coloring algorithm can be used, sorting nodes by degree and assigning each node the available color with the smallest index. Finally, the sample interreflection regions are subdivided into numbered... of Each sample sub-region is independent of the others.

[0211] Finally, the graph is colored to generate labels 1 to 1. The samples are fused to generate a single sample inter-reflection mask. In this sample inter-reflection mask, the pixel value of each pixel represents its category in the physical projection time sequence: Category The sample reflection sub-regions must be mutually exclusive between categories on the projection time slice. This sample synthesis mask serves as the ground truth for the deep learning model, guiding the network to learn how to identify this deep physical conflict relationship based on the texture of a single frame image.

[0212] In the above embodiments, to address the problem that existing methods based on solid geometric constraints or morphological partitioning heavily rely on standard digital models (CAD models) and have poor generalization ability, a cross-reflection conflict graph based on the optical transmission mechanism is constructed offline, and a graph coloring algorithm is used to generate K types of multi-state ground values ​​with real physical constraints. This allows the network model to incorporate powerful optical physical priors without requiring any prior digital model input. To address the shortcomings of existing pure black-box end-to-end deep learning methods that are detached from actual optical transmission mechanisms, leading to unclear decision logic and questionable measurement reliability, the discrete states generated by fusing optical physical interference and graph coloring algorithms are used as supervised ground values. The K channels output at the network's end are no longer abstract features that are difficult to analyze, but rather strictly correspond to mutually exclusive photonic regions in the physical world. This design, which introduces strong physical prior knowledge into network training, makes each step of the model's partitioning prediction clearly physically interpretable, providing highly stable and transparently controllable measurement assurance for stringent industrial-grade inspection.

[0213] In some alternative embodiments, combined with Figure 9 , combined Figure 9 As shown, Figure 9 Here is a block diagram of the attenuation coefficient prediction model in one embodiment; and Figure 5 Unlike the illustrated embodiment, this embodiment uses a pre-trained attenuation coefficient prediction model to process the pure specular mask and the initial image to obtain a first attenuation coefficient map corresponding to the pure specular mask under the field of view of the image acquisition device, including:

[0214] Q1: By using the input network of the attenuation coefficient prediction model, the pure specular mask and the initial image are concatenated in the channel dimension to obtain a concatenated feature map.

[0215] To address the issue of severe pixel saturation and truncation caused by pure specular areas (single specular reflection), this step constructs a lightweight brightness adjustment neural network specifically for image-to-image continuous regression tasks. This network aims to learn the inverse mapping relationship between the continuous reflectivity distribution characteristics of highly reflective materials and the camera's nonlinear response curve, thereby outputting a pixel-level first attenuation coefficient map in real time.

[0216] The network's input receives a single frame of the initial image. With the decoupled pure specular mask The two are concatenated along the channel dimension to form a two-channel input tensor. Initial image It provides a continuous grayscale gradient of the actual overexposure in the highlight areas (i.e., the contextual features of brightness overflow), while the pure highlight mask... This serves as a hard boundary prior, forcing the network to focus its attention on the physically effective area where exposure reduction is required.

[0217] Q2: The output feature map is obtained by performing high-dimensional feature extraction on the cascaded feature map through a multi-layered cascaded inverse residual feature extraction network of the attenuation coefficient prediction model.

[0218] To fit complex mapping relationships with extremely low inference latency, the network backbone employs a multi-layered cascaded inverted residual lightweight module. The multi-layered cascaded inverted residual feature extraction network performs high-dimensional continuous feature extraction on the cascaded feature map. Specifically, the multi-layered cascaded inverted residual feature extraction network consists of N inverted residual modules (N≥2) sequentially cascaded, used for high-dimensional continuous feature extraction from the cascaded feature map.

[0219] Q3: The output feature map will be processed by the output layer of the attenuation coefficient prediction model to output the first attenuation coefficient map corresponding to the pure specular mask.

[0220] Since the goal of this task is to output specific continuous physical control parameters, rather than discrete classification labels, the traditional classifier module is discarded at the end of the network. The output feature map (which is a high-level feature map) extracted by the backbone network is input to the end. A convolutional layer, followed by a strictly connected sigmoid activation function. The sigmoid function smoothly and strictly non-linearly maps the feature map numerically to... Within a continuous interval, the network ultimately outputs a two-dimensional spatial brightness attenuation coefficient map. Each pixel value in this tensor directly corresponds to the percentage of physical luminous intensity that the projector's DMD (Digital Micromirror Array) needs to attenuate. This mechanism allows the system to accurately pull overexposed pixels from the corresponding camera back to the optimal linear response range with only a single forward inference.

[0221] In some optional embodiments, an output feature map is obtained by performing stage-by-stage high-dimensional feature extraction on the cascaded feature map through a multi-layered cascaded inverse residual feature extraction network of the attenuation coefficient prediction model, including:

[0222] R1: The number of channels in the cascaded feature map is expanded by the upscaling extension layer in the inverse residual feature extraction network to obtain an upscaling feature map;

[0223] The dimension-up expansion layer includes a 1×1 convolution and a ReLU6 activation function. Physically, this means that while preserving the spatial structure, the signal is projected into an extremely high-dimensional space, providing sufficient capacity to characterize complex continuous reflectivity gradients.

[0224] Among them, the inverse residual feature extraction network is extended to a higher dimension layer (1×1Conv+ReLU6) to achieve this. Convolutional layers combined with the ReLU6 activation function expand the input feature map (the initial image is used as the input for the first layer) to increase its dimensionality. Let the number of input channels be C_in, and the expansion factor be t (t>1, typically 6), then the number of output channels becomes C_mid = t × C_in. The ReLU6 activation function provides non-linearity but maintains upper bound stability. This step unfolds the data in a very high-dimensional feature space, enabling the network to fully represent the complex and continuous physical reflectivity gradient of highly reflective metallic surfaces.

[0225] R2: Spatial features are extracted from the up-dimensional feature map by using a lightweight spatial filtering layer in the inverse residual feature extraction network to obtain a spatially filtered feature map.

[0226] The lightweight spatial filtering layer consists of a 3×3 depthwise separable convolution and a ReLU6 activation function. Physically, this means independently modeling the diffusion trend of local overexposure energy in each high-dimensional channel, achieving computationally lightweight spatial feature extraction.

[0227] Specifically, a lightweight spatial filtering layer (3×3DWConv+ReLU6) is used to process the upscaled feature map. Depthwise separable convolution performs independent 3×3 spatial convolutions on each channel, with only 1 / C_mid of the parameters required for standard convolution. ReLU6 introduces non-linearity again to extract spatial features independently for each channel, capturing the physical diffusion trend of local overexposure intensity.

[0228] R3: By using the linear dimensionality reduction projection layer in the inverse residual feature extraction network, the number of channels in the spatial filter feature map is compressed to obtain the dimensionality reduction feature map.

[0229] R4: The input feature map of the inverse residual feature extraction network is added element-wise to the dimensionality-reduced feature map through the residual skip connection layer in the inverse residual feature extraction network to obtain the output feature map.

[0230] The linear dimensionality reduction projection layer consists of a 1×1 convolution and a linear activation function (i.e., no activation function). The linear activation function is used to prevent nonlinear information loss in the continuous energy decay physical features of highly reflective materials during feature dimensionality reduction. Physically, it compresses high-dimensional features back to the original dimension. At the same time, linear activation ensures that the continuous energy decay physical quantities in pure specular regions do not suffer information loss due to nonlinear transformation.

[0231] Residual skip connections are used to directly add the module input to the output of the linear dimensionality reduction projection layer. Physically, this preserves the small specular spillover gradients in the original image without loss, mitigating gradient vanishing in deep networks and improving the fidelity of spatial mapping.

[0232] First, a linear dimension reduction projection layer (1×1 Conv+Linear) is used to compress the number of channels from C_mid back to the original input dimension C_in. A linear activation function (Linear) is employed here, abandoning nonlinear operations such as ReLU. The physical significance of this is to prevent numerical truncation or nonlinear distortion of high-dimensional continuous features (representing the physical quantity of energy attenuation in pure specular regions) during the dimension reduction process, thereby ensuring lossless information transmission.

[0233] Then, by using residual skip connections, the cascaded feature map of this module is added element-wise to the output of the linear dimensionality reduction layer to obtain the output feature map of the current module.

[0234] Specifically, the residual skip connection extracts the cascaded feature map from this module across layers and adds it element-wise to the output of the linear dimensionality reduction layer. This skip connection not only alleviates the gradient vanishing problem when the network deepens from the bottom layer, but also ensures that the tiny specular overflow gradient features in the initial image can be preserved across layers without loss, greatly improving the fidelity of pixel-level spatial mapping.

[0235] After cascading processing by multiple inverse residual modules, the final output feature map is the output feature map.

[0236] Among them, for the overexposed center region mapped onto the projection plane, the network output A smaller value (e.g., 0.2) is used to significantly reduce projected brightness; for overexposed edge areas, Smooth transition; for non-highlight areas (corresponding to) area), It remains at 1, meaning it does not decay.

[0237] In some optional embodiments, generating K projected images based on K non-interfering sub-regions includes: acquiring a generated N-step phase-shifting structured light reference image sequence; generating a modulation matrix based on the K non-interfering sub-regions, the region corresponding to the pure specular mask after attenuation of the attenuation coefficient map, and the region corresponding to the normal diffuse reflection mask; and generating K projected images based on the reference image sequence and the modulation matrix.

[0238] During the online measurement phase, the image acquisition device's viewing angle brightness attenuation coefficient map, predicted in real time by the neural network, is used. Using the obtained speckle space mapping function A coordinate system transformation is performed to accurately map and interpolate the coordinates to the projector coordinate system, generating a spatial brightness attenuation mask from the projection device's perspective. When generating phase-shifted structured light fringes for 3D reconstruction, the originally generated standard sine fringes are compared with... Pixel-by-pixel multiplication modulation is performed. A computer-controlled projection device projects this modulated adaptive underexposure fringe pattern. At this point, because the projection device precisely reduces the luminous intensity of the corresponding physical area, the pure highlight areas re-captured by the camera fall back into the linear dynamic response range, successfully eliminating highly reflective areas without significantly increasing the number of projected images. A single highlight overexposure was used to obtain a high-quality high-frequency stripe pattern.

[0239] In some optional embodiments, the method further includes: obtaining the boundary connected region of K non-interfering sub-regions in the modulation matrix, the region corresponding to the pure specular mask after attenuation of the attenuation coefficient map, and the region corresponding to the normal diffuse reflection mask as a boundary transition zone (in another embodiment, if there is no mutual reflection recognition, the boundary connected region of the region corresponding to the pure specular mask after attenuation of the attenuation coefficient map and the region corresponding to the normal diffuse reflection mask are obtained as the boundary transition zone); performing spatial domain convolution smoothing on the boundary transition zone to obtain an optimized modulation matrix; generating K projection images based on the reference image sequence and the modulation matrix (if there is no mutual reflection, one projection image is generated), including: generating K projection images based on the reference image sequence and the optimized modulation matrix.

[0240] Because the DMD chip in a projector inevitably suffers from hardware defocusing during actual projection, directly using a hard-boundary modulation matrix composed of 0s and 1s would be problematic. This will cause high-frequency cutoff at the physical boundary of regions with different reflective properties, which in turn will cause severe phase distortion and ringing effect in the stripes captured by the camera at the boundary.

[0241] Therefore, this embodiment introduces a spatial transition optimization strategy for the mask boundary before actual projection. Modulation matrix extraction. Each mask in the middle ( The boundary connected region of the boundary is used as the boundary transition zone. A two-dimensional Gaussian low-pass filter or morphological smoothing algorithm is used to perform spatial domain convolutional smoothing on this boundary transition zone, constructing an optimized spatial fusion modulation matrix with soft edge transition characteristics. This pre-compensation operation enables the projected light intensity to exhibit a continuous gradient change at the boundary of different modulation regions, suppressing the phase jump caused by hard edges from the physical optical source, and greatly reducing the difficulty of subsequent phase calculation and boundary refinement.

[0242] In the above embodiments, a phase soft edge pre-compensation mechanism is introduced to perform physical-level gradient smoothing on extreme jumps at the algorithm source, significantly reducing the fitting pressure on the neural network. Combined with the large receptive field spatial gating prediction within the BMSG module, the system can adaptively cover different degrees of defocus diffusion, achieving a more thorough and smoother seamless fusion effect. The soft edge pre-compensation operation performs preliminary smoothing of abnormal high-frequency gradients at the gaps at the physical level, effectively reducing the numerical difficulty for subsequent deep learning networks to predict residuals for large step errors.

[0243] In some optional embodiments, a target phase map is obtained based on each target image, a mutual reflection mask, a pure specular mask, and a normal diffuse reflection mask. This includes: extracting the effective phase values ​​of effective pixels in the target image corresponding to each projected image based on the mutual reflection mask, the pure specular mask, and the normal diffuse reflection mask (if there is no mutual reflection identification, the corresponding mutual reflection mask can be removed) to generate a target phase map; determining the fringe modulation value of each effective pixel based on the effective phase values ​​of each effective pixel in the target phase map; and optimizing the target phase map based on the target phase map, the fringe modulation value, the mutual reflection mask (if there is no mutual reflection, the mutual reflection mask can be removed here), the pure specular mask, and the normal diffuse reflection mask.

[0244] Specifically, the controller can control the projection device to project multi-component time-sharing or zoned phase-shifted structured light stripes onto the object under test according to a preset zoning strategy. Simultaneously, the image acquisition device acquires images of each target, extracting the effective phase values ​​of the effective pixels in the target image corresponding to each projected image. For each projection zone, a multi-step phase-shifting algorithm is used to calculate the absolute phase of the effective region, and these are spatially stitched together to form an initial absolute phase map covering the entire field of view. This refers to the target phase map. Then, based on the phase shift intensity distribution formula, the fringe modulation map for each pixel across the entire field is calculated. .

[0245] Specifically, the stripe tone diagram The calculation formula is:

[0246]

[0247] This fringe modulation pattern objectively characterizes the signal-to-noise ratio and solution reliability of the phase signal at the corresponding pixel. Among them, The fringe modulation magnitude at the image plane coordinates (x, y); N is the total number of phase shift steps set by the system; n is the step index of the current phase shift map (n=1, 2, ..., N); This represents the grayscale intensity response value actually acquired by the image acquisition device at pixel (x,y) under the nth phase-shift projection.

[0248] Combination Figure 10a As shown, Figure 10a This is a schematic diagram of the phase optimization network in one embodiment. In this embodiment, the target phase map is optimized based on the target phase map, the fringe modulation value, the interreflection mask (which can be removed), the pure specular mask, and the normal diffuse mask. The optimization includes: cascading the target phase map, the fringe modulation value, the interreflection mask, the pure specular mask, and the normal diffuse mask through the input network of a pre-trained boundary optimization model; extracting features from the cascaded features using a multi-layer cascaded feature extraction network of the pre-trained boundary optimization model, which includes local residuals, to obtain a phase residual correction term; and adding the phase residual correction term and the target phase map pixel by pixel using a global skip connection network of the pre-trained boundary optimization model to obtain the optimized target phase map.

[0249] Combination Figure 10a The input of the phase optimization network receives a concatenated tensor (Concat) consisting of three parts: a preliminary spatial fusion phase map. Evaluation chart of fringe modulation obtained by solving the phase shift formula. And the spatial prior mask tensor containing splicing boundary location information (by... , , The extracted signal can also be obtained by removing the interreflection mask in other embodiments. The modulation graph provides a weighted reference for the reliability of regional signals in the network; while the spatial mask, as a hard prior, mathematically forces the network's receptive field to be locked only in narrow gap regions with physical splicing abrupt changes, thus cutting off the possibility of the network tampering with the normal phase of non-boundary continuous regions from the root of the algorithm.

[0250] The cascaded tensor is input into a multi-layered cascaded feature extraction network. This backbone aims to extract multi-scale boundary distortion context features. Local residual connections are embedded within the hidden layers of this network. During deep feature downsampling and spatial resolution restoration, these local residual connections across layers maximize the preservation of extremely subtle high-frequency real physical textures extracted by shallower networks, preventing detail degradation during deep convolution. This allows the network to keenly distinguish between step artifact noise caused by hardware defocusing and the object's true, subtle high-frequency geometric textures. The network ultimately passes through the end... The convolutional layer outputs only a single continuous phase residual correction term. For details regarding the limitations of this section, please refer to [link / reference]. Figure 10b This will not be elaborated upon here.

[0251] At the very end of the network, a global skip connection spanning the entire feature extraction backbone is used to process the raw input without any processing. Directly derived, and compared with the predicted residual term By adding pixels one by one, the final optimized phase map is obtained:

[0252]

[0253] This formula is the core to ensuring measurement accuracy. For normal diffuse reflection regions far from the splicing mask, the network output is subject to prior constraints. ,Right now It retains 100% of the extremely high analytical accuracy achieved by the traditional optical phase-shifting method; the network only utilizes non-zero [variable] at the physical boundaries. Nonlinear smooth transitions are applied to abnormal high-frequency gradients, perfectly balancing seamless stitching with absolute realism of point cloud features.

[0254] In some optional embodiments, the optimization is achieved by cascading the target phase map, the fringe modulation value, the interreflection mask, the pure specular mask, and the normal diffuse mask through the input network of a pre-trained boundary optimization model, combined with... Figure 10b First, edge processing is performed on each of the projection partition masks, and the target phase map is compensated based on the edge-processed partition masks to obtain a pre-compensated phase map; a spatial attention weighted mask is generated based on each of the projection partition masks and the fringe modulation map; then, a residual phase feature map is generated based on the spatial attention weighted mask and the pre-compensated phase map, replacing... Figure 10a The phase residual correction term is obtained by extracting features from the cascaded features using a pre-trained boundary optimization model and a multi-layered cascaded feature extraction network that includes local residuals. It should be noted that... Figure 10a Multi-layer cascaded feature extraction network for local residuals in Figure 10bThe structure and principle are similar to those in [other applications], but this application only addresses [specific aspects]. Figure 10b The local residuals are constrained by a multi-layered cascaded feature extraction network.

[0255] In this embodiment, the partition masks used by the projection device during partition projection (including K non-interfering sub-regions, the region corresponding to the pure specular mask after attenuation of the attenuation coefficient map, and the normal diffuse mask, or the K non-interfering sub-regions can be removed) can be extracted first to form a partition mask set. This is used to perform edge processing on each projection partition mask.

[0256] In this embodiment, for ease of description, the K non-interfering sub-regions, the region corresponding to the pure specular mask after attenuation of the attenuation coefficient map, and the normal diffuse reflection mask are referred to as projection partition masks. First, edge processing is performed on each projection partition mask, including: extracting the boundary connected region of each projection partition mask; performing a two-dimensional morphological dilation operation on the boundary connected region to obtain the boundary prior range.

[0257] Specifically, the partition mask set can be extracted in this application. The boundary connected domains of each independent partition mask are obtained, and a two-dimensional morphological dilation operation is performed to generate the boundary prior range covering all spatial splicing abrupt regions.

[0258] The target phase map is then compensated based on the processed edges to obtain a pre-compensated phase map. The compensation here mainly targets the physical splicing boundary, that is, the extreme phase jump at the boundary prior range, to reduce the fitting difficulty of the subsequent network.

[0259] In some optional embodiments, the target phase map is compensated based on the edge-processed partition masks to obtain a pre-compensated phase map, including: extracting phase pixels in the target phase map that are located within the boundary prior range; filtering the phase pixels; and obtaining the pre-compensated phase map based on the filtered phase pixels and the target phase map.

[0260] In this embodiment, due to hardware defocusing effects and differences in zone exposure parameters, the initial absolute phase map, i.e., the target phase map, is determined. Within the aforementioned boundary prior range, there exist abrupt polarity jumps and nonlinear steps. To mitigate this hard edge jump, an initial absolute phase map is extracted. The pre-compensated phase map is generated by filtering the phase pixels located within the boundary prior range, such as by performing a two-dimensional Gaussian low-pass filter. The standard deviation σ of the Gaussian low-pass filter is based on the theoretical optimal value σ0 = D / 6 corresponding to the circle of confusion of the projector optical system, and its range is set to... It only smooths the diffusion noise and grayscale distortion introduced by optical hardware defocusing, while preserving the intrinsic high-frequency geometric contour features of the target object.

[0261] In the above embodiments, a phase soft edge pre-compensation mechanism is introduced to perform physical-level gradient smoothing on extreme jumps at the algorithm source, significantly reducing the fitting pressure on the neural network. Combined with the large receptive field spatial gating prediction within the BMSG module, the system can adaptively cover different degrees of defocus diffusion, achieving a more thorough and smoother seamless fusion effect. The soft edge pre-compensation operation performs preliminary smoothing of abnormal high-frequency gradients at the gaps at the physical level, effectively reducing the numerical difficulty for subsequent deep learning networks to predict residuals for large step errors.

[0262] Among them, combined Figure 10b In this embodiment, an independent boundary-modulation collaborative gating module is constructed. This module deeply fuses the boundaries of discrete projection partition masks with continuous modulation features, and outputs a spatial attention weight mask for controlling the network repair intensity. .

[0263] This boundary-modulation co-gating module can be an attention weight generation network. This network can extract pure boundary geometric prior information and eliminate modal aliasing between the hard boundary of the mask geometry and the continuous physical degradation features of the modulation. It achieves an organic unity of discrete structural prior and continuous physical prior. Finally, through a large receptive field convolutional layer, the network can simulate the physical diffusion trend of overexposure energy in space in the optical imaging system. This makes the generated attention weight mask not only have a high response at the overexposure boundary, but also produce a smooth transition of weight attenuation in the modulation degradation region near the boundary. This guides the restoration network to achieve spatially adaptive specular restoration.

[0264] In this process, after obtaining the spatial attention weighted mask, the spatial attention weighted mask is weighted with the pre-compensated phase map to obtain the weighted phase feature map. This allows for explicit physical constraints on the pre-compensated phase map, guiding the network to concentrate computational resources on splicing anomaly regions for multi-scale residual prediction.

[0265] In some optional embodiments, generating a weighted phase feature map based on a spatial attention weighted mask and a pre-compensated phase map includes: performing shallow feature extraction on the pre-compensated phase map to obtain a shallow phase feature map; and multiplying the shallow phase feature map with the spatial attention weighted mask pixel by pixel to obtain a weighted phase feature map.

[0266] Among them, combined Figure 10b In this embodiment, the obtained pre-compensated phase map can be used Enter separately into one In the convolutional layer, a shallow phase feature map containing the basic topology and high-frequency physical details of the object is extracted. Then, the shallow phase feature map... Compared with the predicted spatial attention weighted mask A weighted phase feature map is obtained by performing pixel-by-pixel multiplication. This multiplication gating operation utilizes the underlying physical prior features to forcibly cut off and suppress the network's data response to non-boundary, high-modulus normal phase regions, so that the effective information of the weighted phase feature map is highly focused on the physical distortion gap regions that need to be repaired.

[0267] An optimized target phase map is obtained based on the weighted phase feature map and the pre-compensated phase map. A 3D point cloud of the object under test is then generated based on the optimized target phase map.

[0268] Prediction based on the weighted phase feature map yields a continuous phase residual correction term, which, combined with the pre-compensated phase map, results in an optimized target phase map. Finally, generating a 3D point cloud of the object under test based on this optimized target phase map improves prediction accuracy.

[0269] In some alternative embodiments, combined with Figure 10c As shown, Figure 10c Here is a structural diagram of the attention weight generation network in one embodiment. A spatial attention weighting mask is generated based on each projection partition mask and the fringe modulation map, including:

[0270] F1: The initial convolutional layer of the network is generated by attention weights, and the boundary prior feature map of a single channel is obtained by cross-channel fusion of the projection partition masks.

[0271] This step will include A set of partition masks for each channel. The input is fed into the initial convolutional layer, which is a single convolutional layer. Convolutional layer. This initial convolutional layer acts as a linear combiner across channels, adaptively learning the weight distribution at the boundaries of different partitions, compressing and aggregating a variable number of multi-channel discrete masks into a single-channel boundary prior feature map. This operation effectively solves the problem of mismatch in network input dimensions caused by dynamic changes in the number of partitions.

[0272] F2: A cascaded layer of the network is generated by attention weights, and the boundary prior feature map and the stripe modulation map are cascaded in the channel dimension to obtain a fused feature map.

[0273] F3: A deep prior fusion layer is generated through attention weights to extract features from the fused feature map and obtain a high-dimensional joint prior feature map.

[0274] This step involves deep joint prior feature extraction, specifically the boundary prior feature map of a single channel. Compared with single-channel stripe modulation graph The cascaded feature maps are obtained by concatenating them along the channel dimension, and then fed into the deep prior fusion layer.

[0275] In some optional embodiments, a high-dimensional joint prior feature map is obtained by extracting features from the fused feature map through the deep prior fusion layer of the attention weight generation network. This includes: extracting local spatial common features from the fused feature map through the pre-convolutional layer and non-linear activation layer in the deep prior fusion layer, and expanding the receptive field through the post-convolutional layer in the deep prior fusion layer to obtain the high-dimensional joint prior feature map.

[0276] The deep prior fusion layer contains, in sequence, one A pre-convolutional layer, a LeakyReLU non-linear activation layer, and a Post-convolutional layers. The pre-convolutional layers and non-linear activation operations aim to initially extract the local spatial co-occurrence features of both, while the post-convolutional layers further expand the receptive field. Through this fusion block, the network successfully eliminates modality mixing between the geometric hard boundaries of the mask and the continuous physical degradation features of the modulation, outputting a high-dimensional joint prior feature map. .

[0277] F4: The spatial attention mechanism layer of the network is generated through attention weights. It performs different types of feature extraction on the high-dimensional joint prior feature map and generates a spatial attention weighted mask based on the extracted different types of feature maps.

[0278] This step involves the joint prior feature map. Different types of feature extraction are performed along the channel dimension, such as global max pooling and global average pooling (other types can also be used in other embodiments), to obtain two single-channel feature maps representing significant spatial responses, and then the two are concatenated along the channel dimension. Subsequently, the concatenated features are input into a... In a large receptive field convolutional layer, the physical energy diffusion range of the optical blur circle at the boundary of the projection device is fully covered. Finally, through a sigmoid activation function, a two-dimensional spatial attention weighted mask with values ​​strictly between 0 and 1 is output. .

[0279] In the above embodiments, by constructing a boundary-modulation collaborative gating module and supplementing it with a global identity mapping mechanism, spatial control over the repair behavior is achieved. This allows the system to perform nonlinear compensation only for the physical gap region, while achieving zero tampering with the non-boundary, high-reliability region that occupies most of the entire field area, thus preserving 100% of the extremely high initial solution accuracy of traditional optical algorithms.

[0280] In some optional embodiments, the optimized target phase map is obtained based on the weighted phase feature map and the pre-compensated phase map, including: extracting features from the residual phase map through a multi-layer cascaded residual feature extraction network including local residuals to obtain a phase residual correction term; and adding the phase residual correction term and the pre-compensated phase map pixel by pixel through a global skip connection network to obtain the optimized target phase map.

[0281] In this embodiment, the weighted residual phase map is input into a feature extraction network composed of multiple cascaded residual blocks. This network aims to extract multi-scale boundary distortion context features. Local residual connections are embedded within the hidden layers of this network. During deep feature downsampling and spatial resolution restoration, these local residual cross-layer connections maximize the preservation of extremely subtle high-frequency real physical textures extracted by shallower networks, preventing detail degradation during deep convolution. This allows the network to keenly distinguish between step artifact noise caused by hardware defocusing and the object's true, subtle high-frequency geometric textures. The network terminates at a point... The convolutional layer performs channel dimensionality reduction and outputs a continuous phase residual correction term for a single channel. .

[0282] Finally, the phase residual correction term and the pre-compensated phase map are added pixel by pixel to obtain the optimized target phase map. Specifically, the residual superposition is completed by using jump connections, which eliminates stitching artifacts while ensuring the accuracy of global 3D measurement.

[0283] In practical applications, a global skip connection network spanning the entire feature extraction backbone is used to process the pre-compensated phase map that has not been processed by the network backbone. Directly compared with the continuous phase residual correction term predicted at the network end The optimized target phase map is obtained by pixel-by-pixel summation, and its physical calculation formula is the optimized target phase map. This identity mapping mechanism ensures that for regions far from the splicing mask, the residual correction term output by the network under pre-gating constraints is close to zero, thus absolutely preserving the extremely high accuracy calculated by the traditional optical phase-shifting method; and achieving a seamless and smooth transition by superimposing effective compensation values ​​only at the physical boundaries.

[0284] In some optional embodiments, the image acquisition device is a binocular image acquisition device, and the binocular image acquisition device includes a first image acquisition device and a second image acquisition device; generating a 3D point cloud of the object under test based on the optimized target phase map includes: determining a first confidence region and a second confidence region based on the optimized target phase maps corresponding to the first image acquisition device and the second image acquisition device, wherein the mask value corresponding to the first confidence region is less than the mask value corresponding to the second confidence region; dynamically adjusting the matching strategy of the optimized target phase map corresponding to the first image acquisition device and the optimized target phase map corresponding to the second image acquisition device based on the first confidence region and the second confidence region to obtain a matching result, and generating a 3D point cloud of the object under test based on the matching result.

[0285] In this embodiment, the image acquisition device is a binocular image acquisition device, which includes a first image acquisition device and a second image acquisition device. Each image acquisition device obtains a spatial attention weighting mask according to the steps described above. Assume that the two-dimensional spatial attention weighting masks generated by the first and second image acquisition devices are respectively... and Define the high-confidence region, i.e., the first confidence region (…). The region approaching 0 represents the dominant region of the original optical solution accuracy. The boundary region to be optimized is defined as the second confidence level region. The region with a confidence level significantly greater than 0 is the dominant region for network residual compensation, thus the mask value corresponding to the first confidence region is less than the mask value corresponding to the second confidence region. This division of confidence regions is used to guide the dynamic allocation of weights in the subsequent matching cost function.

[0286] The matching cost function is used to match the pixels in the optimized target phase maps corresponding to the first and second image acquisition devices to determine the matching pixel pairs, and then generate the 3D point cloud of the object under test based on these pixel pairs.

[0287] In some optional embodiments, based on a first confidence region and a second confidence region, the matching strategy of the optimized target phase map corresponding to the first image acquisition device and the optimized target phase map corresponding to the second image acquisition device is dynamically adjusted to obtain the matching result. This includes: obtaining a phase cost term based on the spatial attention weighted masks corresponding to the first and second image acquisition devices; generating a physical correlation cost term corresponding to the point to be matched in the optimized target phase map corresponding to the first or second image acquisition device, centered on the point to be matched in the first or second image acquisition device; determining that the cost weight of the phase cost term of the point to be matched located in the first confidence region is greater than the cost weight of the physical correlation cost term, and determining that the cost weight of the phase cost term of the point to be matched located in the second confidence region is less than the cost weight of the physical correlation cost term; and matching the optimized target phase map corresponding to the first and second image acquisition devices based on the phase cost term, the physical correlation cost term, and the cost weight.

[0288] On the epipolar constraint path of the binocular system, for the pixels to be matched in the first image acquisition device. Establish a joint matching cost function that integrates phase information and physical texture features. :

[0289]

[0290] Phase cost term It calculates the optimized target phase map from the first and second image acquisition devices. and The absolute phase difference between them.

[0291] Physical correlation cost term A local window is constructed centered on the point to be matched, and the obtained stripe modulation map is used. Calculate the zero-mean normalized cross-correlation (ZNCC) value. Since the modulation plot reflects the intrinsic physical reflectivity characteristics of the object's surface, it can serve as an auxiliary physical anchor point in addition to the phase value.

[0292] Using spatial attention weighted masks Real-time adjustment of matching weight coefficients and .

[0293] set up For spatial attention weighted masks The monotonically increasing mapping function, the matching weight coefficients of the first and second image acquisition devices. and Follow these constraints:

[0294]

[0295]

[0296] in, This is the global gain coefficient, used to control the maximum proportion of correlation cost.

[0297] Mapping function Any of the following can be selected, and other options may also be selected in other embodiments; no specific restrictions are imposed here: Linear mapping model (suitable for general out-of-focus scenarios): That is, the weights shift linearly with the repair intensity. Nonlinear power function model (enhancing boundary robustness): (generally This model enables spatial attention-weighted masks. At lower (non-boundary region), Maintain absolute dominance; only when spatial attention-weighted mask The weights only switch rapidly when the value approaches 1. .

[0298] The phase cost term weight of the matching point located in the first confidence zone is determined to be greater than the cost weight of the physical correlation cost term. That is, in the high confidence zone outside the boundary, the phase cost weight is set. Dominantly, it utilizes the high precision characteristics of the phase-shifting method to achieve sub-pixel-level accurate matching.

[0299] The cost weight of the phase cost term for the matching point located in the second confidence region is determined to be less than the cost weight of the physical correlation cost term, i.e., in the spatial attention weighted mask. The marked boundary region is adaptively reduced by the system as the weight value increases. And significantly increase the weight of physical correlation cost. This mechanism weakens the reliance on the consistency of phase values ​​after network repair, and instead relies on the local modulation texture features of the object to lock in the correct corresponding points, effectively avoiding matching misalignment caused by the slight phase deviation introduced by the independent repair of the left and right cameras.

[0300] Finally, based on the phase cost term, the physical correlation cost term, and the cost weight, the optimized target phase map corresponding to the first image acquisition device and the optimized target phase map corresponding to the second image acquisition device are matched.

[0301] Optionally, after obtaining the optimized target phase map, a spatial attention-weighted mask can also be applied. For regions exceeding a preset threshold (i.e., the boundary regions of the projection partitions), a left-right consistency check (L-RConsistencyCheck) is performed to remove mismatched points caused by occlusion or extreme defocus. Guided plane fitting or interpolation optimization is then performed using high-confidence disparity data from both sides of the boundary to ultimately obtain a continuous and physically consistent disparity distribution across the entire field, providing reliable input for subsequent high-precision 3D reconstruction.

[0302] Finally, using pre-calibrated intrinsic and extrinsic parameters of the binocular system, the obtained optimized disparity map and the generated high-fidelity absolute phase map are geometrically reprojected. Through the principle of spatial triangulation, the pixel coordinates and their absolute phases are transformed into three-dimensional coordinates (X,Y,Z) in the world coordinate system, ultimately outputting a high-precision three-dimensional point cloud of the metal part with no boundary gap artifacts and complete topology.

[0303] In some optional embodiments, the weighted phase feature map is obtained based on a pre-trained residual prediction model. The training method of the residual prediction model includes: processing the surface of the sample object to be tested and scanning the sample object to be tested after avoiding processing to obtain the sample labeled phase; scanning the sample object to be tested without surface processing and obtaining the sample target phase map, sample fringe modulation map and sample projection partition mask based on the scanning results; obtaining the sample residual correction term based on the sample labeled phase and sample target phase map; inputting the sample target phase map, sample fringe modulation map and sample projection partition mask into the residual prediction model to be trained, and using the sample residual correction term as a supervision signal for training to obtain the trained residual prediction model.

[0304] To enable the aforementioned local residual learning network based on boundary-modulation co-gating to possess high-precision boundary repair capabilities, a high-quality dataset needs to be constructed for offline training before performing online measurements. Since severe high reflectivity and complex interreflection can lead to severe phase distortion, it is difficult to obtain reliable supervision labels using conventional algorithms. Therefore, this application proposes a ground truth acquisition method based on physical coating transformation, and the specific training process is as follows:

[0305] First, the physical transformation and phase truth acquisition steps for an ideal diffuse reflective surface are performed, including: selecting a representative test workpiece with complex geometric topology (such as deep grooves and steep walls) and high dynamic range reflectivity (such as highly reflective metal components) as the sample test object. The workpiece undergoes surface treatment, such as uniformly spraying paint or applying a developer, using this physical coating to completely mask the workpiece's original high reflectivity and material properties that easily induce mutual reflection, forcibly transforming its surface optical properties into an ideal Lambertian diffuse reflective surface. The treated workpiece is then scanned, for example, using standard non-partitioned multi-step phase-shifting structured light scanning. Since the surface has now been transformed into ideal diffuse reflection, there is no mirror overexposure truncation or optical path crosstalk from multiple bounces. The absolute phase map calculated by the system has extremely high spatial resolution and physical geometric accuracy, and is used as the sample target phase map under the corresponding spatial pose. .

[0306] Then, the physical original input feature alignment acquisition step is performed, including: selecting identical workpieces processed in the same batch that have not been sprayed, and accurately fixing them in the phase map of the sample target. The spatial pose is exactly the same during acquisition. The controller executes the aforementioned online measurement steps, projects partitioned fringes onto the original workpiece according to the set partitioning strategy, and calculates and extracts the pre-compensated phase map with physical splicing tomography and step artifacts. Striped mode diagram and partition mask set This step ensures that the model's input features and ground truth values ​​are tightly aligned in the spatial pixel coordinate system.

[0307] Finally, boundary residual ground truth construction and network loop closure optimization are performed, including: calculating the full-field absolute phase ground truth. With pre-compensation phase diagram The pixel-by-pixel physical deviation between the values ​​is used to construct the true value of the residual correction term, and its physical calculation formula is as follows: The multimodal tensor obtained above ( , and The residual prediction model to be trained is input to the residual correction term truth value. As a rigorous supervisory label, during the training phase, a smoothed L1 loss function is preferentially used to calculate the prediction residual correction term. With truth value The loss between the boundary step gradient values ​​is used to suppress the dominant interference of the boundary step gradient values ​​on the convergence direction of the model training. The network weights of the boundary-modulation co-gating module and the multi-layer residual feature extraction backbone are updated iteratively by combining the backpropagation algorithm. After closed-loop optimization on large-scale data until the model converges, it has the generalization ability to accurately predict physical gap residuals on the surface of unknown complex high reflective workpieces.

[0308] In some alternative embodiments, combined with Figure 11 As shown, Figure 11 The following is a flowchart illustrating a structured light-based 3D point cloud generation method in another embodiment. First, a highly reflective object to be measured is prepared. A target image is projected onto the object using a projection device, and an initial fringe pattern is acquired using an image acquisition device. This initial fringe pattern is then input into a multi-class semantic segmentation model to obtain a mutual reflection mask, a pure specular mask, and a normal diffuse reflection mask. Subsequently, spatiotemporal joint modulation and fusion calculations are performed based on the mutual reflection mask, pure specular mask, and normal diffuse reflection mask to obtain a global spatial fusion modulation matrix. Then, soft edge pre-compensation is performed on the mask boundaries to obtain an optimized spatial fusion modulation matrix. Based on the optimized spatial fusion modulation matrix, parallel projection and synchronous acquisition by the camera (image acquisition device) are performed to obtain a modulated structured light fringe pattern (i.e., the target image). Multi-region phase calculation and preliminary spatial fusion are performed on the adjusted structured light fringe pattern to obtain a full-field initial phase map (i.e., the target phase map). Finally, a boundary refinement network based on residuals, combined with the mutual reflection mask, pure specular mask, and normal diffuse reflection mask, generates a 3D point cloud of the object to be measured.

[0309] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. It is understood that the steps in different embodiments can be freely combined as needed, and all non-contradictory solutions formed by such combinations are within the scope of protection of this application.

[0310] Based on the same inventive concept, this application also provides a highly reflective 3D point cloud generation device for implementing the aforementioned highly reflective 3D point cloud generation method. The solution provided by this device is similar to the solution described in the above method. Therefore, the specific limitations in one or more embodiments of the highly reflective 3D point cloud generation device provided below can be found in the limitations of the highly reflective 3D point cloud generation method described above, and will not be repeated here.

[0311] In one exemplary embodiment, such as Figure 12 As shown, a three-dimensional point cloud generation device based on high reflectivity is provided, comprising:

[0312] The acquisition module is used to acquire the initial image of the object under test under the full white field image projected by the projection device through the image acquisition device;

[0313] The segmentation module is used to process the initial image to obtain a pure specular mask, a normal diffuse mask, and a target brightness attenuation coefficient matrix;

[0314] The projection image generation module is used to modulate the reference image sequence of the projection device based on the target brightness attenuation coefficient matrix to obtain the target projection image.

[0315] The projection module is used to project a target image through a projection device and to acquire a target image of the object under test under the target image projected by the projection device through an image acquisition device.

[0316] The model generation module is used to generate a 3D point cloud of the object under test based on the target image, pure specular mask, and normal diffuse mask.

[0317] In some optional embodiments, the segmentation module is specifically used to process the initial image to obtain a high-dimensional feature map through a multi-layer cascaded inverse residual feature extraction network of a pre-trained segmentation model; to segment the high-dimensional feature map to obtain a pure specular mask and a normal diffuse reflection mask through the segmentation network of the pre-trained segmentation model; and to process the high-dimensional feature map based on the pure specular mask through the brightness regression network of the pre-trained segmentation model to obtain the target brightness attenuation coefficient matrix.

[0318] In some optional embodiments, the segmentation module is specifically used to perform dimensionality expansion on the initial image through the dimensionality expansion layer of a multi-layer cascaded inverse residual feature extraction network; to extract features from the dimensionality-expanded initial image through the lightweight spatial filtering layer of the multi-layer cascaded inverse residual feature extraction network to obtain a target feature map; and to reduce the dimensionality of the target feature map through the linear dimensionality reduction projection layer of the multi-layer cascaded inverse residual feature extraction network, and then fuse it with the initial image output from the residual skip connection layer of the multi-layer cascaded inverse residual feature extraction network to obtain a high-dimensional feature map.

[0319] In some optional embodiments, the segmentation module is specifically used to process the high-dimensional feature map through the brightness regression network of the pre-trained segmentation model to obtain the global initial brightness attenuation coefficient matrix; and to process the global initial brightness attenuation coefficient matrix based on the pure specular mask to obtain the target brightness attenuation coefficient matrix.

[0320] In some optional embodiments, the processing of the initial image to obtain the pure specular mask, normal diffuse mask, and target brightness attenuation coefficient matrix is ​​achieved through a pre-trained segmentation model; the aforementioned apparatus includes: a training module, used to acquire reference images after a full white field image is projected onto the sample test object by a projection device through an image acquisition device with multiple globally decreasing global brightness coefficients; based on each reference image, determining the sample global brightness coefficient corresponding to the projection device when the gray value of any pixel in each reference image under the field of view of the image acquisition device is at the upper limit of the optimal linear response interval of the image acquisition device; when the pixels in the target reference image have undergone gray-level saturation truncation or the sample global brightness coefficient of the pixel is less than the target value, generating a sample pure specular mask based on the pixel, and generating a sample normal diffuse mask based on the sample pure specular mask; the target reference image is the reference image corresponding to the projection of the full white field image with maximum brightness; and training the model based on each reference image, the sample pure specular mask, the sample normal diffuse mask, and the sample global brightness coefficient to obtain a segmentation model.

[0321] In some optional embodiments, the training module is specifically used to process the reference image through the segmentation model to obtain a predicted pure specular mask, a predicted normal diffuse mask, and a predicted global luminance coefficient; generate a mask segmentation loss function based on the predicted pure specular mask and the sample pure specular mask; generate a base loss function based on the predicted global luminance coefficient and the sample global luminance coefficient; simulate the physical space light modulation process based on the predicted global luminance coefficient and the full white field image to generate a modulated image, and generate a physical response consistency loss function under the supervision of the sample pure specular mask based on the modulated image and the ideal linear response target image; perform morphological dilation based on the sample pure specular mask to obtain a transition mask, and generate a boundary smoothing penalty loss function under the supervision of the transition mask based on the spatial gradient of the predicted global luminance coefficient; generate a target loss function based on the mask segmentation loss function, the base loss function, the physical response consistency loss function, and the boundary smoothing penalty loss function; and perform gradient backpropagation based on the target loss function to obtain the trained segmentation model.

[0322] In some optional embodiments, the above-mentioned apparatus further includes: a mapping module, configured to acquire a second speckle image of the sample object under a first speckle image projected by a projection device through an image acquisition device; establish a mapping relationship between the field of view of the image acquisition device and the field of view of the projection device based on the first speckle image and the second speckle image; process the target brightness attenuation coefficient matrix based on the mapping relationship to obtain an attenuation coefficient map corresponding to the pure specular mask under the field of view of the projection device; the above-mentioned projection image generation module is specifically used to modulate the reference image sequence of the projection device based on the attenuation coefficient map to obtain a target projection image.

[0323] The modules in the aforementioned highly reflective 3D point cloud generation device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the computer device's memory as software, so that the processor can call and execute the corresponding operations of each module.

[0324] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 13 As shown, this computer device includes a processor, memory, input / output interfaces (I / O), and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores XX data. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network. When the computer program is executed by the processor, it implements a method for generating 3D point clouds based on high reflectivity.

[0325] In one exemplary embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 13As shown, the computer device includes a processor, memory, input / output interfaces, a communication interface, a display unit, and an input device. The processor, memory, and input / output interfaces are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interfaces are used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, Near Field Communication (NFC), or other technologies. When executed by the processor, the computer program implements a method for generating 3D point clouds based on high reflectivity. The display unit is used to form a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0326] Those skilled in the art will understand that Figure 12 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0327] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0328] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0329] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0330] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0331] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0332] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0333] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for generating three-dimensional point clouds based on high reflectivity, characterized in that, The method includes: The initial image of the object under test is acquired by the image acquisition device under the full white field image projected by the projection device. The initial image is processed to obtain a pure specular mask, a normal diffuse mask, and a target brightness attenuation coefficient matrix; Based on the target brightness attenuation coefficient matrix, the reference image sequence of the projection device will be modulated to obtain the target projection image; The target projection image is projected by the projection device, and the target image of the object under test is acquired by the image acquisition device under the target projection image projected by the projection device. A 3D point cloud of the object under test is generated based on the target image, the pure specular mask, and the normal diffuse mask.

2. The method according to claim 1, characterized in that, The process of processing the initial image to obtain a pure specular mask, a normal diffuse mask, and a target brightness attenuation coefficient matrix includes: The initial image is processed by a multi-layered cascaded inverse residual feature extraction network of a pre-trained segmentation model to obtain a high-dimensional feature map. The high-dimensional feature map is segmented using a pre-trained segmentation model's segmentation network to obtain pure specular masks and normal diffuse masks. The high-dimensional feature map is processed using the pre-trained segmentation model's luminance regression network based on the pure specular mask to obtain the target luminance attenuation coefficient matrix.

3. The method according to claim 2, characterized in that, The high-dimensional feature map is obtained by processing the initial image through a multi-layered cascaded inverse residual feature extraction network of a pre-trained segmentation model, including: The initial image is dimensionally expanded by the dimension-up expansion layer of the multi-layer cascaded inverse residual feature extraction network. The target feature map is obtained by extracting features from the initial image after dimensional expansion through the lightweight spatial filtering layer of the multi-layer cascaded inverse residual feature extraction network. The target feature map is dimensionality reduced by the linear dimensionality reduction projection layer of the multi-layered cascaded inverse residual feature extraction network, and then fused with the initial image output by the residual skip connection layer of the multi-layered cascaded inverse residual feature extraction network to obtain a high-dimensional feature map.

4. The method according to claim 2, characterized in that, The high-dimensional feature map is processed using a pre-trained segmentation model's luminance regression network based on the pure specular mask to obtain the target luminance attenuation coefficient matrix, including: The high-dimensional feature map is processed by the brightness regression network of the pre-trained segmentation model to obtain the global initial brightness attenuation coefficient matrix. The target brightness attenuation coefficient matrix is ​​obtained by processing the global initial brightness attenuation coefficient matrix based on the pure specular mask.

5. The method according to any one of claims 1 to 4, characterized in that, The process of processing the initial image to obtain the pure specular mask, normal diffuse mask, and target brightness attenuation coefficient matrix is ​​achieved through a pre-trained segmentation model; the training method of the segmentation model includes: The image acquisition device acquires reference images of the full white field image projected onto the sample object under test using multiple global brightness coefficients that decrease in gradient. Based on each of the reference images, when the gray value of any pixel in each of the reference images under the field of view of the image acquisition device is at the upper limit of the optimal linear response range of the image acquisition device, the global brightness coefficient of the projection device is determined. If a pixel in the target reference image is truncated due to grayscale saturation or the global luminance coefficient of the pixel is less than the target value, a pure specular mask is generated based on the pixel, and a normal diffuse mask is generated based on the pure specular mask; the target reference image is the reference image corresponding to the projection of the full white field image with maximum luminance. A segmentation model is obtained by training the model based on the reference images, the pure specular mask of the sample, the normal diffuse mask of the sample, and the global luminance coefficient of the sample.

6. The method according to claim 5, characterized in that, The segmentation model is obtained by training the model based on each of the reference images, the pure specular mask of the sample, the normal diffuse reflectance mask of the sample, and the global brightness coefficient of the sample, including: The reference image is processed by a segmentation model to obtain a predicted pure specular mask, a predicted normal diffuse mask, and a predicted global luminance coefficient. A mask segmentation loss function is generated based on the predicted pure specular mask and the sample pure specular mask; A basic loss function is generated based on the predicted global brightness coefficient and the sample global brightness coefficient; Based on the predicted global brightness coefficient and the full white field image, the physical space light modulation process is simulated to generate a modulated image. Based on the modulated image and the ideal linear response target image, under the supervision of the sample pure specular mask, a physical response consistency loss function is generated. Morphological dilation is performed based on the sample pure specular mask to obtain a transition mask, and a boundary smoothing penalty loss function is generated under the supervision of the transition mask based on the spatial gradient of the predicted global brightness coefficient. A target loss function is generated based on the mask segmentation loss function, the basic loss function, the physical response consistency loss function, and the boundary smoothing penalty term loss function; Gradient backpropagation is performed based on the target loss function to obtain the trained segmentation model.

7. The method according to any one of claims 1 to 4, characterized in that, Before modulating the reference image sequence of the projection device based on the target brightness attenuation coefficient matrix to obtain the target projected image, the process further includes: A second speckle image of the sample object under test is obtained by an image acquisition device under a first speckle image projected by a projection device; Based on the first speckle image and the second speckle image, a mapping relationship is established between the field of view of the image acquisition device and the field of view of the projection device; Based on the mapping relationship, the target brightness attenuation coefficient matrix is ​​processed to obtain the attenuation coefficient map corresponding to the pure specular mask under the field of view of the projection device; The reference image sequence of the projection device is modulated based on the target brightness attenuation coefficient matrix to obtain the target projected image, including: Based on the attenuation coefficient diagram, the reference image sequence of the projection device will be modulated to obtain the target projection image.

8. A three-dimensional point cloud generation device based on high reflectivity, characterized in that, The device includes: The acquisition module is used to acquire the initial image of the object under test under the full white field image projected by the projection device through the image acquisition device; The segmentation module is used to process the initial image to obtain a pure specular mask, a normal diffuse mask, and a target brightness attenuation coefficient matrix; The projection image generation module is used to modulate the reference image sequence of the projection device based on the target brightness attenuation coefficient matrix to obtain the target projection image; The projection module is used to project the target projection image through the projection device and to acquire the target image of the object under test under the target projection image projected by the projection device through the image acquisition device. The model generation module is used to generate a 3D point cloud of the object under test based on the target image, the pure specular mask, and the normal diffuse mask.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images