Point cloud enhancement method and system based on polarization structure perception and trend guided restoration

Abstract

The present disclosure relates to the technical field of fringe projection three-dimensional reconstruction, and particularly relates to a point cloud enhancement method and system based on polarization structure perception and trend guided recovery; the method comprises: acquiring polarization images at multiple angles, generating an object region mask based on a polarization angle consistency mask, a polarization angle gradient mask and a linear polarization degree mask; obtaining an original phase image through four-step phase shifting and a complementary Gray code, and then obtaining an internal missing region of an object according to a hole distribution of the original phase image, adding a morphological structure index and a local variance feature to identify a real self-shadow region; for the identified real self-shadow region, a structure guided recovery model combining a first-order structure gradient guide term and a second-order structure gradient guide term is established to obtain a complete object point cloud; the background and invalid shadow in the object point cloud are separated through the object region mask to obtain a point cloud retaining only the object region.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of fringe projection three-dimensional reconstruction, and particularly relates to a point cloud enhancement method and system based on polarization structure perception and trend guided recovery. BACKGROUND

[0002] Structured light three-dimensional measurement technology is one of the core methods of optical three-dimensional imaging, and is widely used in mechanical engineering, biometrics, machine vision and intelligent manufacturing fields. As an important means of this technology, fringe projection profilometry (FPP) projects a periodic fringe pattern onto the surface of an object, then captures the deformed pattern by a camera, and then extracts the three-dimensional information of the object surface. The wrapped phase is converted into a continuous distribution by a phase unwrapping algorithm, and three-dimensional reconstruction is realized by combining the calibration parameters and the triangulation principle.

[0003] Although the FPP method has made significant progress, the limitations of the optical imaging structure such as the projector and the camera, and the influence of imaging noise, make it inevitable that there are invalid points in the captured fringe pattern in the shadow, background and other areas. At the same time, since the FFP system is derived from a stereo vision system based on the triangulation principle, the viewing angle of the projector is different from that of the camera. For a stereo vision system, it is difficult to completely remove the shadow caused by the protrusions of the measured object. Therefore, after the camera captures the fringe image with self-shadows, the self-shadows will cause errors in the phase image, and the unwrapped phase image obtained after time phase unwrapping (TPU) still has errors caused by self-shadows. This causes holes or breaks in the final point cloud map.

[0004] In addition, the traditional invalid point removal technology relies on image modulation degree information. When the modulation degree difference between the measured object and the background is not obvious, only one threshold is used to separate the object and the background, and the shadow, and there are cases of invalid point deletion and effective point deletion in the separation. The traditional missing point cloud recovery method has low recovery point cloud accuracy when facing complex structures, and the actual use effect is poor. SUMMARY

[0005] The present disclosure provides a point cloud enhancement method and system based on polarization structure perception and trend guided recovery to solve the problems in the background art. The technical solution is as follows:

[0006] In a first aspect, the present disclosure provides a point cloud enhancement method based on polarization structure perception and trend guided recovery, comprising the following steps:

[0007] Obtain a plurality of angle polarization images, calculate the polarization degree and the polarization angle according to the polarization images, and generate an object region mask based on the polarization angle consistency mask, the polarization angle gradient mask and the linear polarization degree mask;

[0008] The original phase map is obtained by four-step phase shift and complementary Gray code, and the missing area inside the object is obtained according to the hole distribution of the original phase map, the morphological structure index and the local variance feature are added to identify the real self-shadow area, the structure hole of the object itself is excluded, and the self-shadow mask is constructed;

[0009] For the identified real self-shadow area, a structure guided recovery model combining a first-order structure gradient guide term and a second-order structure gradient guide term is established, and continuous and smooth phase completion is realized through edge modulation weight constraint of the structure guided recovery model, and complete object point cloud is obtained.

[0010] The background and invalid shadow in the object point cloud are separated through the object region mask to obtain the point cloud only retaining the object region.

[0011] Optionally, the polarization degree and the polarization angle are calculated according to the polarization image, comprising:

[0012] The polarization degree and the polarization angle of the polarization image calculated according to the Stokes parameter and the Mueller matrix are represented as:

[0013] ,

[0014] Wherein: I is the intensity of all light, is the intensity difference of linearly polarized light when the polarization plate angle is 0° and 90°, is the intensity difference of linearly polarized light when the polarization plate angle is 45° and 135°, is the intensity of linearly polarized light when the polarization plate angle is 0°, is the intensity of linearly polarized light when the polarization plate angle is 90°, is the intensity of linearly polarized light when the polarization plate angle is 45°, is the intensity of linearly polarized light when the polarization plate angle is 135°.

[0015] Optionally, the object region mask is generated based on the polarization angle consistency mask, the polarization angle gradient mask and the linear polarization degree mask, comprising:

[0016] The object region mask is represented as:

[0017] ，

[0018] In the formula, is the linear polarization degree mask, is the deflection angle consistency mask, is the polarization angle gradient mask.

[0019] The deflection angle consistency mask is represented as follows:

[0020] ,

[0021] The set threshold is automatically determined using the Otsu method. For consistency of polarization angle;

[0022] The polarization angle gradient mask is represented as follows:

[0023] ,

[0024] The threshold value is determined by histogram valley analysis. For gradient strength;

[0025] The linear polarization degree mask is represented as:

[0026] ,in, The set linear polarization threshold is determined by the Otsu automatic threshold segmentation algorithm.

[0027] Optionally, the self-shadowing mask is represented as:

[0028] ,

[0029] in, The shaded area is represented as:

[0030] ,in, Set values ​​for experience. For typical values, These are morphological parameters, with a value range of [value range missing]. , represented as:

[0031] In the formula, For the region The actual pixel area, Let it be the area of ​​its corresponding minimum convex hull region;

[0032] For local variance, it is expressed as:

[0033] , For the input image, For the region Inner pixel mean, This represents the number of pixels in the region.

[0034] Optionally, for the identified true self-shadowing regions, a structure-guided recovery model combining first-order and second-order structure gradient guidance terms is established, including:

[0035] A Laplace model is constructed using first-order and second-order structure gradient guided terms. Continuous and smooth recovery of missing point clouds is achieved through edge-modulated weight constraints, as shown below:

[0036] ,

[0037] In the formula: , is the weighting factor for the first-order structure gradient guiding term. The weighting factor for the second-order structure gradient guiding term. For the first-order structure gradient guiding term and For the second-order structure gradient guiding term, where, The parameters are obtained by constructing a matrix equation based on the coordinates of the fitted points and their phase values ​​in the actual self-shadow region. , , , , and The value; ,in, , For from the effective phase region The original phase value, This represents the true self-shadow region; and In the effective phase region exist and The first-order partial derivative in the direction is expressed as:

[0038] .

[0039] Optionally, the polarization angle consistency is expressed as:

[0040] ,

[0041] Let the magnitude of the sum of unit vectors be _____. It is the sum of the magnitudes of unit vectors. .

[0042] Optionally, the gradient intensity is expressed as:

[0043] ,in, The horizontal rate of change of polarization direction, The rate of vertical change of polarization direction, Horizontal rotation gradient in polarization direction Vertical rotation gradient in polarization direction.

[0044] The second aspect also provides a point cloud enhancement system based on polarization structure perception and trend guidance recovery, comprising:

[0045] An object region mask acquisition module acquires multi-angle polarization images, calculates a polarization degree and a polarization angle based on the polarization images, and generates an object region mask based on a polarization angle consistency mask, a polarization angle gradient mask, and a linear polarization degree mask.

[0046] A self-shadow mask acquisition module obtains an original phase image through four-step phase shifting and a complementary Golay code, obtains an internal missing region of an object based on a hollow distribution of the original phase image, adds a morphological structure index and a local variance feature to identify a real self-shadow region, excludes a structure hollow of the object itself, and constructs a self-shadow mask.

[0047] A self-shadow region missing point cloud recovery module establishes a structure guidance recovery model combining a first-order structure gradient guidance item and a second-order structure gradient guidance item for the identified real self-shadow region, realizes continuous and smooth phase completion through edge modulation weight constraint of the structure guidance recovery model, and obtains complete object point cloud.

[0048] A three-dimensional reconstruction module separates background and invalid shadow in the object point cloud through the object region mask to obtain point cloud that only retains the object region.

[0049] The third aspect also provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the point cloud enhancement method described above when executing the computer program.

[0050] The fourth aspect also provides a computer readable storage medium having a computer program stored thereon, wherein the program is executed by a processor to implement the point cloud enhancement method described above.

[0051] The advantages or beneficial effects of the above technical solutions at least include:

[0052] 1) The point cloud enhancement method for invalid point removal and missing point cloud recovery proposed by the present disclosure is used to remove invalid points in shadow, background, and other regions in the collected fringe pattern due to the limitations of optical imaging structures such as projectors and cameras, and the influence of imaging noise; the object edge recognition accuracy is enhanced by using a linear polarization degree threshold and polarization angle consistency, polarization angle gradient, and edge feature information, so as to extract the measured object and remove invalid points in the background and invalid shadow.

[0053] 2) The disclosure proposes a three-dimensional point cloud integrity enhancement strategy based on polarization structure perception and trend guided recovery. By identifying real self-shadow areas according to the morphological structure index (Solidity) and local variance characteristics, the object structure hollow area is excluded, the self-shadow area is accurately determined, and the self-shadow mask is constructed.

[0054] 3) The disclosure proposes a global and local structure trend guided missing point cloud recovery method. The recovery of the measured object self-shadow area point cloud is realized, which improves the recovery accuracy compared with the classic linear interpolation method, and makes the object point cloud more complete and the structure level more natural.

[0055] The above summary is merely intended to illustrate the present description and is not intended to limit in any way. In addition to the illustrative aspects, embodiments, and features described above, further aspects, embodiments, and features of the present disclosure will be apparent from the drawings and the following detailed description. BRIEF DESCRIPTION OF DRAWINGS

[0056] In the drawings, like reference numerals refer to same or similar functionalities throughout the several views. The drawings are not necessarily to scale. It is to be understood that these drawings only depict some embodiments in accordance with the disclosure and should not be considered to be limiting thereof.

[0057] Figure 1 A point cloud enhancement method flowchart based on polarization structure perception and trend guided recovery in the embodiment of the disclosure is shown;

[0058] Figure 2 A schematic diagram showing the causes of different shadows in the example of the disclosure is shown;

[0059] Figure 3 A sample and its phase diagram used for invalid point removal in the example of the disclosure are shown;

[0060] Figure 4 A separation effect comparison chart in the example of the disclosure is shown;

[0061] Figure 5 Point cloud reconstruction charts obtained by Otsu, K-mean and our method in the example of the disclosure are shown;

[0062] Figure 6 A schematic diagram showing the identification of different structure hollows and object self-shadows in the example of the disclosure is shown;

[0063] Figure 7 Point cloud reconstruction charts of the linear interpolation method and our method in the example of the disclosure are shown;

[0064] Figure 8The manual cutting part point cloud reconstruction graph in the example of the present disclosure is shown;

[0065] Figure 9 The point cloud recovery effect diagram of the linear interpolation method and our method in the example of the present disclosure is shown;

[0066] Figure 10 The reconstruction graph after the point cloud is recovered by removing invalid points and missing points in the embodiment of the present disclosure is shown.

[0067] Figure 11 The structure block diagram of the point cloud enhancement system based on polarization structure perception and trend guiding recovery in the embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0068] Hereinafter, only certain exemplary embodiments are simply described. As can be appreciated by those skilled in the art, the described embodiments can be modified in various different ways without departing from the spirit or scope of the present disclosure. Therefore, the drawings and the description are considered to be exemplary in nature rather than limiting.

[0069] In order to better illustrate the present disclosure, numerous specific details are given in the following detailed description. It should be understood by those skilled in the art that the present disclosure can also be implemented without some specific details. In some examples, methods, means, elements and circuits well known to those skilled in the art are not described in detail in order to highlight the main idea of the present disclosure.

[0070] In order to make the purpose, technical solutions and advantages of the embodiments of the present disclosure clearer, the technical solutions in the embodiments of the present disclosure will be described clearly and completely in the following with reference to the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, rather than all the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present disclosure.

[0071] The present disclosure provides a point cloud enhancement method based on polarization structure perception and trend guiding recovery, as shown in Figure 1 The method comprises the following steps:

[0072] S10, acquiring a plurality of angle polarization images, calculating a polarization degree and a polarization angle according to the polarization images, and generating an object region mask based on a polarization angle consistency mask, a polarization angle gradient mask and a linear polarization degree mask;

[0073] S20. Obtain the original phase map through four-step phase shift and complementary Gray code, and then obtain the missing area inside the object according to the hole distribution of the original phase map. Add morphological structure indicators and local variance features to identify the real self-shadowing area, exclude the structural holes of the object itself, and construct the self-shadowing mask.

[0074] S30. For the identified real self-shadow region, establish a structure-guided recovery model that combines the first-order structure gradient guidance term and the second-order structure gradient guidance term. Through the edge modulation weight constraint of the structure-guided recovery model, achieve continuous and smooth phase completion to obtain the complete object point cloud.

[0075] S40. Separate the background and invalid shadows in the object point cloud using the object region mask to obtain a point cloud that retains only the object region.

[0076] Through the above embodiments, this disclosure can enhance the accuracy of object edge recognition by using a linear polarization degree threshold and leveraging polarization angle consistency, polarization angle gradient, and edge feature information, thereby extracting the object under test and removing invalid points from the background and invalid shadows. The following is an exemplary description of each of the above steps:

[0077] S10. Obtain polarization images from multiple angles, calculate the degree of polarization and polarization angle based on the polarization images, and generate an object region mask based on the polarization angle consistency mask, polarization angle gradient mask and linear polarization degree mask.

[0078] In this embodiment, a polarization camera is used to acquire polarized images from multiple angles. Because there is an angle between the direction projected by the projector and the direction acquired by the polarization camera, there is a height difference between the object being measured and the background, thus forming a shadow area. The remaining area, excluding the surface area of ​​the object being measured and the shadow area, is called the background area. Since neither the background nor the shadow area contains surface information of the object being measured, the 3D point cloud derived from the phase values ​​of these areas is invalid. Therefore, the background and shadow areas can be removed by analyzing the distribution characteristics of the polarization angle and degree of polarization to obtain an object area mask. According to Malus's law, after polarized light passes through a polarizer, the intensity of the transmitted light varies with the direction of the polarizer. Changes:

[0079] ,

[0080] In the formula: The initial intensity of fully polarized light. It is the polarization angle (AoP). The direction is that of the polarizer. However, in actual natural scenes, light is not completely polarized, but partially polarized. This can be represented as:

[0081] ,

[0082] in, The intensity of unpolarized light will not change with the angle of the linear polarizer. The intensity of polarized light varies with the angle of the linear polarizer. According to the trigonometric identities:

[0083] ,

[0084] The final polarization response model of the measured light intensity is obtained by refining the model:

[0085] ,

[0086] In the formula, Total light intensity The degree of linear polarization (DoLP) represents the proportion of polarized light in the total light.

[0087] By using a polarization camera, the intensity can be collected in four directions: 0°, 45°, 90°, and 135°, respectively: , , and .

[0088] Based on the simplified linear polarization model, the Stokes parameters are defined as follows:

[0089] ,

[0090] ,

[0091] ,

[0092] The projection of the polarized light vector onto the two-dimensional plane at this point:

[0093] ,

[0094] In this embodiment, the degree of polarization and polarization angle of the polarization image calculated based on the Stokes parameters and the Mueller matrix are expressed as follows:

[0095] ,

[0096] in: For all light intensities, This represents the intensity difference of linearly polarized light when the polarizer angle is 0° and 90°. This represents the intensity difference of linearly polarized light when the polarizer angle is 45° and 135°. This represents the intensity of linearly polarized light when the polarizer angle is 0°. This represents the intensity of linearly polarized light when the polarizer angle is 90°. This represents the intensity of linearly polarized light when the polarizer angle is 45°. This represents the intensity of linearly polarized light when the polarizer angle is 135°. In polarization imaging systems, a polarization camera can acquire two key polarization parameters: the degree of linear polarization (DoLP) and the angle of polarization (AoP). These parameters are related to the material properties, roughness, geometry, and illumination conditions of the object's surface. In structured light 3D measurement, since the background and shadow regions do not possess the true reflective structure of the object's surface, their polarization characteristics exhibit significant statistical differences.

[0097] The following discrimination strategy can be used to construct a linear polarization degree mask, expressed as:

[0098] ,in, The set linear polarization threshold is determined by the Otsu automatic thresholding algorithm, eliminating the need for manual setting. However, relying solely on DoLP carries some risks: high DoLP regions may exist in non-object areas; polarization mixing effects may occur at object boundaries or in complex geometric regions, leading to falsely low DoLP values ​​and incorrect rejection.

[0099] The polarization angle represents the dominant polarization direction of light. On the surface of an object, due to the uniformity of the normal and the continuity of reflection patterns, the polarization angles of adjacent pixels tend to be consistent, forming a continuous and smooth structural field. However, in shadows, edges, and backgrounds, the incident / reflected directions of light are mixed or even missing, resulting in chaotic and disordered polarization angle directions, appearing as random jumps in the image. To quantitatively characterize this directional consistency and improve the stability of discrimination, we introduce "polarization angle consistency" as a supplementary structural constraint.

[0100] The definition is as follows:

[0101] Let the polarization angle of the pixel Convert to unit vector form:

[0102] ,

[0103] use Because of the polarization angle It is a modulo π angle, and its direction is undirected (180° = 0°). Physically... and Representing the same physical direction, and mapping it to This yields a unique corresponding directed vector (with a period of 2π), avoiding angle jumps.

[0104] The polarization angle consistency is expressed as:

[0105] ,

[0106] Let the magnitude of the sum of unit vectors be _____. It is the sum of the magnitudes of unit vectors. .

[0107] If the directions are completely consistent, the vectors are superimposed and enhanced, the magnitude is close to the size of the neighborhood, and the denominator is the sum of the magnitudes of the unit vectors, which theoretically equals the number of neighborhood pixels N. Therefore The larger the value, the more consistent the orientation of the domain. On the surface of the object being measured: the orientation of the domain is basically consistent, all Pointing to the approximate direction, the magnitudes accumulate, and the consistency is close to 1; in the shadow and background: the direction of the neighborhood is messy, the vector directions cancel each other out, the vectors and magnitudes become smaller, and the consistency tends to 0; while at the edge of the neighborhood, the direction of the neighborhood changes abruptly, the direction vectors are scattered, and the consistency is moderate or low.

[0108] Although polarization angle consistency can be used to determine whether a region is "continuous", in boundary or occluded areas, the reflection direction changes suddenly, often manifested as a sharp jump in polarization angle. In this case, even if the consistency decreases, some directionality may still be retained. Therefore, in order to prevent the erroneous deletion of the edge point cloud of the measured object, polarization angle gradient detection is introduced to identify the edge, crop the misidentified background or occlusion, and make up for the "fuzzy boundary" problem of consistency methods.

[0109] Because the polarization angle is a periodic function, it can be used directly. Calculating gradients by subtraction can lead to periodic jumps. For example: , The direct subtraction results in 178°, but in reality, it's only a 2° directional difference. To address the issues of periodicity and abrupt changes, we first convert the polarization angle into a vector field:

[0110] ,

[0111] To each and Perform gradient calculation: This is the magnitude of the field gradient in the polarization angle direction, and the gradient intensity is expressed as:

[0112] ,in, The horizontal rate of change of polarization direction, The rate of vertical change of polarization direction, Horizontal rotation gradient in polarization direction Vertical rotation gradient in polarization direction.

[0113] The positions with larger values ​​indicate abrupt changes in polarization direction. The gradient intensity is obtained. Then, edge detection can be performed using standard image processing methods. The polarization angle gradient mask is defined as:

[0114] ,

[0115] The threshold is determined by histogram valley analysis. By statistically analyzing the gray-level histogram of polarization angle gradients, the first local minimum after the main peak is found as the boundary point, and this point is used as the threshold. In this way, pixels with low polarization degree and poor orientation consistency that are mistakenly removed as boundary pixels are recovered through edge detection, and closed boundaries are formed, which is beneficial for subsequent processing.

[0116] In one embodiment, generating an object region mask based on a polarization angle consistency mask, a polarization angle gradient mask, and a linear polarization degree mask includes:

[0117] The object region mask is represented as follows:

[0118] ，

[0119] In the formula, For linear polarization degree mask, For the deflection angle consistency mask, This is a polarization angle gradient mask;

[0120] The deflection angle consistency mask is represented as follows:

[0121] ,

[0122] The threshold value is automatically determined using the Otsu method.

[0123] The polarization angle gradient mask is represented as follows:

[0124] ,

[0125] The threshold value is determined by histogram valley analysis.

[0126] The linear polarization degree mask is represented as:

[0127] ,in, The set linear polarization threshold is determined by the Otsu automatic threshold segmentation algorithm.

[0128] Because the three metrics mentioned above output completely different mask information, weighted fusion cannot reflect the logical relationship. Using weighted averaging would introduce fuzzy boundaries. Therefore, a logical AND operation is ultimately performed using DoLP, polarization angle consistency, and polarization angle gradient to ensure that the preserved region has good reflectivity, consistency, and boundary structure stability, thus achieving reliable object region extraction.

[0129] S20. Obtain the original phase map through four-step phase shift and complementary Gray code, and then obtain the missing area inside the object according to the hole distribution of the original phase map. Add morphological structure indicators and local variance features to identify the real self-shadowing area, exclude the structural holes of the object itself, and construct the self-shadowing mask.

[0130] In this embodiment, due to abrupt changes, occlusion, and depressions in the geometry of the object, some areas may be blocked from projected light, resulting in localized dark areas, also known as self-shadow areas. Figure 2 As shown in the diagram, the arrow points to the self-shadowed area. The projected light from the projector is blocked by the protruding parts of the object, causing some areas of the object's surface to remain unilluminated, forming shadowed areas. Furthermore, due to the angle between the camera and the projector, their fields of view are misaligned, resulting in different shadowed areas R1 and R2 in the diagram. R1, covering the surface of the object being measured, is a self-shadowed area, while R2 is an invalid shadowed area. Using the method described above, the self-shadowed areas, such as R1, on the object's surface need to be removed.

[0131] Although self-shadow regions exist on the object's surface, they are not illuminated by projected light, thus failing to yield effective phase values ​​in structured light decoding and appearing as valueless regions in the phase map. These regions differ from the background or external regions at image boundaries; their spatial structure is always enveloped by the complete object phase, presenting as "voids" nested within the object. However, the phase information of structural voids within the object itself is similar to that of self-shadow regions. To accurately distinguish between them, a self-shadow void identification method based on the phase map structure is proposed. This method identifies phase voids enveloped by effective body regions and filters them using morphological features (Solidity) and texture features, achieving highly robust automatic extraction of self-shadow regions.

[0132] Let the three-dimensional phase diagram be Its size is That is, the height and width of the image. Since the decoding failure area has no value in the image, the first step is to convert the phase map into a valid binary mask. The definition is as follows:

[0133] In an effective binary mask: pixels with a value of 1 represent regions successfully decoded from the phase map, forming the complete object structure. Pixels with a value of 0 represent regions that were not decoded, which may be background, shadows, or noise. Accordingly, a phase hole mask is constructed. :

[0134] ,

[0135] The phase hole mask This indicates the location of all void regions in the phase map, including void regions of object structure, self-shadowed regions, background regions, shadowed regions, and noise.

[0136] The next step is to... Extract the true "self-shadow candidate regions" and perform connectivity analysis on them. All connected components form a set:

[0137] Each It is a binary mask representing the hole region. Then, for each hole region... Extract its boundary contour Then, it is determined whether the area is completely enclosed by the measured object. The judgment criteria are as follows:

[0138] ,

[0139] If this condition is met, it means that the edge of the hole is completely nested within the effective area of ​​the object, thus becoming a candidate self-shadow region. The set of all holes that meet the condition is denoted as:

[0140] ,

[0141] The obtained candidate self-shadow regions also include those with genuine penetrating holes or structural defects in the object itself, self-shadow regions, and false closed holes caused by image noise or boundary errors. To further improve the accuracy of shadow recognition, this paper introduces two types of region features based on structural encapsulation: structural indicators: the overall contour saturation (Solidity) of the region; and texture statistics indicators: the local intensity variance within the region.

[0142] Solidity is a morphological parameter that measures the "compactness" of a region's shape, with values ​​ranging from [value range missing]. , represented as:

[0143] In the formula, For the region The actual pixel area, Let it be the area of ​​its corresponding minimum convex hull region;

[0144] When a region has a smooth, abrupt, and relatively rounded shape, its Solidity value is close to 1. When a region has sharp corners, depressions, structural seams, or irregular edges, its Solidity decreases significantly. Regions in shadow, due to the obstruction of light, are smoothly enveloped by their edges, exhibiting a natural, diffused appearance, and thus have a higher Solidity. Conversely, structural seams or perforations are sharp and elongated, with a convex area much larger than their actual area, resulting in a lower Solidity.

[0145] Therefore, candidate regions that meet the following conditions are retained:

[0146] ,

[0147] in The values ​​are set based on experience and adjusted according to the scene. In addition to morphological features, the shadow area also has image characteristics such as weak texture features, low reflection information, and near-uniformity inside.

[0148] Define region Internal pixels in the image Local variance for:

[0149] , For the input image, For the region Inner pixel mean, This represents the number of pixels in the region.

[0150] This variance reflects the fluctuation in pixel intensity within a region. Therefore, regions with extremely low variance (such as those with shadows) are retained, while regions with structured textures have higher variance and are discarded. The discrimination rule is:

[0151] ,

[0152] Typical values The grayscale range of the unit view image is used as a guide. Combining the two filtering conditions above, the final set of shadow regions is constructed:

[0153] ,

[0154] Therefore, the constructed self-shadow mask is represented as:

[0155] ,

[0156] in, The shaded area is represented as:

[0157] ,in, Set values ​​for experience. For typical values, These are morphological parameters, with a value range of [value range missing]. , represented as:

[0158] In the formula, For the region The actual pixel area, Let it be the area of ​​its corresponding minimum convex hull region;

[0159] For local variance, it is expressed as:

[0160] , For the input image, For the region Inner pixel mean, This represents the number of pixels in the region.

[0161] The area it covers is a phase map. A subset of the hollow regions in, all Pixels with invalid phase values ​​are identified as the target area requiring restoration. A self-shadowing mask is used to determine the location for point cloud restoration. Restoring the point cloud using a self-shadowing mask prevents the addition of erroneous point clouds in other areas. After restoration, an object region mask is used to remove point clouds outside the object area. This method accurately identifies self-shadowing regions and distinguishes between structural holes and self-shadowing regions, improving upon the difficulty of differentiating between structural holes and self-shadowing regions in the original method.

[0162] S30. For the identified real self-shadow region, establish a structure-guided recovery model that combines the first-order structure gradient guidance term and the second-order structure gradient guidance term. Through the edge modulation weight constraint of the structure-guided recovery model, achieve continuous and smooth phase completion to obtain the complete object point cloud.

[0163] In this embodiment, the self-shadowed region that needs to be restored was obtained. The area is empty in the original phase map. To obtain a continuous and complete phase map, the area in the shaded region needs to be processed. The phase value is recovered.

[0164] Let the original phase diagram be Define the image domain as The effective phase region is The self-shadowed area to be restored is ,satisfy: ,

[0165] The recovery target can be described as: in the self-shadowed area In, based on the effective phase region Based on the provided phase boundary and structural information, a reasonable and structurally consistent recovery model is constructed to complete the missing phase values.

[0166] Although the self-shaded region lacks phase information, it is still spatially part of the object. Therefore, the phase change should satisfy the continuity of the object's surface, and the phase surface should remain as smooth as possible in the self-shaded region. The classic idea is to solve for the missing region by minimizing the phase curvature energy, i.e., constructing the following optimization objective: This energy functional corresponds to the total bending energy of the surface. Minimizing this function means restoring the surface to its smoothest variation in the missing region. Taking the variational extremum of this function yields the Euler-Lagrange equation, the result of which is:

[0167] ,

[0168] In the formula: It is a two-dimensional Laplace operator, that is:

[0169] ,

[0170] This result indicates that the self-shaded region should satisfy the zero Laplace constraint, i.e., the phase curvature within the region should be zero, to obtain the smoothest surface solution. Furthermore, the recovered region must satisfy the boundary continuity condition, i.e., in... boundary Above, its recovered value should be equal to the known boundary phase value in the phase diagram:

[0171] ,

[0172] In the formula: It comes from the effective phase region The original phase value. This boundary condition ensures a continuous transition between the restored result and the effective region, avoiding faults or jumps.

[0173] However, real-world object surfaces often exhibit significant curvature trends, slope directions, or global topographic features in occluded areas. Reconstruction methods that focus solely on smoothness frequently result in structural "tiling" or loss of hierarchy. Therefore, incorporating structural trend information to enhance the reconstruction process is necessary. In the enhancement model, the right-hand side of the Laplace equation is expanded to:

[0174] ,

[0175] The right end The structure-guided term consists of two parts: the first-order structure gradient-guided term. It is obtained by extrapolating the phase gradient within the non-shaded region; the secondary structure trend term. The model is constructed from a quadratic surface fitted to the boundary region. This model simultaneously preserves a balance between smoothness, structural trend, and boundary consistency.

[0176] In one embodiment, for the identified true self-shadowing regions, a structure-guided recovery model combining first-order and second-order structure gradient guidance terms is established, including:

[0177] A Laplace model is constructed using first-order and second-order structure gradient guided terms. Continuous and smooth recovery of missing point clouds is achieved through edge-modulated weight constraints, as shown below:

[0178] ,

[0179] In the formula: , is the weighting factor for the first-order structure gradient guiding term. The weighting factor for the second-order structure gradient guiding term. For the first-order structure gradient guiding term and For the second-order structure gradient guiding term, where, The parameters are obtained by constructing a matrix equation based on the coordinates of the fitted points and their phase values ​​in the actual self-shadow region. , , , , and The value; ,in, , For from the effective phase region The original phase value, This represents the true self-shadow region; and In the effective phase region exist and The first-order partial derivative in the direction is expressed as:

[0180] .

[0181] in, This represents a weighting factor indicating the strength of the structural gradient guidance. A larger value emphasizes directional consistency, while a smaller value results in a smoother approximation. In addition to local gradient guidance, to further introduce an "overall morphological trend" and recover the generalized structural trend within the region, it can be obtained by fitting effective boundary points.

[0182] A quadratic surface of the form above can be obtained by collecting the coordinates of all fitted points. and its phase value The matrix equation is constructed to calculate:

[0183] ,

[0184] The parameters are obtained by solving the overdetermined system of equations using the least squares method:

[0185] ,

[0186] This quadratic trend surface is introduced within the recovery region as a macroscopic structural guideline:

[0187] ,

[0188] This is a weighting factor for the second-order structural gradient guiding term. It is positive when the fitting quality is high, and zero otherwise to turn off the trend term. This term can significantly enhance the structural hierarchy of the restored results, especially showing better restored morphology in large self-shaded areas.

[0189] To prevent structural trend terms from being mispropagated at image boundaries or object cross-sections, an edge modulation weighting function is introduced:

[0190] ,

[0191] This structure ensures that trend information can only propagate effectively within continuous regions, thus preserving the boundary features of the object's structure. In practical numerical implementation, the above model is discretized. For an image size of... A linear index mapping is established for the region, forming a sparse linear system:

[0192] ,

[0193] For any point The corresponding single-line expression in a sparse system is:

[0194] ,

[0195] in Represents the four neighborhoods of pixel p, with weights From the edge modulation function, The resultant force at the right end of the structural trend term is used; after constructing the above linear system for all recovery points, the phase value of the recovery region can be obtained through a sparse linear solver, thereby realizing the recovery of the point cloud of the object under test from the shadow region.

[0196] S40. Separate the background and invalid shadows in the object point cloud using the object region mask to obtain a point cloud that retains only the object region.

[0197] In this embodiment of the disclosure, the missing point cloud in the shadow region is restored by object region masking, global quadratic surface fitting and local gradient extrapolation, while erroneous point clouds in the background and invalid shadow areas are removed.

[0198] In addition, this embodiment of the present disclosure also constructs a structured light 3D reconstruction system, which mainly consists of an LCD projector (CB-FH52) and a polarized monochrome CMOS camera (FLIR BFS-U3-51S5P-C). The LCD projector has a resolution of 1920×1200 pixels, and the polarized camera has a resolution of 2448×2048 pixels. The measurement system is located approximately 0.7 meters in front of the object to be measured.

[0199] To evaluate the performance of the three-dimensional point cloud integrity enhancement method based on polarization structure perception and trend-guided recovery, this embodiment uses the Otsu method and K-means clustering as control groups for invalid point removal, and linear interpolation as control group for missing point cloud recovery. Comparative experiments were conducted on metal and plaster molds.

[0200] The samples and their phase maps used for invalid point removal are as follows: Figure 3 As shown, for Figure 3 In the test, object a has numerous burrs, increasing the difficulty of edge segmentation. Object b exhibits depth variations and shadows, validating the versatility of the proposed method in situations with depth variations and shadows. Object d, made of plaster, is distinguished from the metallic materials of the previous three objects, verifying the applicability of the proposed method to different materials. The Otsu method utilizes the modulation distribution characteristics of the fringe pattern to adaptively select a threshold to distinguish objects from the background. The K-means method uses modulation, phase consistency, and effective point detection based on neighborhood features to eliminate invalid measurement points. However, because the modulation of the background and the modulation at the junction of the object edge and the background are similar to that of the object, these areas are misclassified as object regions. Different separation effects are shown below. Figure 4 As shown, the first column from the left is the original image, the second column is the image processed by the Ostu method, the third column is the image processed by the K-means method, and the fourth column is the image processed using our method. The red arrows indicate areas that were incorrectly preserved. Because the tone of the background and the tone at the edges of the object where they connect to the background are similar to the object's tone, these areas were mistakenly classified as object regions. Figure 6 In the (d) plaster model, the tonal difference between the plaster model and the black background is not significant enough, resulting in large areas being misidentified as object regions. Point cloud images of the four test samples using different methods are shown below. Figure 5 As shown, it can be seen that Figure 5In the point cloud images obtained by the Otsu and K-mean methods, there are still invalid point clouds that cannot be removed. To quantitatively evaluate these separation methods, we introduce MIoU and ME to reflect the segmentation performance of the image. The closer the MIoU is to 1, the higher the prediction accuracy of the foreground target. ME reflects the percentage of foreground pixels that are misassigned to the foreground; the closer it is to 0, the fewer the misassignments. For metallic objects, the MIoU of the Otsu and K-mean methods is approximately 0.9360 and 0.9376, respectively, while the MIoU of our method is approximately 0.9887. For plaster objects, the MIoU of the Otsu and K-mean methods is 0.5196 and 0.5224, respectively, while the MIoU of our method is 0.9916. The obtained error results show that our invalid point cloud removal method has a significant improvement over the Otsu and K-mean methods, further demonstrating that our proposed method for segmenting the target region based on polarization characteristics has accuracy and robustness.

[0201] Regarding self-shadow recognition, to test the accuracy of the proposed method, we selected three samples for testing: one object with structural holes and three objects with self-occluding shadows. For example... Figure 6 As shown, our self-shadow recognition method can effectively distinguish between structural holes and self-shadow regions in objects. After identifying the self-shadow regions, we selected three objects with self-shadows as test samples and used linear interpolation as a control group for missing point cloud restoration. The results of missing point cloud restoration using linear interpolation and our method are shown in the figure. Figure 7 As shown, by observing the magnified point cloud image, it can be seen that although the linear interpolation method can also recover missing point clouds, the difference between the point cloud and the structure of the object in that region is significant, making it impractical. Our method, however, recovers point clouds that more closely resemble the object's original structure. Figure 8 As shown, to further quantify the effectiveness of point cloud restoration, we cut out a hole in a complete point cloud without shadows, then restored it using the three methods compared. The restored point cloud was then compared with the uncut point cloud to obtain the error of the point cloud restoration. Figure 9The image shows the point cloud images restored by the linear interpolation method and our method. Error analysis was performed between the point clouds restored by the linear interpolation method and our method and the original, uncut point cloud. The mean absolute error (MAE) and root mean square error (MSE) of the linear interpolation method were smaller than those of the linear interpolation method and the curvature distribution difference (CDD) (0.0547, 0.0641, and 1.777, respectively). In contrast, the mean absolute error (MAE) and root mean square error (MSE) of our method were smaller than those of the linear interpolation method and the CDD (0.0199, 0.0249, and 29.25, respectively). This indicates that our method's MAE and MSE are significantly lower than those of the linear interpolation method and the CDD, and much higher than those of the linear interpolation method. Therefore, the point cloud restored by our proposed method is closer to the shape of the object itself, resulting in better restoration. Figure 10 As shown, our method successfully repaired the valid point cloud and then removed the redundant invalid points.

[0202] The method proposed in this disclosure, when faced with complex structures and geometric occlusion, especially in areas with complex structures, enables effective point cloud restoration of shadowed regions while maintaining accuracy. By separating the background from invalid shadows, a complete and clean point cloud image is obtained. Compared to traditional methods, our effective shadow point cloud restoration is more complete and accurate, the separation of the tested object from the background and invalid shadows is more precise, invalid points are removed more comprehensively, and it is applicable to various materials of the tested object. Ultimately, a high-quality image is obtained, improving the completeness and accuracy of the reconstructed point cloud of the tested object.

[0203] As another aspect of the embodiments of this disclosure, a point cloud enhancement system 100 based on polarization structure perception and trend-guided recovery is also provided, such as... Figure 11 As shown, it includes:

[0204] The object region mask acquisition module 1 acquires polarization images from multiple angles, calculates the degree of polarization and polarization angle based on the polarization images, and generates an object region mask based on the polarization angle consistency mask, polarization angle gradient mask and linear polarization degree mask.

[0205] Self-shadowing mask acquisition module 2 obtains the original phase map through four-step phase shift and complementary Gray code, and then obtains the missing area inside the object according to the hole distribution of the original phase map. Morphological structure indicators and local variance features are added to identify the real self-shadowing area, exclude the structural holes of the object itself, and construct the self-shadowing mask.

[0206] The self-shadow region missing point cloud recovery module 3 establishes a structure-guided recovery model that combines a first-order structure gradient guide term and a second-order structure gradient guide term for the identified real self-shadow region. The edge modulation weight constraint of the structure-guided recovery model is used to achieve continuous and smooth phase completion, so as to obtain a complete object point cloud.

[0207] The 3D reconstruction module 4 separates the background and invalid shadows in the object point cloud using the object region mask to obtain a point cloud that retains only the object region.

[0208] The point cloud enhancement method proposed in this disclosure for invalid point removal and missing point cloud recovery addresses the issue that invalid points exist in shadow and background areas of the acquired stripe pattern due to limitations of optical imaging structures such as projectors and cameras, as well as the influence of imaging noise. By using a linear polarization degree threshold and leveraging polarization angle consistency, polarization angle gradient, and edge feature information to enhance the accuracy of object edge recognition, the method extracts the measured object and removes invalid points from the background and invalid shadows.

[0209] This disclosure also proposes a three-dimensional point cloud integrity enhancement strategy based on polarization structure perception and trend-guided recovery. By identifying the true self-shadow region according to the morphological structure index (Solidity) and local variance features, the strategy aims to eliminate the void regions in the object's own structure, accurately determine the self-shadow region, and construct a self-shadow mask.

[0210] This disclosure proposes a method for restoring missing point clouds guided by global and local structural trends, which realizes the restoration of point clouds in the shadow region of the measured object. Compared with the classic linear interpolation method, it improves the restoration accuracy and makes the object point cloud more complete and the structural hierarchy more natural.

[0211] Without causing contradictions, the above-described modules in the system of the present disclosure embodiments can implement any of the above-described methods.

[0212] This disclosure also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the point cloud enhancement method based on polarization structure perception and trend-guided recovery described in the above embodiments. The electronic device can be provided as a terminal, a server, or other types of device.

[0213] This disclosure also provides a computer-readable storage medium storing a computer program thereon. When executed by a processor, the program implements the point cloud enhancement method based on polarization structure perception and trend-guided recovery described in the above embodiments. The computer-readable storage medium may be a non-volatile computer-readable storage medium.

[0214] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this disclosure. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of those different embodiments or examples.

[0215] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this disclosure, "a plurality of" means two or more, unless otherwise explicitly specified.

[0216] The above are merely specific embodiments of this disclosure, but the scope of protection of this disclosure is not limited thereto. Any person skilled in the art can easily conceive of various variations or substitutions within the technical scope disclosed in this disclosure, and these should all be included within the scope of protection of this disclosure. Therefore, the scope of protection of this disclosure should be determined by the scope of the claims.

Claims

1. A point cloud enhancement method based on polarization structure sensing and trend-guided recovery, characterized in that, Includes the following steps: Acquire polarization images from multiple angles, calculate the degree of polarization and polarization angle based on the polarization images, and generate an object region mask based on a polarization angle consistency mask, a polarization angle gradient mask, and a linear polarization degree mask; wherein, the object region mask is represented as: ， In the formula, For linear polarization degree mask, For the deflection angle consistency mask, This is a polarization angle gradient mask; The deflection angle consistency mask is represented as follows: ， The set threshold is automatically determined using the Otsu method. For consistency of polarization angle; The polarization angle gradient mask is represented as follows: ， The threshold value is determined by histogram valley analysis. For gradient strength; The linear polarization degree mask is represented as: ,in, The set linear polarization degree threshold is determined by the Otsu automatic threshold segmentation algorithm; The original phase map is obtained by four-step phase shift and complementary Gray code. Then, the missing regions inside the object are obtained according to the hole distribution of the original phase map. Morphological structural indicators and local variance features are added to identify the real self-shadowing regions, eliminate the structural holes of the object itself, and construct the self-shadowing mask. For the identified real self-shadow regions, a structure-guided recovery model combining first-order and second-order structure gradient guidance terms is established. Continuous and smooth phase completion is achieved through edge modulation weight constraints of the structure-guided recovery model, resulting in a complete object point cloud. Specifically, for the identified true self-shadowing regions, a structure-guided recovery model combining first-order and second-order structure gradient guidance terms is established, including: A Laplace model is constructed using first-order and second-order structure gradient guided terms. Continuous and smooth recovery of missing point clouds is achieved through edge-modulated weight constraints, as shown below: ， In the formula: , is the weighting factor for the first-order structure gradient guiding term. The weighting factor for the second-order structure gradient guiding term. For the first-order structure gradient guiding term and For the second-order structure gradient guiding term, where, The parameters are obtained by constructing a matrix equation based on the coordinates of the fitted points and their phase values ​​in the actual self-shadow region. , , , , and The value; ,in, , For from the effective phase region The original phase value, This represents the true self-shadow region; and In the effective phase region exist and The first-order partial derivative in the direction is expressed as: ； The background and invalid shadows in the object point cloud are separated by the object region mask to obtain a point cloud that retains only the object region.

2. The point cloud enhancement method as described in claim 1, characterized in that, Calculating the degree of polarization and the polarization angle based on the polarization image includes: The degree of polarization and polarization angle of the polarization image calculated from the Stokes parameters and the Mueller matrix are expressed as follows: ， in: For all light intensities, This represents the intensity difference of linearly polarized light when the polarizer angle is 0° and 90°. This represents the intensity difference of linearly polarized light when the polarizer angle is 45° and 135°. This represents the intensity of linearly polarized light when the polarizer angle is 0°. This represents the intensity of linearly polarized light when the polarizer angle is 90°. This represents the intensity of linearly polarized light when the polarizer angle is 45°. This represents the intensity of linearly polarized light when the polarizer angle is 135°.

3. The point cloud enhancement method as described in claim 1, characterized in that, The polarization angle consistency is expressed as: ， Let the magnitude of the sum of unit vectors be _____. It is the sum of the magnitudes of unit vectors. .

4. The point cloud enhancement method as described in claim 1, characterized in that, The gradient intensity is expressed as: ,in, The horizontal rate of change of polarization direction. The rate of vertical change in polarization direction. Horizontal rotation gradient in polarization direction Vertical rotation gradient in polarization direction.

5. A point cloud augmentation system based on polarization structure sensing and trend-guided recovery, characterized in that, include: The object region mask acquisition module acquires polarization images from multiple angles, calculates the degree of polarization and polarization angle based on the polarization images, and generates an object region mask based on a polarization angle consistency mask, a polarization angle gradient mask, and a linear polarization degree mask; wherein, the object region mask is represented as: ， In the formula, For linear polarization degree mask, For the deflection angle consistency mask, This is a polarization angle gradient mask; The deflection angle consistency mask is represented as follows: ， The set threshold is automatically determined using the Otsu method. For consistency of polarization angle; The polarization angle gradient mask is represented as follows: ， The threshold value is determined by histogram valley analysis. For gradient strength; The linear polarization degree mask is represented as: ,in, The set linear polarization degree threshold is determined by the Otsu automatic threshold segmentation algorithm; The self-shadowing mask acquisition module obtains the original phase map through four-step phase shift and complementary Gray code, then obtains the missing regions inside the object based on the hole distribution of the original phase map, adds morphological structure indicators and local variance features to identify the real self-shadowing regions, excludes the structural holes of the object itself, and constructs the self-shadowing mask. The self-shadow region missing point cloud recovery module establishes a structure-guided recovery model combining first-order and second-order structure gradient guidance terms for the identified true self-shadow regions. Continuous and smooth phase completion is achieved through edge modulation weight constraints of the structure-guided recovery model, resulting in a complete object point cloud. Specifically, the structure-guided recovery model combining first-order and second-order structure gradient guidance terms for the identified true self-shadow regions includes: A Laplace model is constructed using first-order and second-order structure gradient guided terms. Continuous and smooth recovery of missing point clouds is achieved through edge-modulated weight constraints, as shown below: ， In the formula: , is the weighting factor for the first-order structure gradient guiding term. The weighting factor for the second-order structure gradient guiding term. For the first-order structure gradient guiding term and For the second-order structure gradient guiding term, where, The parameters are obtained by constructing a matrix equation based on the coordinates of the fitted points and their phase values ​​in the actual self-shadow region. , , , , and The value; ,in, , For from the effective phase region The original phase value, This represents the true self-shadow region; and In the effective phase region exist and The first-order partial derivative in the direction is expressed as: ； The 3D reconstruction module separates the background and invalid shadows in the object point cloud using the object region mask, so as to obtain a point cloud that retains only the object region.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the point cloud enhancement method according to any one of claims 1 to 4.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the point cloud enhancement method according to any one of claims 1 to 4.

Images