Photometric stereo reconstruction method and system for worn surface topography by fusing full-light source images

Abstract

The present invention discloses a method and system for photometric stereo reconstruction of worn surface morphology by fusing full-light source images. The method estimates the normal of the worn surface from the photometric stereo input based on a deep learning algorithm, and uses full-light source images taken with all light sources turned on to provide an estimate of the prior depth and its uncertainty for surface reconstruction. The surface normal and the prior depth are fused pixel by pixel based on the uncertainty and geometric constraints. The fused features and the prior depth are further used to directly generate a surface depth map through a network, thereby obtaining the three-dimensional morphology of the worn surface. The entire network model is trained by a designed total loss function and a constructed synthetic wear surface data set, and is applied to the reconstruction of actual worn surface morphology. The present invention provides prior depth information for reconstructing the worn surface morphology by the photometric stereo method by introducing full-light source images without the need for additional devices, thereby improving the warping and distortion problem of the reconstructed wear morphology caused by the non-Lambertian effect and improving the three-dimensional reconstruction accuracy.

Description

Technical Field

[0001] The present invention belongs to the technical field of mechanical equipment wear state monitoring, and in particular relates to a method and system for photometric stereo reconstruction of wear surface topography by fusing full light source images. Background Art

[0002] In mechanical equipment, friction pairs of key components such as bearings and gears are prone to damage and failure, which may lead to equipment failure. In order to avoid safety issues and economic losses caused by unexpected failures, the concept of condition-based maintenance and wear status monitoring technology have developed rapidly. Compared with technical channels such as vibration signal analysis, lubricant performance decay monitoring and lubricant film thickness ultrasonic measurement, wear surface analysis technology intuitively characterizes the wear information of friction pairs in a visual way of damage, providing an important basis for equipment wear status identification and wear mechanism analysis. In order to improve the non-destructiveness and accuracy of surface morphology monitoring, this technology field is undergoing a transformation from two-dimensional to three-dimensional, and from offline to in-situ. The photometric stereo vision method has become an ideal technical means for in-situ and three-dimensional measurement of wear surface morphology due to its small equipment size and high sampling resolution, and has been introduced into the field of wear surface analysis technology.

[0003] Photometric stereo methods use the variations in brightness and darkness of the surface under different light sources to calculate the surface normal field and reconstruct the surface topography via gradient integration. However, real-world worn surfaces often contain complex topographical damage areas such as scratches and pits. This leads to non-Lambertian effects such as highlights, cast shadows, and interreflections when capturing photometric stereo image sequences. This results in poor surface normal field calculation accuracy and warping of the reconstructed topography. While deep learning-based photometric stereo algorithms have been used to improve the accuracy of surface normals for worn surfaces, the complex topography of worn surfaces still poses the problem of warping in the reconstructed results. Existing approaches to address this problem can be summarized as incorporating prior information about the surface normal to aid in 3D topography reconstruction and improve the reconstruction results. There are two approaches: one involves incorporating additional devices to determine prior information about the surface topography, such as using an accurate mechanical stylus to perform sparse surface measurements; the other involves leveraging prior estimates of the worn surface topography, such as the pin-disc surface being flat and the bearing inner ring surface being cylindrical. The former combines the morphology prior information and the surface normal information through a cost function or a neural network to calculate the surface morphology, but the additional device increases the volume and complexity of the reconstruction system, which limits its application in in-situ reconstruction of worn surfaces; while the reconstruction effect of the latter is greatly affected by the weight parameters of the prior estimation items, and its weight parameters rely on empirical selection.

[0004] Overall, photometric stereo vision methods provide key technical support for in-situ acquisition of the three-dimensional topography of worn surfaces. However, the non-Lambertian effect of worn surfaces leads to low accuracy in surface normal vector calculations and warping of the reconstructed topography, compromising the accuracy of wear analysis. While existing methods attempt to combine prior topographic information with surface normals to improve the accuracy of worn surface reconstruction, each method has significant drawbacks, making it difficult to adapt to the demands of in-situ reconstruction and automated, rapid analysis of worn surfaces. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to address the deficiencies in the above-mentioned existing technologies and provide a method and system for photometric stereo reconstruction of the wear surface morphology by integrating full-light source images, so as to solve the technical problem of warping and distortion of the reconstructed morphology results when photometric stereo technology is used to reconstruct wear surfaces with non-Lambertian effects.

[0006] The present invention adopts the following technical solutions:

[0007] The method for photometric stereo reconstruction of worn surface topography by fusing full-light source images includes the following steps:

[0008] S1. Combining the photometric images and illumination directions of the worn surface under different light sources, a wear surface normal prediction branch based on an encoder-decoder network architecture is established by embedding a photometric feature aggregation layer using a multi-head attention mechanism.

[0009] S2. Combining the full-light source image of the worn surface, a wear surface prior depth estimation branch with the ability to quantify the uncertainty of the estimation result is established by adding a dropout layer and a network weight regularization term;

[0010] S3, based on the wear surface normal prediction branch constructed in step S1 and the wear surface prior depth estimation branch constructed in step S2, construct a wear surface depth reconstruction branch consisting of a geometric information fusion layer and a wear surface depth regressor;

[0011] S4. Use the wear surface normal prediction branch, the wear surface prior depth estimation branch, and the wear surface depth reconstruction branch obtained in step S3 to form a wear surface depth reconstruction model, and construct a loss function for each branch respectively. The loss functions of the three branches are fused in the form of a weighted sum as the total loss function, and a synthetic wear surface data set is constructed as training data. The wear surface depth reconstruction model is trained with the minimization of the total loss function as the optimization goal, and the trained wear surface depth reconstruction model is applied to the three-dimensional morphology reconstruction of the actual wear surface.

[0012] Specifically, step S1 is as follows:

[0013] S101, using k different light sources to obtain the stereo information tensor of the worn surface As the input of the wear surface normal prediction branch, H×W represents the photometric image resolution, 6 means that each tensor has 6 channels, which are the RGB channels of the photometric image and the normalized light source direction [x, y, z], respectively. A shared weight encoder consisting of 1 feature embedding layer and multiple hybrid blocks is constructed to extract the wear surface photometric feature set.

[0014] S102, based on the multi-head attention mechanism, construct the photometric feature aggregation layer in the normal prediction branch, using a learnable seed vector As the query vector in the multi-head attention mechanism, the wear surface photometric feature set {Ψ1,Ψ2,...,Ψ k} as the key vector and value vector in the multi-head attention mechanism to obtain the aggregated wear surface photometric features

[0015] S103, using 5 convolutional layers and 1 L2 normalization layer to establish the normal decoder in the normal prediction branch to wear the surface photometric features Solve for the wear surface normal vector field as input Each convolutional layer consists of a 3×3 convolution kernel and a LeakyReLU activation function, and the first four convolutional layers use 2x upsampling.

[0016] Furthermore, in step S101, the shared weight encoder is constructed as follows:

[0017] S1011, the feature embedding layer is set to a convolutional layer with a kernel size of w×w and a stride of w, and position encoding is added to convert the information input into a specific feature size for further feature extraction. The tensor size of the feature embedding layer output is (H / w)×(W / w)×384;

[0018] S1012, the hybrid block consists of a 3×3 depthwise convolution, layer normalization and MLP in series, and residual connections are added after the depthwise convolution and MLP respectively. The input and output of the hybrid block keep the feature size unchanged.

[0019] Specifically, step S2 is as follows:

[0020] S201, the full light source image of the worn surface Connect along the third dimension channel to form the full light source information tensor To keep the spatial size consistent with the photometric stereo information tensor O;

[0021] S202, the wear surface prior depth estimation branch and the wear surface normal prediction branch share a shared weight encoder to extract the full light source features from the full light source information tensor M0′

[0022] S203, using the normal decoder structure without the L2 normalization layer to establish the prior depth decoder of the prior depth estimation branch, and adding a dropout layer with a dropout rate of r and a regularization term of the network weight to the first four convolutional layers;

[0023] S204, keep the dropout layer open during network training and use, perform T random forward propagations of the full light source feature Φ in the prior depth decoder based on the Monte Carlo dropout method, and obtain the prior depth of the wear surface and its uncertainty

[0024] Specifically, step S3 is as follows:

[0025] S301, select the s×s neighborhood of the pixel point i on the worn surface as the local neighborhood Ω i , s is 3, 5, 7, pixel i and local neighborhood Ω i In the three-dimensional space transformed from the pixel coordinate system to the camera coordinate system, each point has three-dimensional coordinates (x, y, z);

[0026] S302, using the prior depth of pixel j in the local neighborhood As a benchmark, combined with the surface normal n of the pixel i on the worn surface i Calculate the absolute depth value of pixel i in the camera coordinate system

[0027] S303, uncertainty u corresponding to pixel j j Processing to obtain prior depth In the local neighborhood Ω i Relative credibility in j , the dot product of the surface normals of the neighborhood pixel j and the target pixel i is used as the geometric feasibility g of the two points being in the same plane j , s j and g j Commonly represent depth values As the credibility of the depth of the target pixel i, and s j ·g j As depth value The weight of

[0028] S304, calculating the pixel i on the worn surface relative to the local neighborhood Ω i The geometric information fusion feature q of the pixel i on the wear surface is obtained by using the multiple depth values and corresponding weights of all pixels in the image. i, the normal information of the wear surface pixel i itself and the prior depth information of other pixels in its local neighborhood are embedded into the geometric information fusion feature q i ;

[0029] S305: Repeat steps S301 to S304 in the geometric information fusion layer of the wear surface depth reconstruction branch to traverse all pixels on the wear surface and solve the geometric information fusion features to obtain a geometric information feature map.

[0030]

[0031] S306, using 3 dilated convolution layers with an expansion rate of 2 and 3 normal convolution layers to establish the wear surface depth regressor in the depth reconstruction branch, and adding residual connections after the 3 dilated convolution layers. All convolution layers use the LeakyReLU activation function and the convolution kernel size is 3×3. The geometric information feature map and the prior depth are connected along the third dimension as input to reconstruct the wear surface depth.

[0032] Furthermore, in step S302, the depth value corresponding to pixel i The calculation is as follows:

[0033]

[0034] Among them, n ix 、n iy 、n iz is the surface normal n i The three directional components, u i 、v i is the pixel coordinate of pixel i on the worn surface, u0 and v0 are the pixel coordinates of the principal point of the image, and m is the magnification of the weak perspective projection imaging model used in the photometric stereo technology.

[0035] Furthermore, in step S304, the geometric information fusion feature q of the pixel point i on the worn surface is i The calculation is as follows:

[0036]

[0037] Specifically, step S4 is as follows:

[0038] S401, using the wear surface normal prediction branch constructed in step S1, the wear surface priori depth estimation branch constructed in step S2, and the wear surface depth reconstruction branch constructed in step S3 to form a wear surface depth reconstruction model;

[0039] S402, using mean square error loss to calculate the error between the predicted surface normal and the true surface normal and use it as the loss function of the wear surface normal prediction branch The mean square error loss is used to calculate the error between the prior depth and the true depth, and the L2 regularization term of the prior depth decoder weight layer with a weight decay coefficient is added as the loss function of the wear surface prior depth estimation branch. The uncertainty-weighted mean square error loss is used as the loss function of the wear surface depth reconstruction branch. The sum of the loss functions of the three branches is fused in the form of weighted sum as the total loss function of the wear surface depth reconstruction model

[0040]

[0041] S403: Collect the 3D topography of the worn surface, perform random superpixel segmentation on the 3D topography using the Voronoi function, and use the Principled BSDF in the Blender Cycles renderer to set different reflection characteristics for each superpixel area. Randomly sample the number of light reflections and render a photometric stereo image sequence sample. And full light source image samples At the same time, surface normal maps and surface depth maps are generated as sample labels to construct a synthetic dataset of worn surfaces with strong non-Lambertian effects and spatial variations in reflectivity.

[0042] S404, to minimize the total loss function For the training objective, a synthetic dataset of worn surfaces is used as training data, and the AdamW optimizer is used to train the worn surface depth reconstruction model;

[0043] S405. Apply the trained wear surface depth reconstruction model to reconstruct the actual wear surface morphology to obtain a surface depth map of the actual wear surface. Based on the perspective projection relationship, convert the surface depth map from the pixel coordinate system to the camera coordinate system to obtain the three-dimensional morphology of the wear surface to be reconstructed.

[0044] Furthermore, in step S402, the loss function of the wear surface depth reconstruction branch is for:

[0045]

[0046] Among them, u′ p is the uncertainty of the prior depth U at the pixel point p after Min-Max normalization, is the reconstructed depth of the worn surface at pixel point p, H is the height value of the image resolution, and W is the width value of the image resolution. is the true depth of the worn surface at pixel point p.

[0047] In a second aspect, an embodiment of the present invention provides a photometric stereo reconstruction system for a worn surface topography by fusing full-light source images, characterized by comprising:

[0048] The prediction module combines the photometric images of the worn surface under different light sources and the lighting direction, and establishes a wear surface normal prediction branch based on the encoder-decoder network architecture by embedding a photometric feature aggregation layer using a multi-head attention mechanism;

[0049] The estimation module combines the full-light source image of the worn surface and establishes a wear surface prior depth estimation branch with the ability to quantify the uncertainty of the prediction results by adding a dropout layer and a network weight regularization term;

[0050] The depth module constructs a wear surface depth reconstruction branch consisting of a geometric information fusion layer and a wear surface depth regressor based on the wear surface normal prediction branch constructed by the prediction module and the wear surface prior depth estimation branch constructed by the estimation module;

[0051] The reconstruction module uses the wear surface normal prediction branch, the wear surface prior depth estimation branch, and the wear surface depth reconstruction branch obtained by the depth module to form a wear surface depth reconstruction model. The loss function of each branch is constructed separately, and the loss functions of the three branches are fused in the form of a weighted sum as the total loss function. A synthetic wear surface dataset is constructed as training data, and the wear surface depth reconstruction model is trained with the minimization of the total loss function as the optimization goal. The trained wear surface depth reconstruction model is applied to the three-dimensional morphology reconstruction of the actual wear surface.

[0052] Compared with the prior art, the present invention has at least the following beneficial effects:

[0053] A photometric stereo reconstruction method for worn surface morphology fused with full-light source images estimates the normal of the worn surface from the photometric stereo input based on a deep learning algorithm, and uses full-light source images taken with all light sources turned on to provide an estimate of the prior depth and its uncertainty for surface reconstruction. The surface normal and the prior depth are fused pixel by pixel based on uncertainty and geometric constraints. The fused features and the prior depth are further directly generated into a surface depth map through the network to obtain the three-dimensional morphology of the worn surface. The entire network model is trained by the designed total loss function and the constructed synthetic dataset of worn surfaces, and is applied to the reconstruction of actual worn surface morphology. By introducing full-light source images as the wear surface The reconstruction provides prior depth information, solves the warping distortion problem caused by the photometric stereo method that only relies on the surface normal to reconstruct the morphology, improves the accuracy of the three-dimensional morphology reconstruction of the worn surface, and provides more accurate and effective information for the wear analysis of mechanical equipment; based on the uncertainty of the prior depth and the geometric constraint relationship, the prior depth and the surface normal are reasonably integrated, and the depth of the worn surface morphology is directly output through the network, which reduces the dependence of the network model on the training data and can improve the accuracy and speed of the worn surface reconstruction; the full light source image is used and can be easily obtained without adding any additional equipment, so that the present invention can be applied to the in-situ three-dimensional reconstruction of the wear morphology based on photometric stereo.

[0054] Furthermore, step S1 combines the photometric images and lighting directions of the worn surface under different light sources, and establishes a wear surface normal prediction branch based on the encoder-decoder network architecture by embedding a photometric feature aggregation layer using a multi-head attention mechanism, thereby providing surface normal vector field information for subsequent wear surface depth reconstruction. The photometric feature aggregation layer embedded with the multi-head attention mechanism realizes the aggregation of photometric features by modeling the relationship between multiple photometric features, which helps to improve the prediction accuracy of the wear surface normal.

[0055] Furthermore, the shared weight encoder constructed in step S101 is used to realize the simultaneous extraction of the worn surface photometric stereo information tensor and the worn surface full light source information tensor, which can reduce the number of parameters of the worn surface depth reconstruction model to make the model easier to train.

[0056] Furthermore, step S2 combines the full-light source image of the worn surface and establishes a wear surface prior depth estimation branch with the ability to quantify the uncertainty of the estimation results by adding a dropout layer and a network weight regularization term, providing a priori depth for subsequent wear surface depth reconstruction. The uncertainty of the estimated prior depth is quantified based on the instability of estimating depth information from a single image, which helps to improve the utilization of reliable prior depth estimation in subsequent geometric information fusion.

[0057] Furthermore, step S3 constructs a wear surface depth reconstruction branch consisting of a geometric information fusion layer and a wear surface depth regressor, which uses uncertainty as a guide to effectively fuse the two geometric information, the surface normal information provided by the wear surface normal prediction branch and the prior depth provided by the wear surface prior depth estimation branch, and regress the wear surface depth.

[0058] Furthermore, step S302 fuses the prior depth of pixel j in the local neighborhood in an explicit manner according to the geometric constraint relationship that the connection vector of two points in the same plane in three-dimensional space is orthogonal to the surface normal. and the surface normal n of pixel i i Two geometric information, get the corresponding absolute depth value of pixel i in the camera coordinate system It helps to improve the reconstruction accuracy of the wear surface depth reconstruction model and makes the model easier to train.

[0059] Furthermore, step S304 robustly completes the fusion of the normal information of the wear surface pixel i itself and the prior depth information of other pixels in its local neighborhood based on the relative credibility of the prior depth and geometric feasibility, and obtains the geometric information fusion feature q i , providing a reliable feature input basis for subsequent wear surface depth reconstruction.

[0060] Furthermore, step S4 helps to improve the reconstruction accuracy of the trained wear surface depth reconstruction model when it is applied to the reconstruction of the three-dimensional morphology of the actual wear surface by constructing an effective total loss function and a synthetic wear surface dataset that simulates the non-Lambertian effect of the actual wear surface.

[0061] Furthermore, the loss function of the wear surface depth reconstruction branch in step S402 is Using uncertainty for weighting helps the wear surface depth reconstruction model to improve the reconstruction accuracy in places where the uncertainty of the prior depth estimation is high when finally reconstructing the wear surface depth.

[0062] It can be understood that the beneficial effects of the second aspect mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here.

[0063] In summary, the present invention introduces full-light source images to provide prior depth information for reconstructing the wear surface morphology using the photometric stereo method without the need for additional devices, thereby improving the warping and distortion problem of the reconstructed wear morphology caused by the non-Lambertian effect and improving the three-dimensional reconstruction accuracy.

[0064] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 Flowchart of the present invention;

[0066] Figure 2 This is a schematic diagram of the overall structure of the multi-branch wear surface depth reconstruction model constructed by the present invention;

[0067] Figure 3 This is an example of a synthetic dataset for wear surface reconstruction constructed by S2 in step 4, where (a) is the three-dimensional topography of the wear surface acquired by laser confocal microscopy, (b) is the surface depth map generated by the topography, (c) is the surface normal map generated by the topography, (d) is the full-light image sequence generated by the Blender Cycles renderer, and (e) is the photometric stereo image sequence generated by the Blender Cycles renderer.

[0068] Figure 4 This is a schematic diagram of the application effect of the present invention, wherein the first column is the true value of each worn surface morphology, the second column is the result of reconstruction using only the surface normal, and the third column is the reconstruction result of the method of the present invention. DETAILED DESCRIPTION

[0069] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0070] In the description of the present invention, it is to be understood that the terms “include” and “comprise” indicate the presence of the described features, wholes, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or collections thereof.

[0071] It should also be understood that the terms used in the present specification are only for the purpose of describing particular embodiments and are not intended to limit the present invention. As used in the present specification and the appended claims, the singular forms "a", "an", and "the" are intended to include the plural forms unless the context clearly indicates otherwise.

[0072] It should be further understood that the term "and / or" as used in the present specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items. For example, A and / or B may represent: A alone, A and B simultaneously, or B alone. In addition, the character " / " herein generally indicates that the associated items are in an "or" relationship.

[0073] It should be understood that although the terms "first," "second," and "third" may be used to describe preset ranges in embodiments of the present invention, these preset ranges should not be limited to these terms. These terms are merely used to distinguish one preset range from another. For example, without departing from the scope of embodiments of the present invention, the first preset range may also be referred to as the second preset range, and similarly, the second preset range may also be referred to as the first preset range.

[0074] The word "if," as used herein, may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to the determination" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)," depending on the context.

[0075] The accompanying drawings illustrate various schematic diagrams of structures according to embodiments disclosed herein. These figures are not drawn to scale; for clarity, some details are exaggerated and some details may be omitted. The shapes of the various regions and layers shown in the figures, as well as their relative sizes and positional relationships, are merely exemplary and may deviate in practice due to manufacturing tolerances or technical limitations. Those skilled in the art may design regions / layers with different shapes, sizes, and relative positions as needed.

[0076] The present invention provides a method for photometric stereo reconstruction of worn surface morphology by fusing full-light source images. Taking the photometric image of the worn surface and the illumination direction as the objects, the method integrates the deep learning algorithm and the photometric stereo vision principle to establish a normal prediction branch for the worn surface and solve the normal vector field of the worn surface. Taking the full-light source image as the object, the method constructs a priori depth estimation branch for the worn surface with the ability to quantify the uncertainty of the prediction results, and obtains the prior depth of the worn surface and its uncertainty. On this basis, the surface normal vector field and the prior depth are fused according to the uncertainty and geometric constraints, and a wear surface depth reconstruction branch is established to obtain the precise depth of the worn surface, thereby providing more effective three-dimensional characterization information for wear analysis of mechanical equipment.

[0077] See also Figure 1 The present invention provides a method for photometric stereo reconstruction of worn surface topography by fusing full light source images, comprising the following steps:

[0078] S1. Combining the photometric images and illumination directions of the worn surface under different light sources, a wear surface normal prediction branch based on an encoder-decoder network architecture is established by embedding a photometric feature aggregation layer using a multi-head attention mechanism to solve the normal vector field of the worn surface.

[0079] S101, the wear surface normal prediction branch uses the wear surface photometric stereo information tensor under k different light sources As input, where H×W represents the photometric image resolution, 6 means that each tensor has 6 channels, which are the RGB channels of the photometric image and the normalized light source direction [x, y, z], a shared weight encoder consisting of 1 feature embedding layer and multiple hybrid blocks is constructed to extract the photometric feature set of the worn surface.

[0080] The steps to build a shared weight encoder are as follows:

[0081] S1011, the feature embedding layer is set to a convolutional layer with a kernel size of w×w and a stride of w, and position encoding is added to convert the information input into a specific feature size for further feature extraction. The tensor size output by the feature embedding layer is (H / w)×(W / w)×384. Therefore, the resolution size of the image needs to be a multiple of w. In the present invention, w=16 and H=W=224 are used;

[0082] S1012, the hybrid block consists of 3×3 depth convolution, layer normalization and MLP in series, and residual connections are added after the depth convolution and MLP respectively. The input and output of the hybrid block keep the feature size unchanged. By stacking different numbers of hybrid blocks, the network scale can be easily adapted to the size of the training data set. In the present invention, the number of hybrid blocks stacked is 10.

[0083] S102, based on the multi-head attention mechanism, construct the photometric feature aggregation layer in the normal prediction branch, using a learnable seed vector As the query vector in the multi-head attention mechanism, the wear surface photometric feature set {Ψ1,Ψ2,...,Ψ k} as the key vector and value vector in the multi-head attention mechanism to obtain the aggregated wear surface photometric features

[0084] Compared with the conventional maximum pooling or average pooling photometric feature aggregation method, the photometric feature information can be more fully utilized, which helps to improve the calculation accuracy of the surface normal under sparse input of the worn surface photometric image.

[0085] S103, using 5 convolutional layers and 1 L2 normalization layer to establish the normal decoder in the normal prediction branch to wear the surface photometric features Solve for the wear surface normal vector field as input Each convolutional layer consists of a 3×3 convolution kernel and a LeakyReLU activation function, and the first four convolutional layers use 2x upsampling.

[0086] S2. Combining the full-light source image of the worn surface, a wear surface prior depth estimation branch with the ability to quantify the uncertainty of the prediction results is established by adding a dropout layer and a network weight regularization term to obtain the wear surface prior depth and its uncertainty.

[0087] S201, the full light source image of the worn surface Connect along the third dimension channel to form the full light source information tensor To keep the spatial size consistent with the photometric stereo information tensor O;

[0088] S202, the wear surface prior depth estimation branch and the wear surface normal prediction branch share a shared weight encoder to extract the full light source features from the full light source information tensor M0′

[0089] S203, using the normal decoder structure without the L2 normalization layer to establish the prior depth decoder of the prior depth estimation branch, and adding a dropout layer with a dropout rate of r and a regularization term of the network weight to the first four convolutional layers, so that the prior depth decoder has the ability to quantify the uncertainty of the prediction results;

[0090] S204, keep the dropout layer open during network training and use, perform T random forward propagations of the full light source feature Φ in the prior depth decoder based on the Monte Carlo dropout method, and obtain the prior depth of the wear surface and its uncertainty

[0091] S3. Construct a wear surface depth reconstruction branch consisting of a geometric information fusion layer and a wear surface depth regressor. Guided by the prior depth uncertainty, the wear surface prior depth and surface normal are fused pixel by pixel through geometric constraints to achieve accurate estimation of the wear surface depth.

[0092] S301, select the s×s neighborhood of the pixel point i on the worn surface as its local neighborhood Ω i , s takes the values of 3, 5, and 7, and the pixel i and its local neighborhood Ω i In the three-dimensional space transformed from the pixel coordinate system to the camera coordinate system, each point has three-dimensional coordinates (x, y, z);

[0093] S302, according to the geometric constraint relationship that the connection vector of two points in the same plane in three-dimensional space is orthogonal to the surface normal, the prior depth of pixel point j in the local neighborhood is used. As a benchmark, combined with the surface normal n of the pixel i on the worn surface i Calculate the depth value corresponding to pixel i As shown in formula (1), the depth value is the absolute depth value in the camera coordinate system;

[0094]

[0095] Among them, n ix 、n iy 、n iz is the surface normal n i The three directional components, u i 、v i is the pixel coordinate of pixel i on the worn surface, u0 and v0 are the pixel coordinates of the principal point of the image, and m is the magnification of the weak perspective projection imaging model used in the photometric stereo technology;

[0096] S303, the uncertainty u corresponding to the pixel j j According to formula (2), the prior depth is obtained In the local neighborhood Ω i Relative credibility in j , the dot product of the surface normals of the neighborhood pixel j and the target pixel i is used as the geometric feasibility g of the two points being in the same plane j , depth value Based on prior depth The geometric constraint relationship between the depth and the surface normal is calculated, so s j and g j Together they represent the depth value As the credibility of the depth of the target pixel i, and s j ·g j As depth value The weight of

[0097]

[0098] S304, calculating the pixel i on the worn surface relative to the local neighborhood Ω i The multiple depth values and corresponding weights of all pixels in the wear surface are used to obtain the geometric information fusion feature q of the pixel i on the wear surface using the kernel regression method shown in formula (3). i ,This feature embeds the normal information of the pixel i on the wear surface and the prior depth information of other pixels in its local neighborhood;

[0099]

[0100] S305, the geometric information fusion layer in the wear surface depth reconstruction branch repeats steps S301 to S304 to traverse all pixel points on the wear surface and solve the geometric information fusion features to obtain the geometric information feature map

[0101]

[0102] S306, using 3 dilated convolution layers with an expansion rate of 2 and 3 normal convolution layers to establish the wear surface depth regressor in the depth reconstruction branch, and adding residual connections after the 3 dilated convolution layers. All convolution layers use the LeakyReLU activation function and the convolution kernel size is 3×3. The geometric information feature map and the prior depth are connected along the third dimension as input to reconstruct the wear surface depth.

[0103] The three branches from step S1 to step S3 above together constitute the wear surface depth reconstruction network. The structure of the entire network model is as follows: Figure 2 shown.

[0104] S4. Define the loss functions of the three branches in steps S1 to S3 respectively and construct a total loss function as the training target of the network. Construct a synthetic dataset of the wear surface as training data, train the entire network model, and apply the trained network model to the three-dimensional morphology reconstruction of the actual wear surface.

[0105] S401, using the wear surface normal prediction branch constructed in step S1, the wear surface priori depth estimation branch constructed in step S2, and the wear surface depth reconstruction branch constructed in step S3 to form a wear surface depth reconstruction model;

[0106] S402, using mean square error loss to calculate the error between the predicted surface normal and the true surface normal and use it as the loss function of the wear surface normal prediction branch The mean square error loss is used to calculate the error between the prior depth and the true depth, and the L2 regularization term of the prior depth decoder weight layer with a weight decay coefficient is added as the loss function of the wear surface prior depth estimation branch. The uncertainty-weighted mean square error loss is used as the loss function of the wear surface depth reconstruction branch. The loss functions of the three branches are fused in the form of weighted sum as the total loss function of the wear surface depth reconstruction model

[0107]

[0108] Among them, u′ p The uncertainty U of the prior depth at the pixel point p after Min-Max normalization can guide the model to pay more attention to the uncertainty of the prior depth prediction when reconstructing the wear surface depth. Reconstruct the depth of the worn surface at pixel p.

[0109] Total loss function of the wear surface depth reconstruction model Specifically:

[0110]

[0111] Among them, λ n Loss function for predicting the normal branch of the wear surface The weight coefficient is set to 1.0 in the embodiment of the present invention, λ p Loss function of the wear surface prior depth estimation branch The weight coefficient is set to 0.5 in the embodiment of the present invention, λ r Loss function for reconstructing the branch for the wear surface depth The weight coefficient is set to 1.0 in the embodiment of the present invention.

[0112] S402. A batch of 3D morphologies of worn surfaces were collected using a high-precision laser confocal microscope. The worn surfaces were randomly segmented into superpixels using the Voronoi function. Different reflective properties were set for each superpixel region using the PrincipledBSDF in the Blender Cycles renderer. The number of light reflections was randomly sampled, and a photometric stereo image sequence was rendered. And full light source image samples At the same time, surface normal maps and surface depth maps are generated as sample labels to construct a synthetic dataset of worn surfaces with strong non-Lambertian effects and spatial variations in reflection characteristics. Figure 3 An example of synthetic wear surface data is shown; synthetic wear surface datasets help improve the generalization performance of wear surface depth reconstruction models on actual wear surfaces. Figure 3 An example of synthetic data of a worn surface is shown. It can be seen that the photometric stereo image sequence simulates the non-Lambertian effects of an actual worn surface, namely, highlights and shadows, while the full-light source image has fewer non-Lambertian effects and relatively clear surface morphology and structural information. Therefore, the present invention uses the light and dark changes of the photometric stereo image sequence to calculate the high-frequency surface normal vector field, uses the full-light source image to estimate the low-frequency prior depth information, and combines the surface normal to reconstruct the surface depth, thereby improving the problem of warping and distortion in the photometric stereo reconstruction results of the worn surface under the non-Lambertian effect.

[0113] S403, to minimize the total loss function For the training objective, a synthetic dataset of worn surfaces was used as training data, reasonable hyperparameters were set, an initial learning rate of 0.0005 was used, and the AdamW optimizer was used to train the entire network model for 600 epochs. The total network loss remained basically unchanged, and the network model achieved the optimal effect.

[0114] S404. Apply the trained network model to reconstruct the actual worn surface morphology to obtain a surface depth map of the actual worn surface. Based on the perspective projection relationship, convert the surface depth map from the pixel coordinate system to the camera coordinate system to obtain the three-dimensional morphology of the worn surface to be reconstructed.

[0115] In another embodiment of the present invention, a wear surface morphology photometric stereo reconstruction system that fuses full-light source images is provided. The system can be used to implement the above-mentioned wear surface morphology photometric stereo reconstruction method that fuses full-light source images. Specifically, the wear surface morphology photometric stereo reconstruction system that fuses full-light source images includes a prediction module, an estimation module, a depth module and a reconstruction module.

[0116] The prediction module combines the photometric images and illumination directions of the worn surface under different light sources, and establishes a wear surface normal prediction branch based on an encoder-decoder network architecture by embedding a photometric feature aggregation layer using a multi-head attention mechanism.

[0117] The estimation module combines the full-light source image of the worn surface and establishes a wear surface prior depth estimation branch with the ability to quantify the uncertainty of the prediction results by adding a dropout layer and a network weight regularization term;

[0118] The depth module constructs a wear surface depth reconstruction branch consisting of a geometric information fusion layer and a wear surface depth regressor based on the wear surface normal prediction branch constructed by the prediction module and the wear surface prior depth estimation branch constructed by the estimation module;

[0119] The reconstruction module uses the wear surface normal prediction branch, the wear surface prior depth estimation branch, and the wear surface depth reconstruction branch obtained by the depth module to form a wear surface depth reconstruction model. The loss function of each branch is constructed separately, and the loss functions of the three branches are fused in the form of a weighted sum as the total loss function. A synthetic wear surface dataset is constructed as training data, and the wear surface depth reconstruction model is trained with the minimization of the total loss function as the optimization goal. The trained wear surface depth reconstruction model is applied to the three-dimensional morphology reconstruction of the actual wear surface.

[0120] In another embodiment of the present invention, a terminal device is provided, which includes a processor and a memory, wherein the memory is used to store a computer program, the computer program includes program instructions, and the processor is used to execute the program instructions stored in the computer storage medium. The processor may be a central processing unit (CPU), or may be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, which is suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions to implement the corresponding method flow or corresponding function; the processor described in the embodiment of the present invention can be used for the operation of the wear surface morphology photometric stereo reconstruction method that integrates full-light source images, including:

[0121] Combined with the photometric images and lighting directions of the wear surface under different light sources, a wear surface normal prediction branch based on the encoder-decoder network architecture is established by embedding a photometric feature aggregation layer using a multi-head attention mechanism; combined with the full light source image of the wear surface, a wear surface prior depth estimation branch with the ability to quantify the uncertainty of the estimation results is established by adding a dropout layer and a network weight regularization term; based on the constructed wear surface normal prediction branch and the constructed wear surface prior depth estimation branch, a wear surface depth reconstruction branch consisting of a geometric information fusion layer and a wear surface depth regressor is constructed; the wear surface normal prediction branch, the wear surface prior depth estimation branch and the wear surface depth reconstruction branch are used to form a wear surface depth reconstruction model, and the loss function of each branch is constructed separately. The loss functions of the three branches are fused in the form of a weighted sum as the total loss function. A synthetic wear surface dataset is constructed as training data, and the wear surface depth reconstruction model is trained with the minimization of the total loss function as the optimization goal. The trained wear surface depth reconstruction model is applied to the three-dimensional morphology reconstruction of the actual wear surface.

[0122] In another embodiment of the present invention, the present invention further provides a storage medium, specifically a computer-readable storage medium (Memory), which is a memory device in a terminal device for storing programs and data. It can be understood that the computer-readable storage medium here can include both the built-in storage medium in the terminal device and, of course, the extended storage medium supported by the terminal device. The computer-readable storage medium provides a storage space, which stores the operating system of the terminal. In addition, one or more instructions suitable for being loaded and executed by the processor are also stored in the storage space. These instructions can be one or more computer programs (including program codes). It should be noted that the computer-readable storage medium here can be a high-speed RAM memory or a non-volatile memory (Non-Volatile Memory), such as at least one disk memory.

[0123] The processor may load and execute one or more instructions stored in a computer-readable storage medium to implement the corresponding steps of the method for photometric stereo reconstruction of worn surface topography by fusing full-light source images in the above-mentioned embodiment; the processor may load and execute the following steps:

[0124] Combined with the photometric images and lighting directions of the wear surface under different light sources, a wear surface normal prediction branch based on the encoder-decoder network architecture is established by embedding a photometric feature aggregation layer using a multi-head attention mechanism; combined with the full light source image of the wear surface, a wear surface prior depth estimation branch with the ability to quantify the uncertainty of the estimation results is established by adding a dropout layer and a network weight regularization term; based on the constructed wear surface normal prediction branch and the constructed wear surface prior depth estimation branch, a wear surface depth reconstruction branch consisting of a geometric information fusion layer and a wear surface depth regressor is constructed; the wear surface normal prediction branch, the wear surface prior depth estimation branch and the wear surface depth reconstruction branch are used to form a wear surface depth reconstruction model, and the loss function of each branch is constructed separately. The loss functions of the three branches are fused in the form of a weighted sum as the total loss function. A synthetic wear surface dataset is constructed as training data, and the wear surface depth reconstruction model is trained with the minimization of the total loss function as the optimization goal. The trained wear surface depth reconstruction model is applied to the three-dimensional morphology reconstruction of the actual wear surface.

[0125] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings herein can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0126] See also Figure 4 , showing the comparison of reconstruction results of three actual worn surfaces. The first column is the surface morphology collected by a high-precision laser confocal microscope and used as the true value. The second column is the surface morphology reconstructed by using the least squares method in the field of photometric stereo to calculate the surface normal and combining it with the Frankot-Chellappa normal integral algorithm, that is, the reconstruction is only based on the surface normal calculated by photometric stereo. The third column is the surface morphology reconstructed by the method of the present invention. It can be seen that the complete reconstruction result of the present invention is good, which improves the warping distortion problem of the reconstruction result generated by reconstructing the morphology based only on the surface normal, and can accurately restore the wear features such as scratches and pits on the worn surface. In addition, the present invention does not require additional equipment during the reconstruction process, and directly estimates the depth of the worn surface in an end-to-end manner without relying on parameter settings. It can achieve convenient, fast and accurate in-situ reconstruction of the three-dimensional morphology of the worn surface.

[0127] In summary, the present invention provides a method and system for photometric stereo reconstruction of wear surface morphology by integrating full-light source images. Without the need for additional equipment, the full-light source images are introduced to provide prior depth information for the photometric stereo method to reconstruct the wear surface morphology, thereby improving the warping and distortion problem of the reconstructed wear morphology caused by the non-Lambertian effect and improving the three-dimensional reconstruction accuracy.

[0128] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0129] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0130] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present invention.

[0131] In the embodiments provided by the present invention, it should be understood that the disclosed devices / terminals and methods can be implemented in other ways. For example, the device / terminal embodiments described above are merely illustrative. For example, the division of the modules or units is merely a logical functional division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be through some interface, indirect coupling or communication connection of devices or units, and can be electrical, mechanical, or other forms.

[0132] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0133] In addition, the functional units in the various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0134] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the process in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, it can implement the steps of the above-mentioned various method embodiments. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0135] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0136] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0137] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0138] The above content is only for explaining the technical idea of the present invention and cannot be used to limit the protection scope of the present invention. Any changes made on the basis of the technical solution in accordance with the technical idea proposed by the present invention shall fall within the protection scope of the claims of the present invention.

Claims

1. A photometric stereo reconstruction method for worn surface topography by fusing full-light source images, characterized in that: The following steps are involved: S1. Combining the photometric images and illumination directions of the worn surface under different light sources, a wear surface normal prediction branch based on an encoder-decoder network architecture is established by embedding a photometric feature aggregation layer using a multi-head attention mechanism. S2. Combining the full-light source image of the worn surface, a wear surface prior depth estimation branch with the ability to quantify the uncertainty of the estimation result is established by adding a dropout layer and a network weight regularization term; S3. Based on the wear surface normal prediction branch constructed in step S1 and the wear surface prior depth estimation branch constructed in step S2, a wear surface depth reconstruction branch consisting of a geometric information fusion layer and a wear surface depth regressor is constructed, as follows: Select pixels on the worn surface i of Neighborhood as local neighborhood , s 3, 5, 7, the pixel and local neighborhood In the three-dimensional space transformed from the pixel coordinate system to the camera coordinate system, each point has a three-dimensional coordinate ; Pixels in the local neighborhood j Prior depth As a benchmark, combined with the wear surface pixel points i Surface normal Calculate pixel points i The corresponding absolute depth value in the camera coordinate system ; for pixel points j The corresponding uncertainty Processing to obtain prior depth In the local neighborhood Relative credibility in , using neighborhood pixels j and the target pixel i The dot product of the two surface normals is the geometric feasibility of the two points being in the same plane. , and Commonly represent depth values As the target pixel i Deep credibility and As depth value The weight of Calculate the pixels of the worn surface i Relative to the local neighborhood Multiple depth values and corresponding weights of all pixels in the image are used to obtain the pixel points of the worn surface. i Geometric information fusion features , the normal information of the wear surface pixel i itself and the prior depth information of other pixels in its local neighborhood are embedded into the geometric information fusion feature ; S4. Use the wear surface normal prediction branch, the wear surface prior depth estimation branch, and the wear surface depth reconstruction branch obtained in step S3 to form a wear surface depth reconstruction model, and construct a loss function for each branch respectively. The loss functions of the three branches are fused in the form of a weighted sum as the total loss function, and a synthetic wear surface data set is constructed as training data. The wear surface depth reconstruction model is trained with the minimization of the total loss function as the optimization goal, and the trained wear surface depth reconstruction model is applied to the three-dimensional morphology reconstruction of the actual wear surface.

2. The method for photometric stereo reconstruction of worn surface topography by fusing full-light source images according to claim 1 is characterized in that: Step S1 is specifically as follows: S101, use Photometric stereo information tensor of worn surface under different light sources As the input of the wear surface normal prediction branch, Represents the photometric image resolution, 6 means each tensor has 6 channels, which are the RGB channels of the photometric image and the normalized light source direction. , construct a shared weight encoder consisting of a feature embedding layer and multiple hybrid blocks stacked together to extract the wear surface photometric feature set ; S102, based on the multi-head attention mechanism, construct the photometric feature aggregation layer in the normal prediction branch, using a learnable seed vector As the query vector in the multi-head attention mechanism, the wear surface luminosity feature set is adopted As the key vector and value vector in the multi-head attention mechanism, the aggregated wear surface photometric features are obtained. ; S103, using 5 convolutional layers and 1 L2 normalization layer to establish the normal decoder in the normal prediction branch to wear the surface photometric features Solve for the wear surface normal vector field as input , where each convolutional layer consists of a 3×3 convolution kernel and a LeakyReLU activation function, and the first four convolutional layers use 2x upsampling.

3. The method for photometric stereo reconstruction of worn surface topography by fusing full-light source images according to claim 2 is characterized in that: In step S101, the shared weight encoder is constructed as follows: S1011, the feature embedding layer is set to a kernel size of , the step length is w The convolution layer adds position encoding to convert the information input into a specific feature size for further feature extraction. The tensor size of the feature embedding layer output is ; S1012, mixed block consists of The depth convolution, layer normalization and MLP are connected in series, and residual connections are added after the depth convolution and MLP respectively. The input and output of the hybrid block keep the feature size unchanged.

4. The method for photometric stereo reconstruction of worn surface topography by fusing full-light source images according to claim 1, characterized in that: Step S2 is specifically as follows: S201, the full light source image of the worn surface Connect along the third dimension channel to form the full light source information tensor , to maintain the photometric stereo information tensor The space dimensions are consistent; S202, the wear surface prior depth estimation branch and the wear surface normal prediction branch share a shared weight encoder to achieve the full light source information tensor Extract the full light source features ; S203, using the normal decoder structure with the L2 normalization layer removed to establish the prior depth decoder of the prior depth estimation branch, and adding a dropout rate of 1 for the first four convolutional layers The dropout layer and the regularization term of the network weight; S204, keep the dropout layer open during network training and use, and perform full light source feature extraction based on the Monte Carlo dropout method In the prior depth decoder T Random forward propagation to obtain the prior depth of the wear surface and its uncertainty .

5. The method for photometric stereo reconstruction of worn surface topography by fusing full-light source images according to claim 1, characterized in that: Step S3 is specifically as follows: The geometric information fusion layer in the wear surface depth reconstruction branch repeats step S3 to traverse all pixels on the wear surface and solve the geometric information fusion features to obtain the geometric information feature map ; Three dilated convolutional layers with an expansion rate of 2 and three normal convolutional layers are used to establish the wear surface depth regressor in the depth reconstruction branch, and residual connections are added after the three dilated convolutional layers. All convolutional layers use the LeakyReLU activation function and the convolution kernel size is , connect the geometric information feature map and the prior depth along the third dimension as input to reconstruct the wear surface depth 6. The method for photometric stereo reconstruction of worn surface topography by fusing full-light source images according to claim 1, characterized in that: Pixels i The corresponding depth value The calculation is as follows: in, is the surface normal The three directional components of Pixels of the worn surface i The pixel coordinates of is the pixel coordinate of the principal point of the image, This is the magnification of the weak perspective projection imaging model used in photometric stereo technology.

7. The method for photometric stereo reconstruction of worn surface topography by fusing full-light source images according to claim 1, characterized in that: Wear surface pixels i Geometric information fusion features The calculation is as follows: 。 8. The method for photometric stereo reconstruction of worn surface topography by fusing full-light source images according to claim 1, characterized in that: Step S4 is specifically as follows: S401, using the wear surface normal prediction branch constructed in step S1, the wear surface priori depth estimation branch constructed in step S2, and the wear surface depth reconstruction branch constructed in step S3 to form a wear surface depth reconstruction model; S402, using mean square error loss to calculate the error between the predicted surface normal and the true surface normal and use it as the loss function of the wear surface normal prediction branch , the mean square error loss is used to calculate the error between the prior depth and the true depth and the L2 regularization term of the prior depth decoder weight layer with a weight decay coefficient is added as the loss function of the wear surface prior depth estimation branch , the uncertainty-weighted mean square error loss is used as the loss function of the wear surface depth reconstruction branch The sum of the loss functions of the three branches is fused in the form of weighted sum as the total loss function of the wear surface depth reconstruction model ; S403: Collect the 3D topography of the worn surface, perform random superpixel segmentation on the 3D topography using the Voronoi function, and use the Principled BSDF in the Blender Cycles renderer to set different reflection characteristics for each superpixel area. Randomly sample the number of light reflections and render a photometric stereo image sequence sample. And full light source image samples , while generating surface normal maps and surface depth maps as sample labels, and constructing a synthetic dataset of worn surfaces with strong non-Lambertian effects and spatial variations in reflection characteristics; S404, to minimize the total loss function For the training objective, a synthetic dataset of worn surfaces is used as training data, and the AdamW optimizer is used to train the worn surface depth reconstruction model; S405. Apply the trained wear surface depth reconstruction model to reconstruct the actual wear surface morphology to obtain a surface depth map of the actual wear surface. Based on the perspective projection relationship, convert the surface depth map from the pixel coordinate system to the camera coordinate system to obtain the three-dimensional morphology of the wear surface to be reconstructed.

9. The method for photometric stereo reconstruction of worn surface topography by fusing full-light source images according to claim 8, characterized in that: In step S402, the loss function of the wear surface depth reconstruction branch is for: in, is the uncertainty of the prior depth through Min-Max Normalized pixel points p The uncertainty at Pixel p The wear surface reconstruction depth is is the height value of the image resolution, is the width value of the image resolution, Pixel p The true depth of the worn surface.

10. A photometric stereo reconstruction system for worn surface topography by fusing full-light source images, characterized in that: include: The prediction module combines the photometric images of the worn surface under different light sources and the lighting direction, and establishes a wear surface normal prediction branch based on the encoder-decoder network architecture by embedding a photometric feature aggregation layer using a multi-head attention mechanism; The estimation module combines the full-light source image of the worn surface and establishes a wear surface prior depth estimation branch with the ability to quantify the uncertainty of the prediction results by adding a dropout layer and a network weight regularization term; The depth module, based on the wear surface normal prediction branch constructed by the prediction module and the wear surface prior depth estimation branch constructed by the estimation module, constructs a wear surface depth reconstruction branch consisting of a geometric information fusion layer and a wear surface depth regressor, as follows: Select pixels on the worn surface i of Neighborhood as local neighborhood , s 3, 5, 7, the pixel and local neighborhood In the three-dimensional space transformed from the pixel coordinate system to the camera coordinate system, each point has a three-dimensional coordinate ; Pixels in the local neighborhood j Prior depth As a benchmark, combined with the wear surface pixel points i Surface normal Calculate pixel points i The corresponding absolute depth value in the camera coordinate system ; Pixel j The corresponding uncertainty Processing to obtain prior depth In the local neighborhood Relative credibility in , using neighborhood pixels j and the target pixel i The dot product of the two surface normals is the geometric feasibility of the two points being in the same plane. , and Commonly represent depth values As the target pixel i Deep credibility and As depth value The weight of Calculate the pixels of the worn surface i Relative to the local neighborhood Multiple depth values and corresponding weights of all pixels in the image are used to obtain the pixel points of the worn surface. i Geometric information fusion features , the normal information of the wear surface pixel i itself and the prior depth information of other pixels in its local neighborhood are embedded into the geometric information fusion feature ; The reconstruction module uses the wear surface normal prediction branch, the wear surface prior depth estimation branch, and the wear surface depth reconstruction branch obtained by the depth module to form a wear surface depth reconstruction model. The loss function of each branch is constructed separately, and the loss functions of the three branches are fused in the form of a weighted sum as the total loss function. A synthetic wear surface dataset is constructed as training data, and the wear surface depth reconstruction model is trained with the minimization of the total loss function as the optimization goal. The trained wear surface depth reconstruction model is applied to the three-dimensional morphology reconstruction of the actual wear surface.

Images