Metal tooth surface three-dimensional measurement high dynamic range stripe repairing method based on structure self-attention GAN

By constructing a self-attention GAN for 3D measurement of metal tooth surfaces, the problem of overexposure of striped images caused by high reflectivity of metal surfaces is solved, achieving efficient and accurate 3D reconstruction, which is applicable to 3D measurement of metal materials.

CN118941473APending Publication Date: 2024-11-12CHONGQING UNIV
View PDF 0 Cites 2 Cited by

Patent Information

Application Number
CN202411084586.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-08
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

Existing technologies are unable to effectively address the overexposure problem of striped images caused by high reflectivity of metal surfaces, which affects the accuracy of 3D reconstruction. Furthermore, existing methods suffer from high computational complexity, high hardware costs, and low computational efficiency.

Method used

A three-dimensional measurement method for metal tooth surfaces based on structural self-attention GAN is adopted. By constructing a generative adversarial network SSA-GAN with a self-attention hopping upsampling module, and combining phase shifting method and multi-frequency heterodyne method, the stripe image of the metal surface is repaired, thereby improving the accuracy of three-dimensional reconstruction.

Benefits of technology

It effectively corrects overexposure of stripe patterns caused by metallic reflection, improves the accuracy and efficiency of 3D reconstruction, is applicable to various types of metal materials and stripe patterns, and reduces computational complexity and hardware costs.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118941473A_ABST
    Figure CN118941473A_ABST
Patent Text Reader

Abstract

The invention relates to a metal tooth surface three-dimensional measurement high dynamic range fringe repair method based on a structure self-attention GAN, and belongs to the technical field of computer vision three-dimensional reconstruction, and the method comprises the following steps: S1, collecting a paired image data set of a metal tooth surface overexposure fringe pattern and a normal fringe pattern; s2, fusing local high reflection features in an up-sampling process and shallow semantic features in a down-sampling process on the basis of a traditional generative adversarial network GAN, designing a self-attention jump up-sampling module, and constructing a structural self-attention generative adversarial network SSA-GAN; s3, training the SSA-GAN, adding a structural loss function into a loss function during training, focusing on illumination, contrast and texture information of stripes, and endowing the illumination, contrast and texture information with respective weights; and S4, performing three-dimensional reconstruction on the reflective tooth surface by using the repaired fringe pattern generated by the SSA-GAN in combination with a phase shift method and a multi-frequency heterodyne method to obtain phase information and point cloud information.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision three-dimensional reconstruction, and relates to a high dynamic range stripe repair method for three-dimensional measurement of metal tooth surface based on structural self-attention GAN. Background Art

[0002] Gears are key components in industrial equipment. In order to ensure the manufacturing quality and safe operation of gears, it is very important to perform 3D measurement of gear teeth and their defects. Fringe Projection Profilometry (FPP) is a new non-contact 3D measurement technology with the advantages of high accuracy, simple operation, high resolution, non-destructive surface measurement, and low hardware cost. However, for the FPP method, 3D reconstruction of highly reflective surfaces remains challenging. When a fringe pattern is projected onto a metal gear tooth, diffuse and specular reflections occur. For metal surfaces, when the camera optical axis is parallel to the specular reflection direction, the captured fringe intensity exceeds the dynamic range of the sensor, resulting in loss of fringe information. Obviously, the information loss seriously affects the accuracy of 3D reconstruction. This problem stems from the high reflectivity of the metal surface and cannot be completely solved by adjusting the camera exposure or projector intensity. Phase measurement deflectometry (PMD) can solve the problem of excessive reflection, but it is limited to measuring continuous mirror surfaces. In practical engineering, commonly used methods include spraying an imaging agent onto the surface to convert specular reflection into diffuse reflection, thereby alleviating the high reflectivity problem. However, this approach is not applicable to these parts that may not be sprayed, and handling in-line measurements is challenging.

[0003] To address the above problems, a variety of high dynamic range (HDR) methods have been proposed, which can be mainly classified into: (i) Multiple exposure fusion methods. This method adjusts the exposure by changing the camera exposure time, aperture size or sensitivity, and then continuously shoots multiple patterns. Finally, a directional pattern with a high signal-to-noise ratio is synthesized. However, for objects with a large range of reflectivity changes, many fringe patterns need to be collected, so this technology is not suitable for complex scenes. (ii) Adaptive projection method. This type of method adjusts the intensity of the projected fringe to reduce the saturation of the pattern. However, when the projection intensity is low, the dark reflective area is more susceptible to interference from ambient light. At the same time, this technology also requires the fusion of multiple fringes into a single pattern, which makes the calculation very time-consuming. (iii) Polarizer filtering method. This type of method usually adds a special polarizer in front of the camera to filter out the mirror reflection light, thereby effectively reducing the oversaturation of the metal surface. However, the polarization angle needs to adapt to the light and dark changes of the system fringes, requiring multiple polarizers, which increases the complexity of the measurement system and the hardware cost. (iv) Photometric stereo technology. The reconstruction of the object surface is done using different patterns of photometric information from different directional light sources. However, this technology requires the object to have Lambertian reflectance properties, so it cannot correctly handle object areas with specular reflectance properties. Moreover, it is not suitable for absolute height reconstruction. It can be seen that the photometric stereo method cannot replace the typical 3D reconstruction algorithm. Although the above methods have solved the problem of high reflectivity to a certain extent, the following challenges still exist, including the increase in the number of fringe patterns, the decrease in measurement efficiency, the imbalance between exposed areas and underexposed areas, high computational complexity, and the increase in hardware costs.

[0004] Deep learning has shown impressive advantages in measurement and defect detection. Regarding stripe repair, Xi et al. used an additional reflective prior network to generate feature masks to guide the encoding and decoding networks. The network generated high-quality non-reflective stripe patterns and achieved the restoration of stripes on the surfaces of various metal parts. This method also has problems such as long training time, complex training process, and unstable training. In addition, the quality of the generated stripe patterns needs to be further improved. Summary of the invention

[0005] In view of this, the object of the present invention is to provide a high dynamic range stripe repair method for three-dimensional measurement of metal tooth surface based on structural self-attention GAN.

[0006] In order to achieve the above object, the present invention provides the following technical solutions:

[0007] A high dynamic range stripe restoration method for three-dimensional measurement of metal tooth surface based on structured self-attention GAN, comprising the following steps:

[0008] S1: Collect paired image datasets of overexposed fringe patterns and normal fringe patterns on metal tooth surfaces;

[0009] S2: Based on the traditional generative adversarial network (GAN), we integrate the local high-reflection features in the upsampling process and the shallow semantic features in the downsampling process, design a self-attention skipping upsampling block, and construct a structured self-attention generative adversarial network (SSA-GAN);

[0010] S3: training the SSA-GAN, adding a structural loss function to the loss function during training, focusing on the illumination, contrast, and texture information of the stripes, and assigning weights to the three respectively;

[0011] S4: Using the repaired fringe pattern generated by SSA-GAN, combined with the phase shift method and multi-frequency heterodyne method, the reflective tooth surface is reconstructed in three dimensions to obtain phase information and point cloud information.

[0012] Further, in step S1, projecting and capturing images by means of a structured light projector and an industrial camera specifically includes the following steps:

[0013] S11: Control the structured light projector to project a four-frequency nine-step stripe pattern [1, 3, 9, 36] onto the surface of the object, and use an industrial camera to capture the stripe pattern;

[0014] S12: fixing the position of the gear, spraying the developer onto the tooth surface, and when the developer is stable, repeating the projection and image capture steps to obtain the corresponding ideal non-reflective fringe pattern;

[0015] S13: obtaining a paired data set consisting of a plurality of pairs of overexposed fringe images and normal fringe images;

[0016] S14: Divide the paired dataset into training set, test set and validation set in proportion.

[0017] Furthermore, the structural self-attention generative adversarial network SSA-GAN is composed of a generator network G and a discriminator network D;

[0018] The structure of the generator network G is based on U-net, and its downsampling module consists of convolution, regularization, ReLU activation function and Dropout layer; the upsampling module consists of a self-attention jump upsampling module, and the network in the self-attention jump upsampling module has seven layers, the input is an overexposed fringe image of 1024*1280, and the output is a normal exposure fringe image of 1024*1280;

[0019] The discriminator network D consists of three layers: convolution, normalization, and ReLU operations. Finally, the feature map is expanded and then the output is obtained through convolution. Its input is the 1024*1280 fake stripe map generated by G, and the output is a 1024*1280 judgment map. The value of each pixel is 0 or 1, which means that the image generated by the generator is judged to be fake or real.

[0020] G and D are trained alternately through various loss functions until convergence, that is, the generator generates a fringe pattern that is closest to the true normal exposure fringe pattern.

[0021] Furthermore, the input feature map of the self-attention jump upsampling module comes from the upsampling process of the generator U-Net network. First, the features with less irrelevant noise in the upsampling process are obtained, and the high-reflective area is given additional attention weights with the help of SimAM. Then, the features extracted in the downsampling process of the U-net network are spliced ​​and fused. The features after channel filtering are spliced ​​and output with the initial upsampling features to replace the upsampling process of the original U-net network.

[0022] The self-attention jump upsampling module is expressed by the following formula:

[0023]

[0024] Where up represents the up-sampled feature, and Concat represents concatenation operation, Conv represents convolution operation, Drop represents masked neuron operation, Relu represents activation function, Norm represents normalization, and TransConv represents deconvolution operation. represents the features sampled by the U-net network, k represents the number of layers of the U-net network, It means giving SimAM attention to the high-reflectivity areas of the upsampled feature map.

[0025] Furthermore, in the SSA-GAN, attention needs to estimate the importance of individual neurons in the network during the feature information transmission process, and important neurons are identified by measuring the linear separability of the target neuron and other neurons. The solution to the minimum energy of the neuron is:

[0026]

[0027] in, is the mean and variance of all neurons in the channel except the target neuron t; M = H × W represents the number of neurons in the channel, and λ is a constant; the lower the energy, the more different the neuron is from the surrounding neurons, and the higher its importance;

[0028] Use 1 / e *Indicates the importance of each neuron. According to the attention mechanism, different degrees of importance are given to the features, that is, enhancement processing is performed:

[0029]

[0030] Where E represents all channels and spatial dimensions X stands for feature.

[0031] Furthermore, in step S3, the optimization objective of SSA-GAN is expressed as:

[0032]

[0033] Where a represents the input reflective fringe pattern b real are the labels of the fringe plots, G(a) and b fake Represents the stripe graph generated by the network; the generator G minimizes the objective function, and the discriminator D maximizes the objective function;

[0034] In the specific training process, other parameters remain fixed while a single parameter is trained. As the adversarial training proceeds, G and D will reach a Nash equilibrium at some point.

[0035] Furthermore, the steps of constructing the structural loss function are as follows:

[0036] First, the illumination similarity of paired fringe images is expressed as:

[0037]

[0038] where μ a Represents the average pixel value within the pixel range, C 1 =(K 1 L) 2 is a constant, K 1 is a constant, L is the grayscale range, and the fringe pattern contrast similarity is expressed as:

[0039]

[0040] where σ x Represents the standard deviation within the pixel range;

[0041] The structural similarity of the fringe pattern is expressed as:

[0042]

[0043] Convert the overall structural similarity into a structural loss function:

[0044]

[0045] Among them, α, β and γ represent the importance of each similarity, and different values ​​are set according to specific goals.

[0046] Furthermore, the overall loss function of the structured self-attention generation adversarial network SSA-GAN is as follows:

[0047]

[0048]

[0049] Where a represents the input reflective fringe pattern b real Indicates the label of the fringe pattern, b fake represents the fringe plot generated by the network; represents the L1 loss function;

[0050] The purpose is to constrain the false stripe pattern to have a connection with the real stripe pattern. The weight distribution principle is α>β>γ.

[0051] The purpose is to make the generated false stripes close to the real stripe pattern. The principle of allocating weights is α=β=γ.

[0052] Further, step S4 specifically includes the following steps:

[0053] The projected fringes are represented as follows:

[0054]

[0055] Where A(x,y) represents the background light intensity, B(x,y) represents the fringe modulation, is the unwrapped phase of the object, (x, y) is the pixel coordinate of the object, and n is the phase shift coefficient, n = 1, 2, 3...N; the wrapped phase is calculated using the N-step phase shift method; according to the projection fringe formula, the unwrapped phase is expressed as:

[0056]

[0057] After obtaining the phase information, phase unwrapping is required to ensure a one-to-one correspondence between the pixel points and the phase values;

[0058] The full-field phase unwrapping is achieved through time-domain and space-domain-based technologies; the time-based phase unwrapping method is widely used. The multi-frequency heterodyne method is used to achieve time-domain-based phase unwrapping, which is achieved by projecting different fringe patterns of different frequencies. The absolute phase obtained is a continuous phase distribution covering the entire pixel range. The absolute phase is expressed as:

[0059]

[0060] Where Φ(x,y) is the absolute phase and k(x,y) represents the fringe order.

[0061] The beneficial effects of the present invention are that the present invention can effectively repair the situation in which the fringe pattern is overexposed and the phase information is lost due to metal reflection during the fringe projection three-dimensional reconstruction process, and can process various types of modulated fringe patterns, including different metal materials, fringe types and overexposure degrees, and can repair the fringe pattern more accurately than the existing method, thereby improving the accuracy of subsequent 3D detection.

[0062] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below in conjunction with the accompanying drawings, wherein:

[0064] Figure 1 The data set shows paired images of overexposed fringe patterns and normal fringe patterns. The objects under test are metal tooth surfaces and industrial metal parts.

[0065] Figure 2 This is the overall structure of the self-attention skipping upsampling block.

[0066] Figure 3 The overall structure of the structured self-attention stripe restoration network SSA-GAN. DETAILED DESCRIPTION

[0067] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0068] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present invention, and thus the drawings only show components related to the present invention rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0069] In the following description, numerous details are discussed to provide a more thorough explanation of the embodiments of the present invention. However, it is obvious to those skilled in the art that the embodiments of the present invention can be implemented without these specific details. In other embodiments, well-known structures and devices are shown in the form of block diagrams rather than in detail to avoid making the embodiments of the present invention difficult to understand.

[0070] The present invention provides a high dynamic range stripe repair method for three-dimensional measurement of metal tooth surface based on structured self-attention GAN, comprising the following steps:

[0071] S1: A dataset of paired images of overexposed fringe patterns and normal fringe patterns on metal tooth surfaces is collected by a structured light projector and a camera; the paired image dataset of overexposed fringe patterns and normal fringe patterns, the object to be measured is a metal tooth surface with reflective characteristics, and the image is projected and captured by controlling a DLP4500 projector and an industrial camera (MER-131210U3C). First, the development software controlling the projector projects a four-frequency nine-step fringe pattern [1, 3, 9, 36] onto the surface of the object. The camera captures the fringe pattern, then fixes the position of the gear and sprays the developer onto the tooth surface. When the developer is stable, the projection and image capture steps are repeated to obtain the corresponding ideal non-reflective fringe pattern. The paired dataset contains 1728 pairs of overexposed fringe patterns and normal fringe patterns, named GearFace1728, which is divided into training set, test set, and validation set in a ratio of 8:1:1. There is also a similar dataset, which is industrial metal parts, with a total of 1800 pairs for verification, named IndustrialParts1800, with the same division ratio as GearFace1728. Figure 1 As shown, the first row is an overexposed fringe pattern, and the second row is a normal fringe pattern.

[0072] S2: Construct a structured self-attention generative adversarial network (SSA-GAN), the overall network structure is based on the generative adversarial network and the convolutional network.

[0073] The SSA-GAN network consists of a generator network G and a discriminator network D. The structure of the generator network G is based on U-net, and its downsampling module consists of convolution, regularization, ReLU activation function and Dropout layer. The upsampling module is composed of the self-attention jump upsampling module (SSUBlock) in step S4. The network in this module has seven layers. The input is an overexposed fringe image of 1024*1280, and the output is a normally exposed fringe image of 1024*1280. The discriminator network D consists of convolution, normalization, and ReLU operations, with a total of three layers. Finally, the feature map is expanded and then subjected to convolution operation to obtain the output. Its input is a 1024*1280 false fringe image generated by G, and the output is a 1024*1280 judgment image. The value of each pixel is 0 or 1, that is, it is judged whether the image generated by the generator is false or true. G and D are alternately trained through various loss functions until convergence, that is, the generator generates a fringe image that is closest to the true value normally exposed fringe image. The structural diagram of the entire network is shown in the figure. Figure 3 As shown in the figure, the configuration parameters used in the training process of the entire network are as follows: PyTorch2.0, Python3.8, CUDA12.2 and CUDNN8.9.2 are used. The Adam optimizer with a learning rate of 0.0002 is used. A total of 300 epochs were trained on the IndustrialParts1800 dataset and 400 epochs were trained on the GearFace1728 dataset. The network was deployed on a computing platform that includes an RTX3090 GPU, an Intel(R) Core(TM) i5-7500 CPU, and Windows10. The method includes multiple modules and an improved loss function.

[0074] In order to improve the network's attention to the high-reflection stripe area, a self-attention skipping upsampling block is designed by fusing the local high-reflection features in the upsampling process and the shallow semantic features in the downsampling process, so as to strengthen the network's learning of the intrinsic connection between the high-reflection area and the non-reflection area and improve the network's reasoning ability. The self-attention skipping upsampling block used, the input feature map comes from the upsampling process of the generator U-Net network, first obtains the features with less irrelevant noise in the upsampling process, and gives the high-reflection area additional attention weights with the help of SimAM, and then splices and fuses the features extracted in the downsampling process of the U-net network, and splices and outputs the features after channel filtering with the initial upsampling features to replace the upsampling process of the original U-net network. The generator has seven layers of upsampling modules. In the bottom layer, the number of channels of the input feature map is 512, and the size of the feature map is 10*8. After the proposed upsampling module, the number of channels of the output feature map is 1024, and the size of the feature map is 10*8. After 7 layers of upsampling modules, the number of channels of the feature map output of the last layer is 128, and the size is 640*512. After another layer of convolution operation, the final output is obtained, with 3 channels and a size of 1280*1024. The structure diagram of the self-attention jump upsampling module (SSU module) is shown in the figure. Figure 2 shown.

[0075] The proposed upsampling module can not only enable the network to bypass the information bottleneck, but also strengthen the network to learn the intrinsic connection between high-reflection areas and non-reflection areas, thereby improving the network's reasoning ability. The entire module can be expressed by the following formula:

[0076]

[0077] Where up represents the up-sampled feature, and Concat represents concatenation operation, Conv represents convolution operation, Drop represents masked neuron operation, Relu represents activation function, Norm represents normalization, and TransConv represents deconvolution operation. represents the features sampled by the U-net network, k represents the number of layers of the U-net network, It means giving SimAM attention to the high-reflectivity areas of the upsampled feature map.

[0078] Since the required task involves restoring pixels in reflective areas, extra attention should be paid to highly reflective areas. Successful attention requires estimating the importance of individual neurons in the network during feature information transfer. In visual neuroscience, the most informative neurons (the most important neurons) have unique firing patterns compared to surrounding neurons. Measuring the linear separability of target neurons and other neurons can help identify important neurons. The neuron energy function can complete the above identification work, and the solution to the minimum neuron energy is:

[0079]

[0080] in, is the mean and variance of all neurons in the channel except the target neuron t. M = H × W represents the number of neurons in the channel, and λ is a constant. The lower the energy, the more different the neuron is from the surrounding neurons and the more important it is; therefore, 1 / e * It can be used to indicate the importance of each neuron. According to the attention mechanism, different degrees of importance are given to features, that is, enhancement processing is performed:

[0081]

[0082] Where E represents all channels and spatial dimensions X stands for feature.

[0083] S3: Training and ablation experiments on the proposed structured self-attention generative adversarial network (SSA-GAN). According to the characteristics of the stripe pattern itself, a structural loss function is proposed, focusing on the illumination, contrast, and texture information of the stripes, and giving each of the three weights. Combined with other loss functions, the network can accurately restore the high-reflective area of ​​the tooth surface stripe pattern where the phase information is lost; the proposed structural loss function combined with other loss functions can enhance the authenticity and quality of the stripe pattern generated by the network. The loss function of the entire network is as follows:

[0084] The optimization objective of the entire structural GAN ​​can be expressed as:

[0085]

[0086] Where a represents the input reflective fringe pattern b real are the labels of the fringe plots, G(a) and b fake Represents the stripe graph generated by the network. Parameter G minimizes the objective function, while D maximizes it. In the specific training process, when a single parameter is trained, other parameters remain fixed. As adversarial training proceeds, G and D will reach a Nash equilibrium at some point.

[0087] In the traditional image-to-image training process, L1 distance is used instead of L2 distance. This choice enables G to deceive the discriminator D, approach edge facts in the L2 sense, and reduce ambiguity. The parameter L1 loss can be expressed as:

[0088]

[0089] Above The loss forces the generator to learn a pixel-level mapping by minimizing the pixel difference between the generator output and the corresponding stripes. This approach ensures clarity, but it ignores certain characteristics of the stripe pattern dataset, such as brightness variations and high-contrast features.

[0090] Therefore, the structural loss function is proposed To make up for the shortcomings of the L1 loss function, first, the illumination similarity of the paired fringe images can be expressed as:

[0091]

[0092] where μ a Represents the average pixel value within the pixel range, C 1 =(K 1 L) 2 is a constant, K 1 is a constant, and L is the grayscale range. The fringe pattern contrast similarity can be expressed as:

[0093]

[0094] where σ x Represents the standard deviation within the pixel range. The structural similarity of the fringe pattern can be expressed as:

[0095]

[0096] Convert the overall structural similarity into a loss function:

[0097]

[0098] Among them, α, β, and γ represent the importance of each similarity, and different values ​​are set according to specific goals.

[0099] The specific loss function of the network can be expressed by the following formula:

[0100]

[0101] In the above formula, The purpose is to constrain the false fringe pattern to have a connection with the real fringe pattern. Since the illumination similarity between the input and output is relatively low, the importance of illumination similarity is reduced. However, the contrast similarity is high, especially in the light and dark fringes. Therefore, the importance of contrast similarity increases. The importance of texture similarity should be the highest to ensure that the texture structure of the output fringes is not destroyed. Therefore, The weight distribution principle is α>β>γ.

[0102] In the above formula,

[0103] The aim is to make the generated fake stripes close to the real stripe pattern. Illumination, contrast, and texture feature information should be given equal attention to ensure clarity (in addition to using L1 loss). Moreover, the importance of these three aspects is equal. Therefore, in The principle of allocating weights is α=β=γ.

[0104] S4: The repaired fringe image generated by SSA-GAN is combined with the phase shift method and the multi-frequency heterodyne method to perform three-dimensional reconstruction of the reflective tooth surface to obtain phase information and point cloud information. The basic principle of using the phase shift method and the multi-frequency heterodyne method to perform three-dimensional reconstruction of the reflective tooth surface is as follows:

[0105] The basic principles of fringe projection profilometry (FPP) include the projection of sinusoidal fringes, the capture of deformed fringes, system calibration, the acquisition of wrapped phase and absolute phase, point cloud computing, and 3D shape reconstruction. As an important step of FPP, obtaining wrapped phase includes phase shift profilometry (PSP) and Fourier transform profilometry (FTP). PSP provides accurate and reliable results and is less sensitive to environmental conditions and noise. In addition, due to the advancement of projection technology and camera performance, PSP has been increasingly used in real-time 3D measurement in recent years. First, the projected fringes can be expressed as follows:

[0106]

[0107] Where A(x,y) represents the background light intensity, B(x,y) represents the fringe modulation, is the unwrapped phase of the object, (x, y) is the pixel coordinate of the object, and n is the phase shift coefficient (n = 1, 2, 3...N). The wrapped phase can be calculated using the N-step phase shift method. According to the projection fringe formula, the unwrapped phase can be expressed as:

[0108]

[0109] After obtaining the phase information, phase unwrapping is required to ensure a one-to-one correspondence between the pixel points and the phase values. Full-field phase unwrapping can be achieved through techniques based on the time domain and the space domain. Time-based phase unwrapping methods are widely used. Multi-frequency heterodyning is one of the methods, which projects different fringe patterns of different frequencies to achieve this. The absolute phase obtained is a continuous phase distribution covering the entire pixel range. The absolute phase can be expressed as:

[0110]

[0111] Where Φ(x,y) is the absolute phase and k(x,y) represents the fringe order.

[0112] In the above embodiments, the description's reference to "this embodiment" indicates that a particular feature, structure, or characteristic described in conjunction with the embodiment is included in at least some embodiments, but not necessarily all embodiments. Multiple occurrences of "this embodiment" do not necessarily all refer to the same embodiment.

[0113] In the above-described embodiments, although the invention has been described in conjunction with specific embodiments of the invention, many substitutions, modifications, and variations of these embodiments will be apparent to those of ordinary skill in the art based on the foregoing description. For example, other storage structures (e.g., dynamic RAM (DRAM)) may use the embodiments discussed. Embodiments of the invention are intended to encompass all such substitutions, modifications, and variations that fall within the broad scope of the appended claims.

[0114] This embodiment further provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, any one of the methods in this embodiment is implemented.

[0115] This embodiment also provides an electronic terminal, including: a processor and a memory;

[0116] The memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory, so that the terminal executes any one of the methods in this embodiment.

[0117] The computer-readable storage medium in this embodiment can be understood by ordinary technicians in this field: all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to the computer program. The aforementioned computer program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, magnetic disk or optical disk and other media that can store program codes.

[0118] The electronic terminal provided in this embodiment includes a processor, a memory, a transceiver and a communication interface. The memory and the communication interface are connected to the processor and the transceiver and complete communication with each other. The memory is used to store computer programs, the communication interface is used to communicate, and the processor and the transceiver are used to run computer programs so that the electronic terminal executes each step of the above method.

[0119] In this embodiment, the memory may include a random access memory (RAM), and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0120] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0121] The present invention can be used in many general or special computing system environments or configurations, such as personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like.

[0122] The present invention may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present invention may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0123] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.

Claims

1. A high dynamic range stripe restoration method for three-dimensional measurement of metal tooth surface based on structural self-attention GAN, characterized by: The following steps are involved: S1: Collect paired image datasets of overexposed fringe patterns and normal fringe patterns on metal tooth surfaces; S2: Based on the traditional generative adversarial network GAN, we integrate the local high-reflection features in the upsampling process and the shallow semantic features in the downsampling process, design a self-attention jump upsampling module, and construct a structural self-attention generative adversarial network SSA-GAN; S3: training the SSA-GAN, adding a structural loss function to the loss function during training, focusing on the illumination, contrast, and texture information of the stripes, and assigning weights to the three respectively; S4: Using the repaired fringe pattern generated by SSA-GAN, combined with the phase shift method and multi-frequency heterodyne method, the reflective tooth surface is reconstructed in three dimensions to obtain phase information and point cloud information.

2. The high dynamic range stripe restoration method for three-dimensional measurement of metal tooth surface based on structural self-attention GAN according to claim 1 is characterized by: In step S1, an image is projected and captured by a structured light projector and an industrial camera, which specifically includes the following steps: S11: Control the structured light projector to project a four-frequency nine-step stripe pattern [1, 3, 9, 36] onto the surface of the object, and use an industrial camera to capture the stripe pattern; S12: fixing the position of the gear, spraying the developer onto the tooth surface, and when the developer is stable, repeating the projection and image capture steps to obtain the corresponding ideal non-reflective fringe pattern; S13: obtaining a paired data set consisting of a plurality of pairs of overexposed fringe images and normal fringe images; S14: Divide the paired dataset into training set, test set and validation set in proportion.

3. The high dynamic range stripe restoration method for three-dimensional measurement of metal tooth surface based on structural self-attention GAN according to claim 1 is characterized in that: The structured self-attention generative adversarial network SSA-GAN consists of a generator network G and a discriminator network D; The structure of the generator network G is based on U-net, and its downsampling module consists of convolution, regularization, ReLU activation function and Dropout layer; the upsampling module consists of a self-attention jump upsampling module, and the network in the self-attention jump upsampling module has seven layers, the input is an overexposed fringe image of 1024*1280, and the output is a normal exposure fringe image of 1024*1280; The discriminator network D consists of three layers: convolution, normalization, and ReLU operations. Finally, the feature map is expanded and then the output is obtained through convolution. Its input is the 1024*1280 fake stripe map generated by G, and the output is a 1024*1280 judgment map. The value of each pixel is 0 or 1, which means that the image generated by the generator is judged to be fake or real. G and D are trained alternately through various loss functions until convergence, that is, the generator generates a fringe pattern that is closest to the true normal exposure fringe pattern.

4. The high dynamic range stripe restoration method for three-dimensional measurement of metal tooth surface based on structural self-attention GAN according to claim 1 is characterized by: The input feature map of the self-attention jump upsampling module comes from the upsampling process of the generator U-Net network. First, the features with less irrelevant noise in the upsampling process are obtained, and the high-reflective area is given additional attention weights with the help of SimAM. Then, it is spliced ​​and fused with the features extracted in the downsampling process of the U-net network. The features after channel filtering are spliced ​​and output with the initial upsampling features to replace the upsampling process of the original U-net network; The self-attention jump upsampling module is expressed by the following formula: Where up represents the up-sampled feature, and Concat represents concatenation operation, Conv represents convolution operation, Drop represents masked neuron operation, Relu represents activation function, Norm represents normalization, and TransConv represents deconvolution operation. represents the features sampled by the U-net network, k represents the number of layers of the U-net network, It means giving SimAM attention to the high-reflectivity areas of the upsampled feature map.

5. The high dynamic range stripe restoration method for three-dimensional measurement of metal tooth surface based on structured self-attention GAN according to claim 1, characterized in that: In the SSA-GAN, attention needs to estimate the importance of individual neurons in the network during the feature information transmission process. Important neurons are identified by measuring the linear separability of the target neuron and other neurons. The solution to the minimum energy of the neuron is: in, is the mean and variance of all neurons in the channel except the target neuron t; M = H × W represents the number of neurons in the channel, and λ is a constant; the lower the energy, the more different the neuron is from the surrounding neurons, and the higher its importance; Use 1 / e * Indicates the importance of each neuron. According to the attention mechanism, different degrees of importance are given to the features, that is, enhancement processing is performed: Where E represents all channels and spatial dimensions X stands for feature.

6. The high dynamic range stripe restoration method for metal tooth surface three-dimensional measurement based on structural self-attention GAN according to claim 1 is characterized by: In step S3, the optimization objective of SSA-GAN is expressed as: Where a represents the input reflective fringe pattern b real are the labels of the fringe plots, G(a) and b fake Represents the stripe graph generated by the network; the generator G minimizes the objective function, and the discriminator D maximizes the objective function; In the specific training process, other parameters remain fixed while a single parameter is trained. As the adversarial training proceeds, G and D will reach a Nash equilibrium at some point.

7. The high dynamic range stripe restoration method for metal tooth surface three-dimensional measurement based on structured self-attention GAN according to claim 1, characterized in that: The steps for constructing the structural loss function are as follows: First, the illumination similarity of paired fringe images is expressed as: where μ a Represents the average pixel value within the pixel range, C1=(K1L) 2 is a constant, K1 is a constant, L is the grayscale range, and the fringe pattern contrast similarity is expressed as: where σ x Represents the standard deviation within the pixel range; The structural similarity of the fringe pattern is expressed as: Convert the overall structural similarity into a structural loss function: Among them, α, β and γ represent the importance of each similarity, and different values ​​are set according to specific goals.

8. The high dynamic range stripe restoration method for metal tooth surface three-dimensional measurement based on structured self-attention GAN according to claim 1, characterized in that: The overall loss function of the structured self-attention generation adversarial network SSA-GAN is as follows: Where a represents the input reflective fringe pattern b real Indicates the label of the fringe pattern, b fake represents the fringe plot generated by the network; represents the L1 loss function; The purpose is to constrain the false stripe pattern to have a connection with the real stripe pattern. The weight distribution principle is α>β>γ; The purpose is to make the generated false stripes close to the real stripe pattern. The principle of allocating weights is α=β=γ.

9. The high dynamic range stripe restoration method for three-dimensional measurement of metal tooth surface based on structured self-attention GAN according to claim 1, characterized in that: Step S4 specifically includes the following steps: The projected fringes are represented as follows: Where A(x,y) represents the background light intensity, B(x,y) represents the fringe modulation, is the unwrapped phase of the object, (x, y) is the pixel coordinate of the object, and n is the phase shift coefficient, n = 1, 2, 3...N; the wrapped phase is calculated using the N-step phase shift method; according to the projection fringe formula, the unwrapped phase is expressed as: After obtaining the phase information, phase unwrapping is required to ensure a one-to-one correspondence between the pixel points and the phase values; The full-field phase unwrapping is achieved through technologies based on the time domain and the space domain. The multi-frequency heterodyne method is used to achieve the phase unwrapping based on the time domain, and different fringe patterns with different frequencies are projected to achieve it. The absolute phase obtained is a continuous phase distribution covering the entire pixel range. The absolute phase is expressed as: Where Φ(x,y) is the absolute phase and k(x,y) represents the fringe order.

Citation Information

Cited By

  • Complex shell detection method and system based on structured light scanning

    CN121540086A

  • Three-dimensional repair method based on fringe projection profilometry

    CN121982221A