An imaging simulation method for mirror defect detection system based on structured light field

Abstract

An imaging simulation method for a mirror defect detection system based on a structured light field comprises the following steps: S1: calibrating system parameters as input to a simulation model. S2: deriving a light propagation model for a light field camera, tracing rays from the sensor, parallelizing the calculation of the light field sub-graph array, and solving the incident light vector of the light field camera. S3: designing a function to fit the three-dimensional surface shape of the deformation defect, solving the incident light vector at the defect, and tracing the light to obtain the coordinates of the light source point and the light intensity. S4: Based on the Lambert diffuse reflection model, introducing coefficients characterizing the roughness of scratches / the opacity of stains, and establishing a grayscale model of diffuse reflection non-deformable defects. S5: generating a simulated image and its label. Comparing the simulated image with the actual image to evaluate the reliability of the simulation model. Evaluating the operating efficiency of the simulation model. The present invention effectively improves the accuracy and speed of imaging simulation for light field detection of mirror defects, and can be used for applications such as parameter optimization and generation of simulated image data sets before building a detection system.

Description

Technical Field

[0001] The present invention relates to the field of mirror defect detection, and in particular to an imaging simulation method of a mirror defect detection system based on a structured light field. Background Art

[0002] Metal workpieces are often treated with processes such as electroplating and polishing to improve corrosion resistance, wear resistance, and gloss. The resulting highly reflective surface is called a mirror. Defects such as dents and scratches on mirror surfaces not only affect the aesthetics but also compromise the mechanical properties of the workpiece, posing a safety hazard. Traditional defect detection relies on intensive, repetitive manual visual labor, which is not only detrimental to worker health but also suffers from low efficiency and accuracy. Consequently, automated inspection methods based on machine vision systems have attracted considerable attention.

[0003] Compared with the defect detection of ordinary diffuse reflection surfaces, the automatic detection system for mirror surface defects has more stringent requirements on lighting and imaging conditions.

[0004] In terms of lighting, due to the highly reflective nature of mirror surfaces and the different reflective properties of different mirror defects, the use of traditional uniform lighting methods is prone to overexposure or underexposure. Some related methods for lighting enhancement have been developed. The dark field scattering method uses the abnormal reflective properties of the defect to contrast with the background, and it is difficult to capture defect information with normal reflective properties; the photometric stereo method uses the deformation characteristics of the defect to distinguish it from the background, and is insensitive to non-deformation defect information; the structured light imaging method presents different defects through different imaging depths: deformation defects that are still mirror reflections are observed through the distortion of the virtual image formed by the stripes, while non-deformation defects with diffuse reflections on the surface are formed by the integration of ambient light at the diffuse reflection point and the background without defects. The simultaneous imaging of the two puts forward requirements for the depth of field capability of the imaging equipment.

[0005] In terms of imaging, in order to increase the exposure of defects, some imaging enhancement methods have been developed, such as time-domain multi-exposure and multi-camera exposure methods. The former sacrifices time complexity in exchange for images of different exposure scenes, while the latter is very sensitive to noise interference such as vibration when calibrating multiple cameras together. Light field cameras record four-dimensional light field information by installing a microlens array (MLA) between the ordinary camera sensor and the main lens. Multi-perspective imaging of the same target can be performed in a single exposure, and the depth of field of the camera can be extended without changing the aperture.

[0006] Therefore, the detection system using structured light as the lighting method and light field camera as the imaging device has the ability to quickly and reliably image mirror defects.

[0007] The machine vision system mainly includes hardware systems and software systems. The former captures defect images and focuses on the imaging effect of defects; the latter processes defect images and focuses on the detection effect of defects.

[0008] Hardware system design typically involves system construction, defect image acquisition, and imaging evaluation, followed by repeated adjustments of system parameters based on experience. This trial-and-error process is inherently time-consuming and expensive, while repeated replacement of system components also incurs significant financial costs. Imaging simulation of machine vision systems greatly simplifies the parameter design process before system construction, saving both time and money.

[0009] Software system design typically involves collecting defective workpieces, collecting and annotating defect image samples, and then fine-tuning the detection algorithm based on the constructed defect image dataset. On actual production lines, the number of defective workpieces is extremely small, resulting in a small defect image dataset and, in turn, insufficient generalization performance of the detection algorithm. Defect imaging simulation can automatically generate large datasets and automatically annotate them by varying defect parameters, expanding the defect dataset without increasing labor costs.

[0010] Therefore, accurate and fast imaging simulation models play an important role in system parameter design and defect data set expansion.

[0011] Existing imaging simulation technologies for machine vision defect detection systems include data-driven methods, wave optics methods, and geometric optics methods.

[0012] The data-driven method uses neural networks, usually generative adversarial networks (GANs), for training to fit the imaging results of a specific system. It can achieve detailed simulation of imaging, including vignetting, aberrations, distortion, and noise of cameras and sensors, and is suitable for black box situations such as unknown optical designs.

[0013] When the optical design is known, optical design software such as CODE V and Zemax can be used for imaging simulation. The software is generally based on the principles of wave optics and geometric optics.

[0014] Wave optics methods can simulate diffraction effects by analyzing light wave propagation models in different media and deriving the point spread function (PSF) to describe the imaging system's response to point sources. Therefore, it is necessary to calculate the holographic imaging response to each point source.

[0015] Geometric optics methods trace light paths starting from the image plane and recursively solve for incident light rays reflected or refracted on the object's surface, ultimately determining the light source point and its intensity corresponding to each pixel on the image plane. Ray tracing that accounts for nonlinear processes such as distortion and aberrations is computationally intensive. Paraxial ray tracing, which employs Taylor expansion and first-order approximation, greatly simplifies the calculations, but at the expense of accuracy for distortion and aberrations.

[0016] The above imaging simulation technology has the following problems when applied to defect detection systems:

[0017] (1) The data-driven method is based on a large number of real system images, which makes it impossible to apply to imaging simulation before the system is built. In addition, due to the small number of defect samples in actual production and the limited size of the defect data set, it is difficult to train defective system imaging even for a system that has been actually built. In addition, this method is essentially a black-box simulation of the imaging system using a nonlinear neural network. Once the system parameters are changed, the trained model needs to be repeatedly fine-tuned or even retrained.

[0018] (2) The wave optics method requires calculating the imaging response of each image point in the image space to each object point in the object space and superimposing them. The computational complexity is high, and when used to assist in the optimization design of system parameters, it will lead to a long design cycle.

[0019] (3) The geometric optics method can significantly reduce the amount of ray tracing calculations through paraxial approximation, and the simulation speed is faster, but it loses the accuracy of distortion and aberration related to camera optical design.

[0020] (4) Currently, there is a lack of complete imaging simulation models for defect detection systems, especially for defect modeling. Research on imaging simulation focuses more on the simulation of imaging systems.

[0021] In summary, for mirror defect detection systems using structured light and light field cameras, it is necessary to improve the imaging simulation method and establish a complete imaging simulation model that optimizes the accuracy and speed of mirror defect detection.

[0022] It should be noted that the information disclosed in the above background technology section is only used to understand the background of this application, and therefore may include information that does not constitute prior art known to ordinary technicians in this field. Summary of the Invention

[0023] The main purpose of the present invention is to overcome the defects of the above-mentioned background technology and provide an imaging simulation method of a mirror defect detection system based on structured light field.

[0024] To achieve the above object, the present invention adopts the following technical solutions:

[0025] An imaging simulation method for a mirror defect detection system based on a structured light field comprises the following steps:

[0026] S1: calibrate system parameters as input for simulation;

[0027] S2: Using the calibrated system parameters, ray tracing is performed from the sensor. The incident light vector of the light field camera is solved by parallel computing methods to construct a light propagation model for the light field camera.

[0028] S3: Based on the light propagation model of the light field camera, a function is designed to fit the three-dimensional surface shape of the deformation defect. The model is used to solve the incident light vector at the defect, and the light is traced to determine the coordinates of the light source point and the light intensity, thereby performing imaging simulation of the deformation defect;

[0029] S4: Using the incident light information provided by the light propagation model of the light field camera, the grayscale value of the diffuse reflection defect is calculated using the Lambert diffuse reflection model, and a coefficient representing the roughness of the scratch or the opacity of the stain is introduced to establish a grayscale model of the diffuse reflection non-deformable defect, thereby simulating the visual appearance of the diffuse reflection defect under different lighting conditions;

[0030] S5: Based on the imaging simulation of the deformation defect in step S3 and the grayscale model of the diffuse reflection non-deformation defect established in step S4, a simulation image and a label are generated.

[0031] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the imaging simulation method of a mirror defect detection system based on a structured light field.

[0032] The present invention has the following beneficial effects:

[0033] The present invention proposes an imaging simulation method for a mirror defect detection system based on a structured light field. It proposes the construction of a geometric optical imaging simulation model for a mirror defect detection system based on a structured light field. This method solves the problem of a lack of imaging simulation methods for mirror defects under a structured light field and the difficulty of existing imaging simulation methods in achieving both high precision and high efficiency. It improves the accuracy and speed of imaging simulation for mirror defect detection and can be used for parameter optimization and generation of simulated image data sets before the detection system is built. Specifically, the present invention can be used to simulate imaging effects before the system is built, assist in system parameter design, and save time and money on trial and error in system design. It can also be used to generate simulated images and automatically annotate data labels, expand defect data sets, and save labor costs for data collection and annotation.

[0034] Compared with the prior art, the main advantages of the embodiments of the present invention are:

[0035] This invention uses a computationally simplified pinhole camera model to derive the light propagation model for a light field camera. Simulation accuracy is improved through subpixel segmentation and function fitting for vignetting, distortion, and noise. Simulation speed is also increased by parallelizing the computation of the subgraph array. Furthermore, a distribution function is used to fit the three-dimensional shape of deformation defects, and a modified Lambert diffuse reflectance model is used to fit the grayscale distribution of diffuse reflectance defects. Simulation speed is also increased by eliminating the surface intersection and ambient light integration processes. This results in a light field imaging simulation model for mirror-surface defect structures that achieves both high accuracy and high efficiency.

[0036] Other beneficial effects of the embodiments of the present invention will be further described below. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 This is the overall process of an imaging simulation method for a mirror defect detection system based on a structured light field according to a preferred embodiment of the present invention;

[0038] Figure 2 1 is a simulation image of the mirror defect illumination model before and after simplification according to a preferred embodiment of the present invention;

[0039] Figure 3 1. Comparison of real images (upper row) and simulated images (lower row) of different types of mirror defects in a preferred embodiment of the present invention;

[0040] Figure 4A and Figure 4B 1 is a comparison between the actual image (upper row) and the simulated image (lower row) of the pit defect under different system parameters of the preferred embodiment of the present invention. DETAILED DESCRIPTION

[0041] The following is a detailed description of the embodiments of the present invention. It should be emphasized that the following description is only exemplary and is not intended to limit the scope of the present invention and its application.

[0042] See Figure 1 The embodiment of the present invention provides an imaging simulation method for a mirror defect detection system based on a structured light field, comprising the following steps:

[0043] S1: calibrate system parameters as input for simulation;

[0044] S2: Using the calibrated system parameters, ray tracing is performed from the sensor. The incident light vector of the light field camera is solved by parallel computing methods to construct a light propagation model for the light field camera.

[0045] S3: Based on the light field camera light propagation model, a function is designed to fit the three-dimensional surface shape of the deformation defect. The model is used to solve the incident light vector at the defect, and these light rays are traced to determine the coordinates of the light source point and the light intensity, thereby accurately simulating the imaging of the deformation defect.

[0046] S4: Using the incident light information provided by the light propagation model of the light field camera, the grayscale value of the diffuse reflection defect is calculated using the Lambert diffuse reflection model. A coefficient representing the roughness of scratches or the opacity of stains is introduced to establish a grayscale model of diffuse reflection non-deformable defects, thereby accurately simulating the visual appearance of diffuse reflection defects under different lighting conditions;

[0047] S5: Based on the imaging simulation of the deformation defect in step S3 and the grayscale model of the diffuse reflection non-deformation defect established in step S4, a simulation image and a label are generated.

[0048] In some embodiments, step S1 specifically includes determining an encoding method of the structured light source, a light field camera architecture, and simulation model input parameters, and calibrating light field camera parameters.

[0049] In some embodiments, step S2 specifically includes calculating the one-to-one correspondence between the image plane sub-image array and the microlens array; ignoring the defocus situation and using the pinhole model to derive the light propagation model of the light field camera; through sub-pixel subdivision and average pooling, the edges of the structured light stripes are smoothed to improve the simulation accuracy; through function fitting of vignetting, distortion, noise and other phenomena, the simulation accuracy is further improved; through mask value superposition, the sub-image calculation is parallelized to improve the simulation speed.

[0050] In a preferred embodiment, step S2 specifically includes:

[0051] S21: Calculate the one-to-one correspondence between the image plane sub-image array and the microlens array;

[0052] S22: solving the diameter and spacing of the sub-image to generate a coordinate array of the array sub-image center and its corresponding microlens center;

[0053] S23: Taking the center of the image plane as the base point, perform layer-order traversal to generate a light field sub-image array until the sub-image has no intersection with the sensor;

[0054] S24: Ignoring the defocus phenomenon, the pinhole model is used to derive the light propagation model of the light field camera;

[0055] S25: Using the derived light propagation model and the generated coordinate array, the incident light vector of the light field camera is solved through parallel computing.

[0056] In a preferred embodiment, when deriving the light propagation model of the light field camera, a fitting function model is introduced, in which vignetting and distortion are represented by polynomials:

[0057]

[0058] Among them, r d is the distance between the pixel and the center of the sub-image, γ1, γ2, and γ3 are fitting coefficients, which are generally negative numbers. Vignetting fitting is to multiply the grayscale value by the above polynomial, and distortion fitting is to multiply the pixel coordinates by the above polynomial. The noise phenomenon is simulated using the Gaussian noise model.

[0059] In a preferred embodiment, in step S25, an image of the same size as the image plane is initialized for each sub-image. After the sub-images are calculated in parallel, the circular areas are extracted by multiplying each sub-image by a pre-calculated 0-1 mask matrix, and then all the results are superimposed into a complete light field image to achieve parallel calculation.

[0060] In some embodiments, step S3 specifically includes designing a function to fit the three-dimensional surface shape of the deformation defect; solving the incident light vector at the deformation defect, tracing the light to obtain the coordinates of the light source point and the light intensity; simplifying the calculation of the intersection point of the surface of the deformation defect to improve the simulation speed.

[0061] In a preferred embodiment, step S3 specifically includes:

[0062] S31: Using analytical function expressions to fit the three-dimensional shape of deformation defects, including using piecewise distribution functions to fit deformation defects in real-life images;

[0063] S32: Based on the fitted 3D surface shape, solve the incident light vector at the defect;

[0064] S33: trace the light to the light source point, and calculate the coordinates of the light source point and the light intensity;

[0065] S34: Directly trace the ray to the defect-free plane to calculate the intersection point; use a matrix method to parallelly calculate the intersection point of the ray and the plane.

[0066] In some embodiments, step S4 specifically includes describing diffuse reflection defects based on the Lambert diffuse reflection model, combining actual defect images, and introducing coefficients characterizing scratch roughness / stain opacity to improve the model; by fixing coefficients and adding random fluctuations, canceling the angle segmentation and ambient light integration process, the simulation speed is improved.

[0067] In a preferred embodiment, step S4 specifically includes:

[0068] S41: Based on the Lambert diffuse reflection model, a grayscale model of diffuse reflection non-deformation defects is established;

[0069] S42: A coefficient characterizing the scratch roughness or stain opacity is introduced, combining the ideal diffuse reflection grayscale and background grayscale.

[0070] More preferably, in step S42, the grayscale distribution at the diffuse reflection defect point p is characterized by the following formula:

[0071] I d (p)=λ(p)×I0(p)+(1-λ(p))×I n (p)

[0072] Among them, I0(p) is the grayscale distribution under ideal diffuse reflection, which is linearly related to L0(p). By introducing the coefficient distribution λ(p) that characterizes the roughness of scratches or the opacity of stains, the grayscale distribution I0 of the defect-free background stripe light source is converted to n (p) is taken into consideration, the larger the λ value, the greater the I d The less affected by the background grayscale;

[0073] More preferably, the amount of calculation is reduced by fixing I0(p) and λ(p) to coefficients I0 and λ, and the grayscale distribution exhibits certain random fluctuations by adding Gaussian random variables to the parameter values ​​of I0 and λ at different p.

[0074] In some embodiments, step S5 specifically includes generating labels for the simulation images according to the nature of the defect detection task; comparing the simulation images of various defect types and various system parameters with the actual images to evaluate the reliability of the simulation model; and evaluating the operating efficiency of the simulation model.

[0075] In a preferred embodiment, step S5 specifically includes:

[0076] S51: Generate a simulation image data set, and perform a difference between the defect image and the normal image to obtain a residual image;

[0077] S52: For the target detection task, traverse the pixels of the entire residual map and generate labels by maintaining the coordinate information in the residual map;

[0078] S53: For semantic segmentation tasks, the residual map is processed by closing operations and binarized to generate labels.

[0079] In some embodiments, the imaging simulation method for a mirror defect detection system based on a structured light field of the present invention includes a light field camera ray propagation model and a mirror defect illumination model. The light field camera ray propagation model includes the calculation of the correspondence between the light field sub-image array and microlenses, the derivation of the ray propagation model based on the pinhole model, image smoothing based on sub-pixel segmentation and average pooling, the simulation of phenomena such as vignetting, distortion, and noise based on function fitting, and the parallelization of sub-image array calculations based on mask value superposition. Mirror defect types include deformation defects and diffuse reflection defects. The deformation defect illumination model includes function fitting of the defect's three-dimensional surface shape, ray tracing at the defect and solving the light source point, and simplified surface intersection. The diffuse reflection defect illumination model includes the construction of a diffuse reflection grayscale model and a simplified ambient light integration process. This invention can be used to simulate imaging effects before system construction, assist in system parameter design, and save time and money in trial and error during system design. It can also be used to generate simulated images and automatically annotate data labels, expand defect datasets, and save labor costs in data collection and annotation.

[0080] Specific embodiments of the present invention are further described below.

[0081] The embodiment of the present invention provides an imaging simulation method for a structured light field mirror defect detection system based on geometric optics. Figure 1 FIG. 1 is an overall flow chart of an imaging simulation method for a mirror defect detection system based on a structured light field according to a preferred embodiment of the present invention, comprising the following steps:

[0082] S1: Calibrate system parameters as input to the simulation model. Specifically:

[0083] All input system parameters of the simulation model include system component parameters and system structural parameters. System component parameters include structural light source parameters (light source height H, light source width W, fringe period c), light field camera parameters (main lens focal length f, micro lens focal length f MLA , microlens diameter d, distance a between the microlens and the relay image plane, distance b between the microlens and the sensor image plane), sensor parameters (image plane height h, image plane width w, pixel size p), and workpiece parameters (workpiece width L). System structural parameters include angle parameters (angle α between the optical axis of the light field camera and the plane of the workpiece to be measured) and distance parameters (the distance m between the workpiece to be measured and the optical center on the optical axis of the light field camera, the distance n between the workpiece to be measured and the center of the light source on the perpendicular axis to the center of the light source, the distance v between the main lens of the light field camera and the relay image plane, and its corresponding object distance u). It is worth noting that in order to minimize the depth range of the imaging target, the mirror virtual image plane formed by the structured light source is perpendicular to the camera optical axis. At this time, the angle between the structured light source and the plane of the workpiece to be measured is the same as the angle between the optical axis of the light field camera and the plane of the workpiece to be measured, both of which are α.

[0084] The above parameters are generally provided by the manufacturer or obtained by measurement. Since the parameters a and b of the light field camera microlens array will inevitably produce certain errors during the assembly process, and small changes in the parameters will have a significant impact on the camera magnification, the main focus is on the calibration of these two parameters. There are many types of light field cameras, which can be divided into defocusing and focusing types according to the position of the MLA: when the distance b between the MLA and the sensor is equal to the focal length f of the MLA, the light field camera can be calibrated in the following way: MLA It is called a defocused light field camera. When b is not equal to f MLA When b>f MLA and b <f MLA When the relay image plane is located between the MLA and the primary lens and behind the sensor, they are respectively called Keplerian and Galilean types. This embodiment uses the Keplerian focusing light field camera as an example to introduce the imaging simulation model. The related derivation can be easily extended to other architectures.

[0085] The calibration of the a and b parameters requires finding the constraint relationship between the parameters. In a light field camera, the image point focused on the relay image plane is refocused on the image plane after passing through the microlens. According to Gauss's theorem, the a and b parameters as the object distance and image distance of the secondary imaging satisfy:

[0086]

[0087] This is the first constraint. The other constraint is provided by the imaging distance of the same object point under the sub-image formed by adjacent microlenses, assuming it is d p , according to similar triangles we can get:

[0088]

[0089] By measuring d p Solve the above two equations together to find a and b. p Distance can be measured by corner detection. Take a picture of an object with significant corners, such as a chessboard, and measure the average pixel distance of the same corner in adjacent sub-images. Multiply it by the pixel size to get d. p The value of .

[0090] S2: Derive the light propagation model of the light field camera, perform ray tracing from the sensor, parallelize the calculation of the light field sub-graph array, and solve the incident light vector of the light field camera. Specifically:

[0091] Specifically, in order to perform ray tracing from pixels, it is necessary to calculate the one-to-one correspondence between the image plane sub-image array and the microlens array. In theory, the coordinate position of the sub-image or microlens center can be calibrated through a variety of calibration methods. However, if it is necessary to simulate other set values ​​of parameters such as sensor size, microlens size, etc. that do not correspond to the actual object, it is impossible to obtain the coordinates of the sub-image center through direct calibration methods; on the other hand, compared with the assembly error in the axial direction of the microlens (parameters a and b), the position deviation in the plane direction of the microlens does not have a significant effect on the defect imaging effect. Therefore, this section calculates the average diameter and ideal position of the sub-image based on the macro parameters of the camera, and then generates a sub-image array. First, solve the sub-image diameter and spacing. Let the spacing between the sub-image centers be d s It is easy to prove that the center of the main lens, the center of the microlens, the center of the image on the relay surface and the center of the sub-image are collinear. According to similar triangles, we can get:

[0092]

[0093] In order to make full use of the pixels, the light field camera is usually adjusted to a suitable aperture value so that the sub-images are tangent to each other and the sub-image diameters and spacings are equal. s Generate the coordinate array of the center of the array sub-image and its corresponding microlens center, denoted as C s and C m In order to improve the pixel utilization rate of the image plane, the microlens array is generally arranged in a hexagonal shape. The initialization process of the light field sub-image array is as follows: Taking the center of the image plane as the base point, generate the first layer of sub-image, that is, the center sub-image; according to the microlens diameter d and the sub-image diameter d s , traverse outwards in layer order; calculate the position of each layer of microlens center and sub-image center and store them in array C s and C m The traversal is terminated when all sub-graphs and sensors of a certain layer have no intersection. Thus, the sub-graph to which each pixel belongs and the corresponding micro-lens are obtained.

[0094] Secondly, the light propagation model of the light field camera is derived. Since the defect detection system must focus the target object, the simulation model ignores the defocus phenomenon and directly adopts the pinhole model to reduce the computational complexity. The coordinates of the main lens optical center are denoted as O. For the i-th sub-image (with coordinates C si As the center of the circle, with d s The jth pixel in the circle with a diameter of sij , the center coordinate of the corresponding microlens is C mi , the coordinates of the corresponding relay image point are P rij , then the corresponding camera incident light is:

[0095]

[0096] Based on this, a light propagation model of the light field camera tracing from the image plane to the optical center of the camera was constructed.

[0097] To improve simulation accuracy, sub-pixel ray tracing is performed. The sub-pixel subdivision number is denoted as s. Each pixel is subdivided into an s×s grid, and ray tracing is performed on the sub-pixels after subdivision. Then, all the results within the pixel are averaged and pooled. This operation simulates the process of integrating light on the sensor image plane, achieving smoothing of the stripe edges in the defect image. One question is how many sub-pixels are more appropriate. Here is a method: Taking a smaller stripe period (rich in high-frequency components) as an example, the mean squared error (MSE) between the simulated image with different s and the simulated image with s=1 is calculated to represent the improvement in the accuracy of the sub-pixel simulation relative to the original pixel simulation. The optimal setting for s is: after further increasing s, the MSE value just stops increasing, and the sub-pixel subdivision accuracy has just reached a bottleneck.

[0098] To further improve simulation accuracy, we use function fitting to account for vignetting, distortion, and noise. These factors are related to the optical design of lenses and lens groups and are often not considered when designing system parameters. Therefore, the simulation model here does not accurately simulate the vignetting, distortion, and noise of a specific camera, but instead introduces a fitting function model. Vignetting and distortion are represented by polynomials:

[0099]

[0100] Among them, r d is the distance between the pixel and the sub-image center, and γ1, γ2, and γ3 are fitting coefficients, generally negative. The polynomial implies that the farther the pixel is from the sub-image center, the more severe the grayscale attenuation or pixel distortion. Vignetting fitting multiplies the grayscale value by the polynomial, while distortion fitting multiplies the pixel coordinates by the polynomial. Noise is simulated using a Gaussian noise model.

[0101] By superimposing mask values, sub-image computations can be parallelized, improving simulation speed. The traversal loop of the sub-image array modifies the same image matrix, making parallel computation impossible. To enable independent computation of each sub-image and thus parallelization, an image the size of the image plane is initialized for each sub-image. After parallel computation of the sub-images, each is multiplied by a pre-computed 0-1 mask matrix to extract the circular regions within it. All the results are then superimposed to form a complete light field image.

[0102] Mirror surface defects are divided into two types: abnormal surface shape, such as convex hulls and pits, and abnormal surface reflectivity, such as scratches and stains. Therefore, typical mirror surface defect types can be characterized by the presence of deformation that still exhibits specular reflection (hereinafter referred to as deformation defects), and the absence of deformation that exhibits diffuse reflection (hereinafter referred to as diffuse reflection defects). The modeling of these two types of defects is specifically described in S3 and S4, respectively.

[0103] S3: Design a function to fit the three-dimensional shape of the deformation defect, solve the incident light vector at the defect, and trace the light to obtain the coordinates of the light source point and the light intensity. Specifically:

[0104] The modeling of deformation defects mainly simulates their three-dimensional shape through analytical function expressions in order to calculate the surface normal vector and the incident light vector from the light source, and then solve the intersection of the light vector and the light source, and assign the grayscale of the pixel according to the light intensity at that point.

[0105] First, the three-dimensional shape of the deformation defect is fitted using a function analytical expression. For real defects collected in industry, three-dimensional measurement methods such as interference method and phase method can be used to restore their three-dimensional shape, and then different function systems such as polynomial function, distribution function, etc. are used for fitting. Here, the real-shot image of the deformation defect is fitted by a piecewise distribution function: in order to better simulate the center shape of the defect, a peak function showing the shape of a blunt object impact is designed based on the t distribution and Gaussian distribution, and then a Gaussian model is used as the tail function to connect and smoothly transition to a defect-free plane. It is worth noting that the piecewise function should satisfy the same first-order derivative and the same second-order derivative at the connection point to ensure the smoothness of the connection. The only difference in the positive and negative signs of the function is the simulation of pits and convex hull defects.

[0106] Secondly, solve the incident light vector at the defect, and trace the light to obtain the coordinates of the light source point and the light intensity. Based on the analytical representation of the three-dimensional surface shape of the above-mentioned deformation defect, the light vector calculated by the light field camera light propagation model can be traced to the three-dimensional surface of the deformation defect to solve the intersection of the light and the surface and the normal vector at the intersection. Then, according to the principle of mirror reflection, the light vector emitted by the light source is calculated, and the intersection of the light and the light source plane is further solved. Here, it is only necessary to calculate the coordinates of the light source point perpendicular to the stripe direction, and the light intensity can be calculated based on the period c. Different light source encoding methods correspond to different solution methods. For example, for square wave stripes, the light intensity is binarized, and for sinusoidal stripes, the light intensity is calculated using a sinusoidal distribution. At this point, ray tracing and pixel assignment are completed.

[0107] By simplifying the calculation of the intersection point of the deformation defect surface, the simulation speed is improved. The above-mentioned ray tracing process involves the process of finding the intersection point of the light ray and the surface function. For each ray, it is necessary to cyclically use the bisection method or the analytical method to find the intersection. Both methods are very time-consuming. In fact, the height difference between the deformation defect and the defect-free plane is very small. The tiny height difference will only cause a slight offset in the solved light source point, and the normal vector at the intersection is the key influencing factor of the light source point because it determines the direction of the light source light. Therefore, this height difference can be ignored, and the process of solving the intersection point of the light vector and the deformation defect three-dimensional surface fitting function can be cancelled. The ray can be directly traced to the defect-free plane to calculate the intersection point, thereby avoiding the solution process of the bisection or analytical method; at the same time, the process of finding the intersection with the plane can be realized through matrixing to achieve parallel calculation, eliminating the cyclic calculation of the light. From Figure 2 Comparing the simulation images of the pit and convex hull models before and after simplification reveals only minor changes in defect morphology, which has no impact on the simulation objectives of studying and optimizing system parameters and generating simulation datasets. Therefore, this simplification operation does not result in a loss of accuracy, while significantly improving simulation speed.

[0108] S4: Based on the Lambert diffuse reflection model, the coefficients representing scratch roughness / stain opacity are introduced to establish a grayscale model for diffuse reflection invisible deformation defects. Specifically:

[0109] The Lambert diffuse reflection model describes an ideal diffuse reflection surface. With the diffuse reflection point as the center, the angles within the surface hemisphere are divided and the incident light from various angles is tracked. The illumination intensity obtained from each incident light is their projection on the surface normal vector. At the same time, the illumination intensity of the light source at each position decays as the square of the distance. Finally, all incident light rays are integrated, and the radiant brightness in any reflection direction at the diffuse reflection point p can be expressed as:

[0110]

[0111] Among them, the first and second terms represent the integral of the radiance of the incident light from the structured light source and the ambient light source, respectively; ω i is the direction of the incident light; θ i is the zenith angle of the incident light, that is, the angle with the surface normal vector at point p; r i is the distance between the light source and point p; L I (r i ,ω i ), L E (r i ,ω i ) are the structured light source and the ambient light source in the direction of ω i , the distance from point p is r iThe radiant brightness distribution at point p is calculated based on the distance r. i Square attenuation; Ω I ,Ω E Contains the possible incident light of the structured light source and the ambient light source at point p respectively.

[0112] If the grayscale of the diffuse reflection defect is calculated according to the above formula, then the first and second terms of I0 have a low correlation with the position, that is, its grayscale distribution should be similar everywhere. However, from the actual image of the scratch, there is a significant difference in the grayscale value of the scratch when the background stripes are black and white, which shows that the mirror scratch is a relatively rough surface. When establishing the grayscale model, in addition to considering the ambient light, the background grayscale also needs to be considered. There are currently some microsurface models used to characterize the reflective properties of diffuse reflection surfaces, such as the GGX model. In order to simplify the decomposition and calculation of the microsurface, the following formula is established to characterize the grayscale distribution at the diffuse reflection defect point p:

[0113] I d (p)=λ(p)×I0(p)+(1-λ(p))×I n (p)

[0114] Among them, I0(p) is the grayscale distribution under ideal diffuse reflection, which is linearly related to L0(p). By introducing the coefficient distribution λ(p) that characterizes the roughness of scratches or the opacity of stains, the grayscale distribution I0 of the defect-free background stripe light source is converted to n (p) is taken into consideration, the larger the λ value, the greater the I d The less affected by the background grayscale.

[0115] By fixing the coefficients and adding random fluctuations, the process of angle division and ambient light integration is cancelled, thereby improving the simulation speed. The calculation of I0 mentioned above involves the angle division and light integration of the surface hemisphere for each incident point. Since the grayscale values ​​of the ambient light component in the defect area are relatively small, I0 is almost equal everywhere, and the grayscale difference in different areas has a low correlation with the influence of system parameters on the defect imaging quality. Therefore, I0 is set to a constant value and is directly simplified to a fixed coefficient representing the ambient light intensity. This avoids the process of angle division and light integration, greatly reducing the amount of calculation. Similarly, the λ parameter can also be simplified to a fixed coefficient. By adjusting the two coefficients I0 and λ, the imaging of scratches and stains in different situations can be simulated. In order to further enhance the authenticity, certain Gaussian random variables can also be added to the I0 and λ parameter values ​​at different positions in the defect area to make the grayscale distribution show certain random fluctuations. From Figure 2 Comparing the simulation images before and after simplification of the scratch and stain models, we can see that this simplification also does not result in loss of accuracy but improves the simulation speed.

[0116] S5: Generate simulated images and their labels. Compare the simulated images with the real images to evaluate the reliability of the simulation model. Evaluate the operating efficiency of the simulation model. Specifically:

[0117] First, generating a simulated image dataset involves label generation. The label format is related to the defect detection task: image classification tasks only require generating category labels; object detection tasks require generating box coordinates containing four-dimensional information and category labels; semantic segmentation tasks require generating binary images and category labels that match the image size.

[0118] For target detection tasks, labels can be calculated based on the coordinates of the defects. This only requires an affine transformation of the actual defect coordinates, which requires minimal computation. However, this method has the following disadvantages: on the one hand, its computational chain is long, resulting in large accumulated truncation errors during floating-point calculations and low accuracy; on the other hand, when used for semantic segmentation tasks, especially for narrow and long scratches, the impact of small errors can be catastrophic.

[0119] Therefore, this embodiment starts from the defect image itself. The specific method is: the defect image and the normal image are subtracted to obtain a residual map. For the target detection task, the pixels of the entire residual map are traversed, and the coordinate information of the four boxes of the leftmost, rightmost, topmost, and bottommost are continuously maintained, that is, the minimum and maximum values ​​of the horizontal and vertical coordinates of pixels with grayscale greater than 0 (or a certain threshold). For the semantic segmentation task, the residual map is first expanded and then corroded to fill the holes and smooth the edges, and then the image is binarized to obtain the label. Compared with the label calculation method based on defect coordinates, its disadvantage is that it needs to traverse the entire image and the amount of calculation is large, but it can ensure the accuracy of the generated label.

[0120] Secondly, the effect of the simulation model is evaluated by comparing it with the real images.

[0121] On the one hand, there are simulation effects of various defect types, such as Figure 3 The figure compares the real-life images (upper row) and simulated images (lower row) of typical deformation defects (pits and convex hulls) and diffuse reflection defects (scratches and stains). The simulation system parameters are consistent with the real-life system, as shown in the parameter group in Table 1.

[0122] Table 1 System parameter settings for comparing simulation results in a preferred embodiment of the present invention

[0123]

[0124]

[0125] Depend on Figure 3 It can be seen that the simulation results are consistent with the real images in many aspects, including:

[0126] 1) Position and diameter of the sub - figure;

[0127] 2) Vignetting phenomenon within each sub - figure;

[0128] 3) Imaging size of the defect;

[0129] 4) Lateral imaging size of the stripe;

[0130] 5) Longitudinal imaging size of the workpiece (reflected as the truncation boundary position of the stripe pattern);

[0131] 6) Curvature of the stripe deformation caused by pits and protrusions;

[0132] 7) Gray - scale distribution of scratches and stains under black - stripe background and white - stripe background.

[0133] This indicates that the simulation model is reliable for generating the simulation image dataset.

[0134] On the one hand, it is to evaluate the simulation effects of various system parameters, such as Figure 4A and Figure 4B shown. The key parameter settings of each group of system parameters in the figure are shown in Table 1. Among them, for the parameter groups m1 - m3, by controlling variables, the imaging of different parameters m (and the focusing object distance u also changes synchronously) is shown. The same applies to the parameters n, α, and c. Taking the pit defect as an example, it can be seen that in the simulation image and the actual - shot image:

[0135] 1) As m increases (m1 < m2 < m3), the imaging morphology of the defect is the same, the imaging size of the defect becomes smaller, the lateral imaging size of the stripe becomes smaller, and the longitudinal imaging size of the workpiece becomes smaller;

[0136] 2) As n increases (n1 < n2 < n3), the imaging morphology of the defect is the same, the imaging size of the defect becomes larger, the lateral imaging size of the stripe becomes smaller, and the longitudinal imaging size of the workpiece remains unchanged;

[0137] 3) As α increases (α1 < α2 < α3), the imaging morphology of the defect is the same, the longitudinal imaging size of the defect becomes smaller, the lateral imaging size of the stripe remains unchanged, and the longitudinal imaging size of the workpiece becomes smaller;

[0138] 4) As c increases (c1 < c2 < c3), the imaging morphology of the defect is the same, the longitudinal imaging size of the defect remains unchanged, the lateral imaging size of the stripe becomes larger, and the longitudinal imaging size of the workpiece remains unchanged.

[0139] Thus, it can be seen that when the system parameters change, the changing trends of the simulation image and the actual - shot image are the same, which indicates that the simulation model is effective for studying the influence of system parameters on defect imaging.

[0140] As described above, the effect of the simulation model has been verified by comparing it with the real images, and the operating efficiency of the simulation model has also been proven to be acceptable. Table 2 lists other system parameter settings of the preferred embodiment of the present invention.

[0141] Table 2 Other system parameter settings of the preferred embodiment of the present invention

[0142]

[0143] Finally, the simulation model's running efficiency was evaluated. For example, the simulation model in this embodiment runs at a speed of approximately 1 μs per ray on a single-core processor model AMD Ryzen 7 6800H and Radeon Graphics @ 3.20 GHz, meaning that simulating an image with a pixel size of 1000×1000 takes approximately 1 second.

[0144] In summary, the present invention proposes a high-precision and fast imaging simulation method for a mirror defect detection system based on a structured light field. Compared with the existing technology, the present invention has the following advantages:

[0145] The imaging simulation method of the present invention has the advantages of high precision and high efficiency, and solves the problem that the existing imaging simulation method is difficult to achieve both high precision and high efficiency.

[0146] The present invention is used to simulate imaging effects before system construction, assist in system parameter design, and effectively save time and money costs of trial and error in system design; through simulation, imaging effects under different parameter configurations can be predicted, which better helps to select the optimal system configuration.

[0147] To address the overfitting of deep learning detection algorithms caused by the scarcity of defect samples and small defect datasets in industry, this method generates simulated images and automatically labels them, expanding the defect dataset. This aids in the development and training of detection algorithms, particularly deep learning algorithms. The diverse defect images generated by this method enhance the generalization capabilities of the algorithm and reduce overfitting.

[0148] The optimized simulation method of the present invention greatly reduces the computational complexity of the simulation, improves the simulation efficiency, significantly shortens the system design cycle, effectively improves the system design, and quickly expands the trainable data set, thereby improving the system detection accuracy, which helps to efficiently reduce the quality fluctuations of industrial products and improve product consistency.

[0149] The imaging simulation method proposed in the present invention has great application potential and is suitable for mirror defect detection in various industrial scenarios. It helps to improve the reliability and practicality of automated detection systems and has high application value and good application prospects.

[0150] An embodiment of the present invention further provides a storage medium for storing a computer program, which at least performs the above method when executed.

[0151] An embodiment of the present invention further provides a control device, comprising a processor and a storage medium for storing a computer program; wherein the processor is configured to execute at least the method described above when executing the computer program.

[0152] An embodiment of the present invention further provides a processor, which executes a computer program and at least performs the method described above.

[0153] The storage medium can be implemented by any type of non-volatile storage device, or a combination thereof. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM); the magnetic surface memory can be a magnetic disk memory or a magnetic tape memory. The storage medium described in the embodiments of the present invention is intended to include, but is not limited to, these and any other suitable types of memory.

[0154] In the several embodiments provided by the present invention, it should be understood that the disclosed systems and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0155] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0156] In addition, all functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may be separately used as a unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0157] Those skilled in the art will appreciate that all or part of the steps of the above-mentioned method embodiments may be implemented by hardware associated with program instructions, and the aforementioned program may be stored in a computer-readable storage medium. When the program is executed, the program executes the steps of the above-mentioned method embodiments. The aforementioned storage medium includes various media that can store program codes, such as mobile storage devices, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical disks.

[0158] Alternatively, if the above-mentioned integrated unit of the present invention is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROM, RAM, magnetic disks or optical disks.

[0159] The methods disclosed in the several method embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new method embodiments.

[0160] The features disclosed in several product embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new product embodiments.

[0161] The features disclosed in several method or device embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0162] The above is a further detailed description of the present invention in conjunction with specific preferred embodiments, and the specific implementation of the present invention should not be considered to be limited to these descriptions. Those skilled in the art will recognize that, without departing from the scope of the present invention, several equivalent substitutions or obvious variations can be made, and the performance or use of the same should be considered to fall within the scope of protection of the present invention.

Claims

1. An imaging simulation method for a mirror defect detection system based on structured light field, characterized in that: The steps include: S1: calibrate system parameters as input for simulation; S2: Using the calibrated system parameters, ray tracing is performed from the sensor. The incident light vector of the light field camera is solved by parallel computing methods to construct a light propagation model for the light field camera. Step S2 specifically includes: S21: Calculate the one-to-one correspondence between the image plane sub-image array and the microlens array; S22: Solve the diameter and spacing of the sub-image to generate the coordinate array of the array sub-image center and its corresponding microlens center; S23: Taking the center of the image plane as the base point, perform layer-order traversal to generate a light field sub-image array until the sub-image has no intersection with the sensor; S24: Ignoring the defocus phenomenon, the pinhole model is used to derive the light propagation model of the light field camera; S25: Using the derived light propagation model and the generated coordinate array, the incident light vector of the light field camera is solved through parallel calculation; S3: Based on the light propagation model of the light field camera, a function is designed to fit the three-dimensional surface shape of the deformation defect. The model is used to solve the incident light vector at the defect, and the light is traced to determine the coordinates of the light source point and the light intensity, thereby performing imaging simulation of the deformation defect; Step S3 specifically includes: S31: Using analytical function expressions to fit the three-dimensional shape of deformation defects, including using piecewise distribution functions to fit deformation defects in real-life images; S32: Based on the fitted 3D surface shape, solve the incident light vector at the defect; S33: trace the light to the light source point, and calculate the coordinates of the light source point and the light intensity; S34: Directly trace the ray to the defect-free plane to calculate the intersection point; use matrix method to parallelly calculate the intersection point of the ray and the plane; S4: Using the incident light information provided by the light propagation model of the light field camera, the grayscale value of the diffuse reflection defect is calculated using the Lambert diffuse reflection model, and a coefficient representing the roughness of the scratch or the opacity of the stain is introduced to establish a grayscale model of the diffuse reflection non-deformable defect, thereby simulating the visual appearance of the diffuse reflection defect under different lighting conditions; S5: Based on the imaging simulation of the deformation defect in step S3 and the grayscale model of the diffuse reflection non-deformation defect established in step S4, a simulation image and a label are generated.

2. The imaging simulation method for a mirror defect detection system based on structured light field according to claim 1, characterized in that: Step S1 specifically includes determining the encoding method of the structured light source, the light field camera architecture, and the simulation model input parameters, and calibrating the light field camera parameters.

3. The imaging simulation method for a mirror defect detection system based on a structured light field according to claim 1 or 2, characterized in that: When deriving the light propagation model of the light field camera, a fitting function model is introduced, in which vignetting and distortion are represented by polynomials: ； in, is the distance between the pixel and the center of the sub-image, is the fitting coefficient, which is generally a negative number. Vignetting fitting is to multiply the grayscale value by the above polynomial, and distortion fitting is to multiply the pixel coordinates by the above polynomial. The noise phenomenon is simulated using the Gaussian noise model.

4. The imaging simulation method for a mirror defect detection system based on a structured light field according to claim 1 or 2, wherein: In step S25, an image of the same size as the image plane is initialized for each sub-image. After the sub-images are calculated in parallel, the circular areas are extracted by multiplying each sub-image by a pre-calculated 0-1 mask matrix, and then all the results are superimposed into a complete light field image to achieve parallel calculation.

5. The imaging simulation method of a mirror defect detection system based on structured light field according to any one of claims 1 to 2, characterized in that: Step S4 specifically includes: S41: Based on the Lambert diffuse reflection model, a grayscale model of diffuse reflection non-deformation defects is established; S42: A coefficient characterizing the scratch roughness or stain opacity is introduced, combining the ideal diffuse reflection grayscale and background grayscale.

6. The imaging simulation method of a mirror defect detection system based on structured light field according to claim 5, characterized in that: In step S42, the diffuse reflection defect point is characterized by the following formula: Grayscale distribution at: ； in, is the grayscale distribution under ideal diffuse reflection, and the radiation brightness Linear dependence, by introducing a coefficient distribution that characterizes the roughness of scratches or the opacity of stains , the grayscale distribution of the defect-free background stripe light source Take into consideration, The larger the value, the The less affected by the background grayscale.

7. The imaging simulation method for a mirror defect detection system based on structured light field according to claim 6, characterized in that: In step S42, by and Fixed as coefficient and , so that the amount of calculation is reduced, and by and The parameter value adds Gaussian random variables to make the grayscale distribution show a certain random fluctuation.

8. The imaging simulation method of a mirror defect detection system based on structured light field according to any one of claims 1 to 2, characterized in that: Step S5 specifically includes: S51: generating a simulation image data set, performing a difference between the defect image and the normal image to obtain a residual image; S52: For the target detection task, traverse the pixels of the entire residual map and generate labels by maintaining the coordinate information in the residual map; S53: For semantic segmentation tasks, the residual map is processed by closing operations and binarized to generate labels.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the imaging simulation method of a mirror defect detection system based on a structured light field as described in any one of claims 1 to 8 is implemented.

Images