Mirror surface defect detection system imaging simulation method based on structured light field

Abstract

A mirror surface defect detection system imaging simulation method based on a structured light field comprises the steps that S1, system parameters are calibrated and serve as simulation model input; and S2, deducing a light propagation model of the light field camera, performing light tracing from a sensor, performing parallel calculation on the light field sub-graph array, and solving an incident light vector of the light field camera. And S3, designing a three-dimensional surface shape of a function fitting deformation defect, solving an incident light vector at the defect, and tracing light to obtain a light source point coordinate and illumination intensity. And S4, based on a Lambert diffuse reflection model, introducing a coefficient representing scratch roughness / stain opacity, and establishing a diffuse reflection non-deformation defect gray model. And S5, generating a simulation image and a label thereof. And comparing the simulation image with the real shot image, and evaluating the reliability of the simulation model. And evaluating the operation efficiency of the simulation model. According to the method, the precision and speed of mirror surface defect light field detection imaging simulation are effectively improved, and the method can be applied to parameter optimization, simulation image data set generation and the like before construction of a detection system.

Description

Technical Field

[0001] The present invention relates to the field of mirror defect detection, and in particular to an imaging simulation method of a mirror defect detection system based on a structured light field. Background Art

[0002] Metal workpieces are often treated with processes such as electroplating and polishing to improve corrosion resistance, wear resistance and glossiness. The resulting high-reflectivity surface is called a mirror surface. Defects such as bruises and scratches on the mirror surface not only affect the appearance, but also damage the mechanical properties of the workpiece, posing a safety hazard. Traditional defect detection relies on high-intensity, repetitive manual visual labor, which is not only detrimental to personnel health, but also has problems of low efficiency and low precision. Therefore, the method of realizing automatic detection based on machine vision systems has always attracted much attention.

[0003] Compared with the defect detection of ordinary diffuse reflection surfaces, the automatic detection system of mirror surface defects has more stringent requirements on lighting and imaging conditions.

[0004] In terms of lighting, due to the highly reflective characteristics of the mirror surface and the different reflective characteristics of different mirror defects, the traditional uniform lighting method is prone to overexposure or underexposure. There are already some related methods for lighting enhancement. The dark field scattering method uses the abnormal reflective characteristics of the defect to form a contrast with the background, and it is difficult to capture defect information with normal reflective characteristics; the photometric stereo method uses the deformation characteristics of the defect to distinguish it from the background, and is not sensitive to non-deformation defect information; the structured light imaging method presents different defects through different imaging depths: the deformation defects that still belong to mirror reflection are observed through the distortion of the virtual image formed by the stripes, while the non-deformation defects with diffuse reflection on the surface are formed by the integral of the ambient light at the diffuse reflection and the background without defects. The simultaneous imaging of the two puts forward requirements for the depth of field capability of the imaging equipment.

[0005] In terms of imaging, in order to increase the exposure of defects, there are some imaging enhancement methods, such as time domain multi-exposure and multi-camera exposure methods. The former sacrifices time complexity in exchange for images of different exposure scenes, and the latter is very sensitive to noise interference such as vibration for the joint calibration results of multiple cameras. The light field camera records four-dimensional light field information by installing a microlens array (MLA) between the ordinary camera sensor and the main lens. The same target can be imaged from multiple perspectives in a single exposure, and the depth of field of the camera can be extended without changing the aperture.

[0006] Therefore, the inspection system using structured light as the lighting method and light field camera as the imaging device has the ability to quickly and reliably image mirror defects.

[0007] The machine vision system mainly includes hardware system and software system. The former captures defect images and focuses on the imaging effect of defects; the latter processes defect images and focuses on the detection effect of defects.

[0008] The design of a hardware system usually involves system construction, defect image acquisition, and imaging effect evaluation, and adjusting system parameters based on experience and repeating the above steps. The trial and error process is somewhat blind, resulting in a large time cost; at the same time, the repeated replacement of system components also results in a large monetary cost. Imaging simulation of machine vision systems greatly simplifies the parameter design process before system construction, saving time and money.

[0009] The design of software systems usually involves the collection of defective workpieces, the collection and annotation of defect image samples, and then the precision debugging of the detection algorithm based on the constructed defect image dataset. In actual production lines, the number of defective workpieces is very small, resulting in a small defect image dataset, which further leads to insufficient generalization performance of the detection algorithm. Imaging simulation of defects can automatically generate a large number of datasets by changing defect parameters and realize automatic annotation, which can expand the defect dataset without increasing labor costs.

[0010] Therefore, accurate and fast imaging simulation models play an important role in system parameter design and defect data set expansion.

[0011] Existing imaging simulation technologies for machine vision defect detection systems include data-driven methods, wave optics methods, and geometric optics methods.

[0012] The data-driven method uses neural networks, usually generative adversarial networks (GAN) for training, to fit the imaging results of a specific system. It can achieve detailed simulation of imaging, including vignetting, aberration, distortion, and noise of cameras and sensors, and is suitable for black box situations such as unknown optical designs.

[0013] When the optical design is known, optical design software such as CODE V and Zemax can be used for imaging simulation. The software is generally based on the principles of wave optics and geometric optics.

[0014] The wave optics method can simulate the diffraction effect by analyzing the light wave propagation model in different media and deriving the point spread function (PSF) to describe the response of the imaging system to a point source. Therefore, it is necessary to calculate the imaging response of the hologram to each point source.

[0015] The method of geometric optics traces the path of light rays starting from the image plane, and recursively solves the incident light rays reflected or refracted on the surface of the object, and finally solves the light source point and its light intensity corresponding to the image plane pixel. Ray tracing considering nonlinear processes such as distortion and aberration involves a huge amount of calculation. Paraxial ray tracing using Taylor expansion and first-order approximation greatly simplifies the calculation, but loses the accuracy of distortion and aberration.

[0016] The above imaging simulation technology has the following problems when applied to defect detection systems:

[0017] (1) The data-driven method is based on a large number of real system images, which makes it impossible to apply to imaging simulation before the system is built. In addition, due to the small number of defect samples in actual production and the limited size of defect data sets, it is difficult to train defective system imaging even for a system that has been actually built. In addition, this method is essentially a black-box simulation of the imaging system using a nonlinear neural network. Once the system parameters are changed, the trained model needs to be repeatedly fine-tuned or even retrained.

[0018] (2) The wave optics method requires calculating the imaging response of each image point in the image space to each object point in the object space and superimposing them. The computational complexity is high, and when used to assist in the optimization design of system parameters, it will lead to a long design cycle.

[0019] (3) The geometric optics method can significantly reduce the amount of ray tracing calculations through paraxial approximation, and the simulation speed is faster, but it loses the accuracy of distortion and aberration related to camera optical design.

[0020] (4) Currently, there is a lack of complete imaging simulation models for defect detection systems, especially for defect modeling. Related research on imaging simulation focuses more on the simulation of imaging systems.

[0021] In summary, for the mirror defect detection system using structured light and light field cameras, it is necessary to improve the imaging simulation method and establish a complete imaging simulation model for optimizing the accuracy and speed of mirror defect detection.

[0022] It should be noted that the information disclosed in the above background technology section is only used for understanding the background of the present application, and therefore may include information that does not constitute prior art known to ordinary technicians in the field. Summary of the invention

[0023] The main purpose of the present invention is to overcome the defects of the above-mentioned background technology and provide an imaging simulation method of a mirror defect detection system based on a structured light field.

[0024] To achieve the above object, the present invention adopts the following technical solutions:

[0025] An imaging simulation method for a mirror defect detection system based on a structured light field comprises the following steps:

[0026] S1: calibrate system parameters as input for simulation;

[0027] S2: Using the calibrated system parameters, ray tracing is performed from the sensor, and the incident light vector of the light field camera is solved by parallel computing methods, thereby constructing a light propagation model of the light field camera;

[0028] S3: Based on the light propagation model of the light field camera, a function is designed to fit the three-dimensional surface shape of the deformation defect, the model is used to solve the incident light vector at the defect, and the light is traced to determine the coordinates of the light source point and the light intensity, so as to perform imaging simulation on the deformation defect;

[0029] S4: Utilizing the light incident information provided by the light propagation model of the light field camera, the grayscale value of the diffuse reflection defect is calculated through the Lambert diffuse reflection model, and a coefficient representing the scratch roughness or the opacity of the stain is introduced to establish a grayscale model of the diffuse reflection non-deformable defect, thereby simulating the visual performance of the diffuse reflection defect under different lighting conditions;

[0030] S5: Generate a simulation image and a label based on the imaging simulation of the deformation defect performed in step S3 and the grayscale model of the diffuse reflection non-deformation defect established in step S4.

[0031] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the imaging simulation method of a mirror defect detection system based on a structured light field is implemented.

[0032] The present invention has the following beneficial effects:

[0033] The present invention proposes an imaging simulation method for a mirror defect detection system based on a structured light field, and proposes the construction of a geometric optical imaging simulation model for a mirror defect detection system based on a structured light field, which solves the problem that there is a lack of imaging simulation methods for mirror defects under a structured light field, and that existing imaging simulation methods are difficult to balance high precision and high efficiency, thereby improving the accuracy and speed of imaging simulation for mirror defect detection, and can be used for parameter optimization before the detection system is built, generating simulated image data sets, and other applications. Specifically, the present invention can be used to simulate imaging effects before the system is built, assist in system parameter design, and save time and money costs for trial and error in system design; it can also be used to generate simulated images and automatically annotate data labels, expand defect data sets, and save the labor costs of data collection and annotation.

[0034] Compared with the prior art, the main advantages of the embodiments of the present invention are as follows:

[0035] The present invention uses a more simplified pinhole camera model to derive the light propagation model of the light field camera, and improves the simulation accuracy through sub-pixel subdivision and function fitting of vignetting, distortion, and noise phenomena, and improves the simulation speed by parallelizing the calculation of the sub-image array; secondly, the distribution function is used to fit the three-dimensional surface shape of the deformation defect, and the improved Lambert diffuse reflection model is used to fit the grayscale distribution of the diffuse reflection defect, and the simulation speed is improved by eliminating the process of surface intersection and ambient light integration. In this way, a light field imaging simulation model of a mirror defect structure that takes into account both high precision and high efficiency is realized.

[0036] Other beneficial effects of the embodiments of the present invention will be further described below. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 This is the overall process of an imaging simulation method for a mirror defect detection system based on a structured light field according to a preferred embodiment of the present invention;

[0038] Figure 2 It is a simulation image before and after simplification of the mirror defect illumination model of the preferred embodiment of the present invention;

[0039] Figure 3 1. The figure is a comparison between the real image (upper row) and the simulated image (lower row) of different types of mirror surface defects in the preferred embodiment of the present invention;

[0040] Figure 4A and Figure 4B It is a comparison between the actual image (upper row) and the simulated image (lower row) of the pit defect of the preferred embodiment of the present invention under different system parameters. DETAILED DESCRIPTION

[0041] The following is a detailed description of the embodiments of the present invention. It should be emphasized that the following description is only exemplary and is not intended to limit the scope and application of the present invention.

[0042] See also Figure 1 The embodiment of the present invention provides an imaging simulation method of a mirror defect detection system based on a structured light field, comprising the following steps:

[0043] S1: calibrate system parameters as input for simulation;

[0044] S2: Using the calibrated system parameters, ray tracing is performed from the sensor, and the incident light vector of the light field camera is solved by parallel computing methods, thereby constructing a light propagation model of the light field camera;

[0045] S3: Based on the light propagation model of the light field camera, a function is designed to fit the three-dimensional surface shape of the deformation defect, the model is used to solve the incident light vector at the defect, and these light rays are traced to determine the coordinates of the light source point and the light intensity, so as to accurately simulate the imaging of the deformation defect;

[0046] S4: Utilizing the light incident information provided by the light propagation model of the light field camera, the grayscale value of the diffuse reflection defect is calculated through the Lambert diffuse reflection model, and a coefficient characterizing the scratch roughness or the opacity of the stain is introduced to establish a grayscale model of the diffuse reflection non-deformable defect, thereby accurately simulating the visual performance of the diffuse reflection defect under different lighting conditions;

[0047] S5: Generate a simulation image and a label based on the imaging simulation of the deformation defect performed in step S3 and the grayscale model of the diffuse reflection non-deformation defect established in step S4.

[0048] In some embodiments, step S1 specifically includes determining a coding method of the structured light source, a light field camera architecture, and simulation model input parameters, and calibrating light field camera parameters.

[0049] In some embodiments, step S2 specifically includes calculating the one-to-one correspondence between the image plane sub-image array and the microlens array; ignoring the defocus situation and using the pinhole model to derive the light propagation model of the light field camera; through sub-pixel subdivision and average pooling, the edges of the structured light stripes are smoothed to improve the simulation accuracy; through function fitting of vignetting, distortion, noise and other phenomena, the simulation accuracy is further improved; through mask value superposition, the sub-image calculation is parallelized to improve the simulation speed.

[0050] In a preferred embodiment, step S2 specifically includes:

[0051] S21: Calculate a one-to-one correspondence between the image plane sub-image array and the microlens array;

[0052] S22: solving the diameter and spacing of the sub-image to generate a coordinate array of the center of the array sub-image and the center of its corresponding microlens;

[0053] S23: Taking the center of the image plane as the base point, perform layer-order traversal to generate a light field sub-image array until the sub-image has no intersection with the sensor;

[0054] S24: Ignoring the defocus phenomenon, the pinhole model is used to derive the light propagation model of the light field camera;

[0055] S25: Using the derived light propagation model and the generated coordinate array, the incident light vector of the light field camera is solved through parallel calculation.

[0056] In a preferred embodiment, when deriving the light propagation model of the light field camera, a fitting function model is introduced, in which vignetting and distortion are represented by polynomials:

[0057]

[0058] Among them, r d is the distance between the pixel and the center of the sub-image, γ1, γ2, and γ3 are fitting coefficients, which are generally negative numbers. The vignetting fitting is to multiply the grayscale value by the above polynomial, and the distortion fitting is to multiply the pixel coordinates by the above polynomial. The noise phenomenon is simulated using the Gaussian noise model.

[0059] In a preferred embodiment, in step S25, an image of the same size as the image plane is initialized for each sub-image. After the sub-images are calculated in parallel, the circular areas are extracted by multiplying each sub-image by a pre-calculated 0-1 mask matrix, and then all the results are superimposed into a complete light field image to achieve parallel calculation.

[0060] In some embodiments, step S3 specifically includes designing a function to fit the three-dimensional surface shape of the deformation defect; solving the incident light vector at the deformation defect, tracing the light to obtain the coordinates of the light source point and the light intensity; simplifying the calculation of the intersection point of the surface of the deformation defect to improve the simulation speed.

[0061] In a preferred embodiment, step S3 specifically includes:

[0062] S31: fitting the three-dimensional surface shape of the deformation defect using a function analytical expression, including fitting the deformation defect in the real shot image using a piecewise distribution function;

[0063] S32: Based on the fitted three-dimensional surface shape, solve the incident light vector at the defect;

[0064] S33: trace the light to the light source point, and calculate the coordinates of the light source point and the light intensity;

[0065] S34: directly trace the light to the defect-free plane to calculate the intersection point; use the matrix method to parallelly calculate the intersection point of the light and the plane.

[0066] In some embodiments, step S4 specifically includes describing diffuse reflection defects based on the Lambert diffuse reflection model, combining actual images of defects, and introducing coefficients that characterize scratch roughness / stain opacity to improve the model; by fixing coefficients and adding random fluctuations, canceling the angle division and ambient light integration process, the simulation speed is improved.

[0067] In a preferred embodiment, step S4 specifically includes:

[0068] S41: Based on Lambert diffuse reflection model, a grayscale model of diffuse reflection non-deformation defects is established;

[0069] S42: A coefficient characterizing the roughness of scratches or the opacity of stains is introduced, combining the ideal diffuse reflection grayscale and the background grayscale.

[0070] More preferably, in step S42, the grayscale distribution at the diffuse reflection defect point p is characterized by the following formula:

[0071] I d (p)=λ(p)×I0(p)+(1-λ(p))×I n (p)

[0072] Among them, I0(p) is the grayscale distribution under ideal diffuse reflection, which is linearly related to L0(p). By introducing the coefficient distribution λ(p) that characterizes the roughness of scratches or the opacity of stains, the grayscale distribution I0(p) of the defect-free background stripe light source is converted to n (p) is taken into consideration. The larger the λ value, the greater the I d The less affected by the background grayscale;

[0073] More preferably, the amount of calculation is reduced by fixing I0(p) and λ(p) to coefficients I0 and λ, and the grayscale distribution exhibits certain random fluctuations by adding Gaussian random variables to the I0 and λ parameter values ​​at different p.

[0074] In some embodiments, step S5 specifically includes generating labels for simulation images according to the nature of the defect detection task; comparing the simulation images of various defect types and various system parameters with the actual images to evaluate the reliability of the simulation model; and evaluating the operating efficiency of the simulation model.

[0075] In a preferred embodiment, step S5 specifically includes:

[0076] S51: Generate a simulation image data set, and perform a difference between the defect image and the normal image to obtain a residual image;

[0077] S52: For the target detection task, traverse the pixels of the entire residual map and generate labels by maintaining the coordinate information in the residual map;

[0078] S53: For the semantic segmentation task, the residual map is processed by closing operation and binarization to generate labels.

[0079] In some embodiments, the imaging simulation method of the mirror defect detection system based on the structured light field of the present invention includes a light field camera light propagation model and a mirror defect illumination model. The light field camera light propagation model includes the calculation of the correspondence between the light field sub-image array and the microlens, the derivation of the light propagation model based on the pinhole model, the image smoothing based on sub-pixel subdivision and average pooling, the simulation of vignetting, distortion, noise and other phenomena based on function fitting, and the parallelization of the sub-image array calculation based on the mask value superposition. The mirror defect types include deformation defects and diffuse reflection defects. The deformation defect illumination model includes the function fitting of the defect three-dimensional surface shape, the ray tracing at the defect and the solution of the light source point, and the simplification of the surface intersection process. The diffuse reflection defect illumination model includes the construction of the diffuse reflection grayscale model and the simplification of the ambient light integration process. The present invention can be used to simulate the imaging effect before the system is built, assist the system parameter design, and save the time and money cost of system design trial and error; it is also used to generate simulated images and automatically annotate data labels, expand the defect data set, and save the labor cost of data collection and annotation.

[0080] The specific embodiments of the present invention are further described below.

[0081] The embodiment of the present invention provides an imaging simulation method for a structured light field mirror defect detection system based on geometric optics. Figure 1 FIG. 1 is an overall flow chart of an imaging simulation method for a mirror defect detection system based on a structured light field according to a preferred embodiment of the present invention, comprising the following steps:

[0082] S1: Calibrate system parameters as input to the simulation model. Specifically:

[0083] All input system parameters of the simulation model include system component parameters and system structural parameters. System component parameters include structural light source parameters (light source height H, light source width W, fringe period c), light field camera parameters (main lens focal length f, micro lens focal length f MLA , microlens diameter d, distance a between microlens and relay image plane, distance b between microlens and sensor image plane), sensor parameters (image plane height h, image plane width w, pixel size p), and workpiece parameters (workpiece width L). The structural parameters of the system include angle parameters (angle α between the optical axis of the light field camera and the plane of the workpiece to be measured) and distance parameters (the distance m between the workpiece to be measured and the optical center on the optical axis of the light field camera, the distance n between the workpiece to be measured and the center of the light source on the vertical axis of the light source center, the distance v between the main lens of the light field camera and the relay image plane and its corresponding object distance u). It is worth noting that in order to minimize the depth range of the imaging target, the mirror virtual image plane formed by the structured light source is perpendicular to the camera optical axis. At this time, the angle between the structured light source and the plane of the workpiece to be measured is the same as the angle between the optical axis of the light field camera and the plane of the workpiece to be measured, both of which are α.

[0084] The above parameters are generally provided by the manufacturer or obtained by measurement. Since the a and b parameters of the light field camera microlens array will inevitably produce certain errors during the assembly process, and a small change in the parameters will have a significant impact on the camera magnification, the main focus is on the calibration of these two parameters. There are many types of light field cameras, which can be divided into defocusing and focusing types according to the position of the MLA: when the distance b between the MLA and the sensor is equal to the focal length f of the MLA MLA It is called a defocused light field camera. When b is not equal to f MLA When b>f MLA and b <f MLA When the relay image plane is located between the MLA and the main lens and behind the sensor, they are called Keplerian type and Galilean type respectively. This embodiment will take the Keplerian type focusing light field camera as an example to introduce the imaging simulation model, and the related derivation can be easily extended to other architectures.

[0085] The calibration of a and b parameters requires finding the constraint relationship between the parameters. In a light field camera, the image point focused on the relay image plane is refocused on the image plane after passing through the microlens. According to Gauss's theorem, a and b parameters as the object distance and image distance of the secondary imaging satisfy:

[0086]

[0087] This is the first constraint. The other constraint is provided by the imaging distance of the same object point in the sub-image formed by adjacent microlenses, which is assumed to be d p , according to similar triangles:

[0088]

[0089] By measuring d p By combining the above two equations, we can solve a and b. p Distance measurement can be performed by corner detection. Take pictures of objects with significant corners, such as a chessboard, measure the average pixel distance of the same corner in adjacent sub-images, and multiply it by the pixel size to get d. p The value of .

[0090] S2: Derive the light propagation model of the light field camera, perform ray tracing from the sensor, parallelize the calculation of the light field sub-graph array, and solve the incident light vector of the light field camera. Specifically:

[0091] Specifically, in order to perform ray tracing from pixels, it is necessary to calculate the one-to-one correspondence between the image plane sub-image array and the microlens array. In theory, the coordinate position of the sub-image or microlens center can be calibrated through a variety of calibration methods. However, if it is necessary to simulate other set values ​​of parameters such as sensor size and microlens size that do not correspond to the actual object, it is impossible to obtain the coordinates of the sub-image center through direct calibration methods; on the other hand, compared with the assembly error in the axial direction of the microlens (parameters a and b), the position deviation in the plane direction of the microlens does not have a significant effect on the defect imaging effect. Therefore, this section calculates the average diameter and ideal position of the sub-image based on the camera macro parameters, and then generates a sub-image array. First, solve the sub-image diameter and spacing. Let the spacing of the sub-image center be d s It is easy to prove that the center of the main lens, the center of the microlens, the center of the image on the relay surface and the center of the sub-image are collinear. According to similar triangles:

[0092]

[0093] In order to make full use of the pixels, the light field camera is usually adjusted to a suitable aperture value so that the sub-images are tangent to each other and the sub-image diameters and spacings are equal. s Generate the coordinate array of the center of the array sub-image and its corresponding microlens center, denoted as C s and C m In order to improve the pixel utilization rate of the image plane, the microlens array is generally arranged in a hexagonal shape. The initialization process of the light field sub-image array is as follows: Taking the center of the image plane as the base point, generate the first layer of sub-image, that is, the central sub-image; according to the microlens diameter d and the sub-image diameter d s , traverse outward in layer order; calculate the position of each layer of microlens center and sub-image center and store them in array C s and C m ; When all sub-images and sensors of a certain layer have no intersection, the traversal is terminated. Thus, the sub-image to which each pixel belongs and the corresponding micro-lens are obtained.

[0094] Secondly, the light propagation model of the light field camera is derived. Since the defect detection system will inevitably focus the target object, the simulation model ignores the defocus phenomenon and directly uses the pinhole model to reduce the computational complexity. Let the coordinate of the optical center of the main lens be O. For the i-th sub-image (with coordinate C si As the center of the circle, with d s The jth pixel in a circle with a diameter of sij , the corresponding center coordinate of the microlens is C mi , the coordinates of the corresponding relay image point are P rij , then the corresponding camera incident light is:

[0095]

[0096] Based on this, a light propagation model of the light field camera tracing from the image plane to the optical center of the camera was constructed.

[0097] In order to improve the simulation accuracy, sub-pixel level ray tracing is performed, and the number of sub-pixel divisions is recorded as s. Each pixel is divided into a grid of size s×s, and the sub-pixels after division are subjected to ray tracing, and then all the results in the pixel are averaged and pooled. This operation is equivalent to the process of simulating the light integration on the image plane of the sensor, and the smoothing of the stripe edges in the defective image is achieved. One question is how many sub-pixels are more reasonable to set. Here is a method: taking a smaller stripe period as an example (rich in high-frequency components), the mean square error of the simulation image with different s and the simulation image with s=1 is solved to represent the improvement of the accuracy of the sub-pixel simulation relative to the original pixel simulation. The optimal setting value of s is: after s continues to increase, the mean square error value just stops increasing, and the sub-pixel division accuracy just reaches the bottleneck.

[0098] In order to further improve the simulation accuracy, vignetting, distortion, noise and other phenomena are fitted by function. Factors such as vignetting, distortion, and noise are related to the optical design of lenses and lens groups, and are often not considered when designing system parameters. Therefore, the simulation model here does not accurately simulate the vignetting, distortion, and noise of a specific camera, but introduces a fitting function model. Vignetting and distortion are represented by polynomials:

[0099]

[0100] Among them, r d is the distance between the pixel and the center of the sub-image, γ1, γ2, and γ3 are fitting coefficients, which are generally negative. The meaning of the polynomial is that the farther the pixel is from the center of the sub-image, the more serious the grayscale attenuation or pixel distortion. The vignetting fitting is to multiply the grayscale value by the above polynomial, and the distortion fitting is to multiply the pixel coordinates by the above polynomial. The noise phenomenon is simulated using the Gaussian noise model.

[0101] By superimposing mask values, the sub-image calculation can be parallelized to improve the simulation speed. The traversal loop of the sub-image array modifies the same image matrix and cannot be directly parallelized. In order to make the calculation of each sub-image independent and thus realize parallelization, an image of the same size as the image plane is initialized for each sub-image. After the sub-images are calculated in parallel, they are multiplied by the pre-calculated 0-1 mask matrix to extract the circular area, and then all the results are superimposed into a complete light field image.

[0102] Mirror surface defects are divided into two types of anomalies: one is the abnormal shape of the surface, such as convex hulls and pits, and the other is the abnormal reflection characteristics of the surface, such as scratches and stains. Therefore, the typical mirror defect types can be characterized by the type of surface deformation but still mirror reflection (hereinafter referred to as deformation defects) and the type of surface no deformation but diffuse reflection characteristics (hereinafter referred to as diffuse reflection defects). The modeling of the two types of defects is specifically given by S3 and S4 respectively.

[0103] S3: Design a function to fit the three-dimensional shape of the deformation defect, solve the incident light vector at the defect, and trace the light to obtain the coordinates of the light source point and the light intensity. Specifically:

[0104] The modeling of deformation defects is mainly to simulate its three-dimensional shape through analytical function expressions in order to calculate the surface normal vector and the incident light vector from the light source, and then solve the intersection of the light vector and the light source, and assign the grayscale of the pixel according to the light intensity at that point.

[0105] First, the three-dimensional shape of the deformation defect is fitted by the analytical expression of the function. For real defects collected in industry, the three-dimensional shape can be restored by three-dimensional measurement methods such as interference method and phase method, and then fitted by different function systems such as polynomial function, distribution function, etc. Here, the real-shot image of the deformation defect is fitted by the piecewise distribution function: in order to better simulate the central shape of the defect, a peak function showing the shape of blunt object impact is designed based on t distribution and Gaussian distribution, and then the Gaussian model is used as the tail function to connect and smoothly transition to the defect-free plane. It is worth noting that the piecewise function should satisfy the same first-order derivative and the same second-order derivative at the connection point to ensure the smoothness of the connection. The only difference between the positive and negative signs of the function is the simulated pit and convex hull defects.

[0106] Secondly, solve the incident light vector at the defect, and trace the light to obtain the coordinates of the light source point and the light intensity. Based on the analytical representation of the three-dimensional surface shape of the above-mentioned deformation defect, the light vector calculated by the light propagation model of the light field camera can be traced to the three-dimensional surface of the deformation defect to solve the intersection of the light and the surface and the normal vector at the intersection. Then, according to the principle of mirror reflection, the light vector emitted by the light source is calculated, and the intersection of the light and the light source plane is further solved. Here, only the coordinates of the light source point perpendicular to the stripe direction need to be calculated, and the light intensity can be calculated based on the period c. Different light source encoding methods correspond to different solution methods. For example, for square wave stripes, the light intensity is binarized, and for sinusoidal stripes, the light intensity is calculated using a sinusoidal distribution. At this point, ray tracing and pixel assignment are completed.

[0107] The simulation speed can be improved by simplifying the calculation of the intersection points of the deformation defect surface. The above-mentioned ray tracing process involves the process of finding the intersection points of the light ray and the surface function. For each ray of light, it is necessary to cyclically use the binary search method or the analytical method to find the intersection. Both methods are very time-consuming. In fact, the height difference between the deformation defect and the defect-free plane is very small. The tiny height difference will only cause a slight offset in the solved light source point. The normal vector at the intersection is the key influencing factor of the light source point because it determines the direction of the light source. Therefore, this height difference can be ignored, and the process of solving the intersection point between the light vector and the deformation defect three-dimensional surface fitting function can be cancelled. The light ray is directly traced to the defect-free plane to calculate the intersection point, thereby avoiding the solution process of the binary search method or the analytical method; at the same time, the process of finding the intersection with the plane can be realized through matrixization. Parallel calculation eliminates the need for cyclic calculation of the light. From Figure 2 Comparing the simulation images of the pit and convex hull before and after simplification, the defect morphology has only a slight change, which has no effect on the simulation goal of studying and optimizing system parameters and generating simulation data sets. Therefore, this simplification operation did not lead to a loss of accuracy, but greatly improved the simulation speed.

[0108] S4: Based on the Lambert diffuse reflection model, the coefficients that characterize the scratch roughness / stain opacity are introduced to establish a grayscale model of diffuse reflection non-deformation defects. Specifically:

[0109] The Lambert diffuse reflection model describes an ideal diffuse reflection surface. With the diffuse reflection point as the center, the angles in the surface hemisphere are divided and the incident light from various angles is tracked. The illumination intensity obtained from each incident light is their projection on the surface normal vector. At the same time, the illumination intensity of the light source point at each position decays as the square of the distance. Finally, all incident light rays are integrated, and the radiant brightness in any reflection direction at the diffuse reflection point p can be expressed as:

[0110]

[0111] Among them, the first and second terms represent the integral of the radiance of the incident light from the structured light source and the ambient light source, respectively; ω i is the direction of the incident light; θ i is the zenith angle of the incident light, that is, the angle with the surface normal vector at point p; r i is the distance between the light source and point p; L I (r i ,ω i )、L E (r i ,ω i ) are the structured light source and the ambient light source in the direction of ω i , the distance from point p is r iThe radiant brightness distribution at point p is calculated based on the distance r. i The square attenuation of Ω I ,Ω E Contains the possible incident light of the structured light source and the ambient light source at point p respectively.

[0112] If the grayscale of diffuse reflection defects is calculated according to the above formula, the first and second terms of I0 have low correlation with position, that is, its grayscale distribution should be similar everywhere. However, from the actual images of scratches, there is a significant difference in the grayscale value of the scratch when the background stripes are black and white, which shows that the mirror scratch is a relatively rough surface. When establishing the grayscale model, in addition to considering the ambient lighting, the background grayscale also needs to be considered. At present, there are some micro-surface models used to characterize the reflection characteristics of diffuse reflection surfaces, such as the GGX model. In order to simplify the decomposition and calculation of micro-surfaces, the following formula is established to characterize the grayscale distribution at the diffuse reflection defect point p:

[0113] I d (p)=λ(p)×I0(p)+(1-λ(p))×I n (p)

[0114] Among them, I0(p) is the grayscale distribution under ideal diffuse reflection, which is linearly related to L0(p). By introducing the coefficient distribution λ(p) that characterizes the roughness of scratches or the opacity of stains, the grayscale distribution I0(p) of the defect-free background stripe light source is converted to n (p) is taken into consideration. The larger the λ value, the greater the I d The less affected by the background grayscale.

[0115] By fixing the coefficients and adding random fluctuations, the process of angle division and ambient light integration is cancelled, thereby increasing the simulation speed. The above calculation of I0 involves the angle division and light integration of the surface hemisphere for each incident point. Since the grayscale values ​​of the ambient light component in the defect area are slightly different, I0 is almost equal everywhere, and the grayscale difference in different areas has a low correlation with the influence of system parameters on the defect imaging quality. Therefore, I0 is set to a constant value and is directly simplified to a fixed coefficient that characterizes the ambient light intensity. This avoids the process of angle division and light integration, greatly reducing the amount of calculation. Similarly, the λ parameter can also be simplified to a fixed coefficient. By adjusting the two coefficients I0 and λ, the imaging of scratches and stains defects in different situations can be simulated. In order to further enhance the authenticity, certain Gaussian random variables can also be added to the I0 and λ parameter values ​​at different positions in the defect area to make the grayscale distribution show certain random fluctuations. From Figure 2 Comparing the simulation images before and after simplification of the scratch and stain models, this simplification also has no loss of accuracy and improves the simulation speed.

[0116] S5: Generate simulated images and their labels. Compare the simulated images with the real images to evaluate the reliability of the simulation model. Evaluate the operating efficiency of the simulation model. Specifically:

[0117] First, generating a simulated image dataset involves label generation. The label format is related to the defect detection task: image classification tasks only need to generate category labels; object detection tasks need to generate box coordinates and category labels containing four-dimensional information; semantic segmentation tasks need to generate binary images and category labels consistent with the image size.

[0118] For target detection tasks, labels can be calculated based on the coordinates of the defects. Only affine transformation of the actual defect coordinates is required, and the amount of calculation is extremely small. However, this method has the following disadvantages: on the one hand, its calculation chain is relatively long, resulting in large accumulated truncation errors in the floating-point calculation process and low accuracy; on the other hand, when used for semantic segmentation tasks, especially for narrow and long scratches, the impact of small errors may be catastrophic.

[0119] Therefore, this embodiment starts from the defect image itself. The specific method is: subtract the defect image from the normal image to obtain a residual map. For the target detection task, traverse the pixels of the entire residual map, and continuously maintain the coordinate information of the four boxes on the left, right, top, and bottom, that is, the minimum and maximum values ​​of the horizontal and vertical coordinates of pixels with grayscale greater than 0 (or a certain threshold). For the semantic segmentation task, the residual map is closed by first dilating and then corroding to fill the holes and smooth the edges, and then the image is binarized to obtain the label. Compared with the label calculation method based on defect coordinates, its disadvantage is that it needs to traverse the entire image and the amount of calculation is large, but the accuracy of the generated label can be ensured.

[0120] Secondly, the effect of the simulation model is evaluated by comparing it with the real images.

[0121] On the one hand, there are simulation effects of various defect types, such as Figure 3 The figure compares the real images (upper row) and simulated images (lower row) of typical deformation defects (pits and convex hulls) and diffuse reflection defects (scratches and stains). The parameters of the simulation system are consistent with those of the real system, as shown in the parameter group of Table 1.

[0122] Table 1 System parameter settings for comparing simulation results in a preferred embodiment of the present invention

[0123]

[0124]

[0125] Depend on Figure 3 It can be seen that the simulation results are consistent with the real images in many aspects, including:

[0126] 1) Position and diameter of the sub - figure;

[0127] 2) Vignetting phenomenon within each sub - figure;

[0128] 3) Imaging size of the defect;

[0129] 4) Lateral imaging size of the stripe;

[0130] 5) Longitudinal imaging size of the workpiece (reflected as the truncation boundary position of the stripe pattern);

[0131] 6) Stripe deformation curvature caused by pits and protrusions;

[0132] 7) Gray - scale distribution of scratches and stains under black - stripe background and white - stripe background.

[0133] This indicates that the simulation model is reliable for generating the simulation image dataset.

[0134] On the one hand, it is to evaluate the simulation effects of various system parameters, such as Figure 4A and Figure 4B shown. The key parameter settings of each group of system parameters in the figure are shown in Table 1. Among them, for the parameter groups m1 - m3, by controlling variables, the imaging of different parameters m (the focusing object distance u also changes synchronously) is shown. The same applies to the parameters n, α, and c. Taking the pit defect as an example, it can be seen that in the simulation image and the actual - shot image:

[0135] 1) As m increases (m1 < m2 < m3), the imaging morphology of the defect is the same, the imaging size of the defect becomes smaller, the lateral imaging size of the stripe becomes smaller, and the longitudinal imaging size of the workpiece becomes smaller;

[0136] 2) As n increases (n1 < n2 < n3), the imaging morphology of the defect is the same, the imaging size of the defect becomes larger, the lateral imaging size of the stripe becomes smaller, and the longitudinal imaging size of the workpiece remains unchanged;

[0137] 3) As α increases (α1 < α2 < α3), the imaging morphology of the defect is the same, the longitudinal imaging size of the defect becomes smaller, the lateral imaging size of the stripe remains unchanged, and the longitudinal imaging size of the workpiece becomes smaller;

[0138] 4) As c increases (c1 < c2 < c3), the imaging morphology of the defect is the same, the longitudinal imaging size of the defect remains unchanged, the lateral imaging size of the stripe becomes larger, and the longitudinal imaging size of the workpiece remains unchanged.

[0139] Thus, it can be seen that when the system parameters change, the changing trends of the simulation image and the actual - shot image are the same, which indicates that the simulation model is effective for studying the influence of system parameters on defect imaging.

[0140] As described above, the effect of the simulation model has been verified by comparing it with the real images, and the operating efficiency of the simulation model has also been proven to be acceptable. Table 2 lists other system parameter settings of the preferred embodiment of the present invention.

[0141] Table 2 Other system parameter settings of the preferred embodiment of the present invention

[0142]

[0143] Finally, the running efficiency of the simulation model is evaluated. For example, the speed of the simulation model in this embodiment on a single-core processor model AMD Ryzen 7 6800H and Radeon Graphics @ 3.20 GHz is about 1 μs per ray, that is, it takes about 1 second to simulate an image with a pixel size of 1000×1000.

[0144] In summary, the present invention proposes a high-precision and fast imaging simulation method for a mirror defect detection system based on a structured light field. Compared with the prior art, the present invention has the following advantages:

[0145] The imaging simulation method of the present invention has the advantages of high precision and high efficiency, and solves the problem that the existing imaging simulation method is difficult to achieve both high precision and high efficiency.

[0146] The present invention is used to simulate imaging effects before the system is built, assist in system parameter design, and effectively save time and money costs of trial and error in system design; through simulation, the imaging effects under different parameter configurations can be predicted, which better helps to select the optimal system configuration.

[0147] In view of the problem that defect samples in industry are scarce and defect data sets are small, which causes deep learning detection algorithms to overfit, the present invention is used to generate simulated images and automatically annotate data labels to expand defect data sets, which can assist in the development and training of detection algorithms, especially deep learning algorithms. The diversified defect images generated by simulation by the present invention can enhance the generalization ability of the algorithm and reduce overfitting.

[0148] The optimized simulation method of the present invention greatly reduces the computational complexity of the simulation, improves the simulation efficiency, significantly shortens the system design cycle, effectively improves the system design, and quickly expands the trainable data set, thereby improving the system detection accuracy, which helps to efficiently reduce the quality fluctuations of industrial products and improve product consistency.

[0149] The imaging simulation method proposed in the present invention has great application potential and is suitable for mirror defect detection in various industrial scenarios. It helps to improve the reliability and practicality of the automated detection system and has high application value and good application prospects.

[0150] An embodiment of the present invention further provides a storage medium for storing a computer program, which at least performs the above method when executed.

[0151] An embodiment of the present invention further provides a control device, comprising a processor and a storage medium for storing a computer program; wherein the processor is configured to execute at least the method described above when executing the computer program.

[0152] An embodiment of the present invention further provides a processor, wherein the processor executes a computer program and at least executes the method described above.

[0153] The storage medium can be implemented by any type of non-volatile storage device, or a combination thereof. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a ferromagnetic random access memory (Flash Memory), a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM); the magnetic surface memory can be a disk memory or a tape memory. The storage medium described in the embodiments of the present invention is intended to include, but is not limited to, these and any other suitable types of memory.

[0154] In the several embodiments provided by the present invention, it should be understood that the disclosed systems and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0155] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0156] In addition, all functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may be separately used as a unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0157] A person skilled in the art can understand that: all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), disks or optical disks, etc. Various media that can store program codes.

[0158] Alternatively, if the above-mentioned integrated unit of the present invention is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present invention can be essentially or partly reflected in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROM, RAM, magnetic disks or optical disks.

[0159] The methods disclosed in the several method embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new method embodiments.

[0160] The features disclosed in several product embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new product embodiments.

[0161] The features disclosed in several method or device embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0162] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art of the present invention, several equivalent substitutions or obvious variations can be made without departing from the concept of the present invention, and the performance or use is the same, which should be regarded as belonging to the protection scope of the present invention.

Claims

1. An imaging simulation method for a mirror defect detection system based on a structured light field, characterized in that: The steps include: S1: calibrate system parameters as input for simulation; S2: Using the calibrated system parameters, ray tracing is performed from the sensor, and the incident light vector of the light field camera is solved by parallel computing methods, thereby constructing a light propagation model of the light field camera; S3: Based on the light propagation model of the light field camera, a function is designed to fit the three-dimensional surface shape of the deformation defect, the model is used to solve the incident light vector at the defect, and the light is traced to determine the coordinates of the light source point and the light intensity, so as to perform imaging simulation on the deformation defect; S4: Utilizing the light incident information provided by the light propagation model of the light field camera, the grayscale value of the diffuse reflection defect is calculated through the Lambert diffuse reflection model, and a coefficient representing the scratch roughness or the opacity of the stain is introduced to establish a grayscale model of the diffuse reflection non-deformable defect, thereby simulating the visual performance of the diffuse reflection defect under different lighting conditions; S5: Generate a simulation image and a label based on the imaging simulation of the deformation defect in step S3 and the grayscale model of the diffuse reflection non-deformation defect established in step S4.

2. The imaging simulation method of the mirror defect detection system based on structured light field according to claim 1, characterized in that: Step S1 specifically includes determining the encoding method of the structured light source, the light field camera architecture, and the simulation model input parameters, and calibrating the light field camera parameters.

3. The imaging simulation method of the mirror defect detection system based on structured light field according to claim 1 or 2, characterized in that: Step S2 specifically includes: S21: Calculate a one-to-one correspondence between the image plane sub-image array and the microlens array; S22: solving the diameter and spacing of the sub-image to generate a coordinate array of the center of the array sub-image and the center of its corresponding microlens; S23: Taking the center of the image plane as the base point, perform layer-order traversal to generate a light field sub-image array until the sub-image has no intersection with the sensor; S24: Ignoring the defocus phenomenon, the pinhole model is used to derive the light propagation model of the light field camera; S25: Using the derived light propagation model and the generated coordinate array, the incident light vector of the light field camera is solved through parallel calculation.

4. The imaging simulation method of the mirror defect detection system based on structured light field according to claim 3, characterized in that: When deriving the light propagation model of the light field camera, a fitting function model is introduced, in which vignetting and distortion are represented by polynomials: Among them, r d is the distance between the pixel and the center of the sub-image, γ1, γ2, and γ3 are fitting coefficients, which are generally negative numbers. The vignetting fitting is to multiply the grayscale value by the above polynomial, and the distortion fitting is to multiply the pixel coordinates by the above polynomial. The noise phenomenon is simulated using the Gaussian noise model.

5. The imaging simulation method of the mirror defect detection system based on structured light field according to claim 3 or 4, characterized in that: In step S25, an image of the same size as the image plane is initialized for each sub-image. After the sub-images are calculated in parallel, the circular areas are extracted by multiplying each sub-image by a pre-calculated 0-1 mask matrix, and then all the results are superimposed into a complete light field image to achieve parallel calculation.

6. The imaging simulation method of a mirror defect detection system based on a structured light field according to any one of claims 1 to 5, characterized in that: Step S3 specifically includes: S31: fitting the three-dimensional surface shape of the deformation defect using a function analytical expression, including fitting the deformation defect in the real shot image using a piecewise distribution function; S32: Based on the fitted three-dimensional surface shape, solve the incident light vector at the defect; S33: trace the light to the light source point, and calculate the coordinates of the light source point and the light intensity; S34: directly trace the light to the defect-free plane to calculate the intersection point; use the matrix method to calculate the intersection point of the light and the plane in parallel.

7. The imaging simulation method of a mirror defect detection system based on a structured light field according to any one of claims 1 to 6, characterized in that: Step S4 specifically includes: S41: Based on Lambert diffuse reflection model, a grayscale model of diffuse reflection non-deformation defects is established; S42: A coefficient characterizing the scratch roughness or stain opacity is introduced, combining the ideal diffuse reflection grayscale and the background grayscale.

8. The imaging simulation method of a mirror defect detection system based on structured light field according to claim 7, characterized in that: In step S42, the grayscale distribution at the diffuse reflection defect point p is characterized by the following formula: I d (p)=λ(p)×I0(p)+(1-λ(p))×I n (p) Among them, I0(p) is the grayscale distribution under ideal diffuse reflection, which is linearly related to L0(p). By introducing the coefficient distribution λ(p) that characterizes the roughness of scratches or the opacity of stains, the grayscale distribution I0(p) of the defect-free background stripe light source is converted to n (p) is taken into consideration. The larger the λ value, the greater the I d The less affected by the background grayscale; Preferably, the amount of calculation is reduced by fixing I0(p) and λ(p) to coefficients I0 and λ, and the grayscale distribution exhibits certain random fluctuations by adding Gaussian random variables to the I0 and λ parameter values ​​at different p.

9. The imaging simulation method of a mirror defect detection system based on a structured light field according to any one of claims 1 to 8, characterized in that: Step S5 specifically includes: S51: Generate a simulation image data set, and perform a difference between the defect image and the normal image to obtain a residual image; S52: For the target detection task, traverse the pixels of the entire residual map and generate labels by maintaining the coordinate information in the residual map; S53: For the semantic segmentation task, the residual map is processed by closing operation and binarization to generate labels.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the imaging simulation method of a mirror defect detection system based on a structured light field as described in any one of claims 1 to 9 is implemented.

Images