A surface defect detection method and device based on multi-view synthesis imaging

Abstract

The application relates to the technical field of surface defect intelligent detection, and discloses a surface defect detection method and device based on multi-view synthetic imaging. The method comprises the following steps: acquiring a multi-view synthetic image without dead angles of a detected target after quality evaluation; performing highlight suppression and shadow enhancement on the multi-view synthetic image to obtain an enhanced multi-view synthetic image; separating the enhanced multi-view synthetic image into corresponding multiple single-view images, respectively performing defect detection on each single-view image to obtain corresponding single-view defect detection results, and generating a final overall detection result based on the single-view defect detection results. Through multi-view synthetic imaging, image quality control, high-reflective surface enhancement processing and multi-view joint defect detection, the application can effectively solve the problems of light interference and defect missing detection in high-reflective surface detection, and can realize high-precision surface defect detection with no dead angle coverage in a single detection.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and industrial surface defect detection technology, and in particular to a surface defect detection method and apparatus based on multi-view synthetic imaging. Background Technology

[0002] Highly reflective surfaces (such as electroplated parts, polished metals, and coated curved surfaces) are prone to specular reflection, bright spots, shadows, and image saturation in machine vision inspection. This can cause the target texture and defect details to be obscured or distorted, thus affecting the stability and accuracy of defect detection. For this type of scenario, current industrial vision technologies are mainly developing along two technical routes: one is reflection suppression and measurement at the imaging and optical modeling level, and the other is learning-based anomaly / defect detection methods. Both methods have limitations and cannot simultaneously meet the comprehensive requirements of "single-shot acquisition, full coverage (no blind spots), and high robustness" under complex working conditions.

[0003] In imaging and optical modeling, Phase Measuring Deflectometry (PMD) / Deflectometry, which images stripes or patterns on a highly reflective surface and then infers surface morphology and defects from reflection distortion, has been widely used for the inspection of mirrors and highly reflective curved surfaces. It can be extended to multi-view / multi-camera stitched measurements to improve coverage and detection sensitivity. However, this method is highly dependent on the light source, screen / pattern, camera pose, and calibration accuracy, resulting in complex system setups. Furthermore, it suffers from viewpoint occlusion and blind spot issues on high-speed production lines and on complex-shaped parts.

[0004] Polarization imaging separates or suppresses specular components by controlling or estimating the polarization state of reflected light, thereby mitigating the interference of highlights and glare on texture features. In recent years, this approach has been combined with deep networks and laser / multi-source modulation to improve the imaging uniformity of highly reflective scenes to some extent. However, the robustness of polarization parameter setting and hardware configuration remains a challenge in situations with complex morphology, large curvature variations, strong material anisotropy, or motion.

[0005] At the algorithmic level, industrial anomaly / defect detection has evolved from early feature engineering to deep learning-driven approaches. Unsupervised / weakly supervised anomaly detection for unlabeled or poorly labeled environments has become the mainstream research direction, with typical datasets such as MVTec AD and its subsequent versions (containing multiple object classes and textures) widely used as benchmarks. However, in real production lines, factors such as weak defects, large intra-class variations, and camera / lighting domain shifts still lead to missed and false detections. To compensate for the scarcity of real anomaly samples, anomaly synthesis strategies (synthesizing defects in images or feature spaces) have emerged in recent years, which can expand the training distribution. However, the domain difference problem between synthetic samples and real defects still exists.

[0006] To address the issues of "weak defects" and "synthesis-reality domain difference," research in 2024-2025 proposed a unified anomaly synthesis framework (such as GLASS) combining feature-level and image-level methods. This framework generates "near-distribution anomalies" near the feature manifold through gradient ascent guidance and truncation projection, and trains in conjunction with image-level synthesis to improve the detectability and localization accuracy of weak defects. However, this type of method still relies on single-view or few-view input, which limits its ability to guarantee full coverage and no blind spots for highly reflective surfaces within a single frame. Furthermore, it still requires multiple imaging or pose switching to fill blind spots on extreme specular reflections and complex structures.

[0007] On the other hand, joint learning and decision-making by multiple perspectives / multiple cameras can alleviate the problem of missed detection caused by perspective occlusion to some extent (early / late fusion, feature or decision layer integration). However, in highly reflective surface scenes, there are significant differences in imaging between perspectives and inconsistent distribution of illumination and highlights, making cross-perspective alignment and confidence fusion more challenging. Moreover, traditional multi-perspective solutions often require multiple shots or mechanical motion to synthesize multiple frames, which is difficult to meet the real-time requirements of "single acquisition" in high-speed production lines.

[0008] In summary, existing technologies still present trade-offs between suppressing strong specular reflections, detecting weak defects, covering blind spots in the field of view of complex-shaped parts, and balancing real-time performance and stability in a single acquisition. Optical approaches such as deflection measurement / polarization imaging are limited in terms of setup complexity and versatility. While depth anomaly detection improves adaptability, it still falls short in handling the "synthetic-real domain difference" and the consistency of multi-viewpoints. For typical scenarios involving highly reflective surfaces, such as quality inspection of small precision workpieces and sealing process packaging inspection in intelligent manufacturing, there is an urgent need for a solution that can acquire multi-view synthesized images and perform multi-view joint defect detection in a single acquisition to achieve blind-spot-free coverage and stable online inspection results. Summary of the Invention

[0009] The purpose of this invention is to overcome the problems of specular reflection, high light spots, shadow interference, and blind spots in the detection of highly reflective surfaces in existing technologies. It proposes a method and device for detecting defects on highly reflective surfaces based on multi-view synthetic imaging. By obtaining multiple viewpoints in a single imaging process and combining quality assessment, image enhancement, and multi-view joint defect detection, it achieves blind-spot-free coverage detection of highly reflective surfaces, thereby improving the accuracy, stability, and real-time performance of the detection. This invention can be widely applied to quality control and inspection in intelligent manufacturing processes, and is particularly suitable for surface quality inspection of small precision workpieces, as well as sealing processes involving highly reflective surfaces.

[0010] To achieve the objective of this invention, a surface defect detection method based on multi-view synthetic imaging is provided, comprising the following steps:

[0011] Acquire a multi-view synthetic image of the target under test without blind spots after quality assessment;

[0012] Spectrum suppression and shadow enhancement are applied to the multi-view composite image to obtain the enhanced multi-view composite image.

[0013] The enhanced multi-view composite image is separated into multiple corresponding single-view images. Defect detection is performed on each single-view image to obtain the corresponding single-view defect detection results. The final overall detection result is generated based on the single-view defect detection results.

[0014] Furthermore, acquiring a multi-view synthetic image of the target object without blind spots after quality assessment includes:

[0015] Acquire multi-view synthetic images of the target under test, and perform target detection and instance segmentation on the multi-view synthetic images to obtain instance segmentation masks corresponding to each view. The instance segmentation masks are used to indicate the spatial position and visible area of ​​the target under test in the multi-view synthetic images.

[0016] Combining instance segmentation masks, the quality of multi-view composite images is evaluated using image observation quality assessment methods to determine whether the acquired multi-view composite images meet preset quality conditions. When the acquired multi-view composite image of the current frame meets the preset quality conditions, the multi-view composite image of the current frame is used as the quality-evaluated multi-view composite image without blind spots.

[0017] Furthermore, the preset quality condition is that the multi-view synthesized image simultaneously satisfies the geometric center constraint and the edge coverage constraint. The geometric center constraint is that the center of the minimum circumcircle corresponding to the instance segmentation mask on the upper surface of the target under test is located within the reference circle predetermined in the calibration stage. The edge coverage constraint is that the difference between the coverage of the instance mask contour points of the target under test on each side of the preset reference rectangular area and the reference coverage of the upper surface of the target under test on the corresponding side does not exceed the preset coverage tolerance threshold.

[0018] Furthermore, a high-reflectivity surface image enhancement method is used to suppress highlights and enhance shadows in the multi-view composite image. This high-reflectivity surface image enhancement method includes:

[0019] The luminance component is extracted from the multi-view composite image, and the pixel values ​​are normalized to obtain the normalized luminance component.

[0020] Low-pass filtering is applied to the normalized luminance components to obtain a low-frequency luminance guide map.

[0021] Using the low-frequency brightness map as the guide map and the original brightness map as the input, a joint guided filtering is performed to obtain the edge-preserving base map;

[0022] A percentile-adaptive S-shaped dynamic range remapping is performed to enhance the dark areas of the image and compress the bright areas, resulting in an enhanced multi-view composite image.

[0023] Furthermore, the step of separating the enhanced multi-view synthesized image into multiple corresponding single-view images includes:

[0024] Obtain a preset reference rectangular region in the enhanced multi-view synthesized image;

[0025] The enhanced multi-view synthetic image is subjected to target detection and instance segmentation to obtain instance segmentation masks. Based on the relative position of each instance segmentation mask and the reference rectangular region, it is divided into top view instances and side view instances.

[0026] For each side view instance, its corresponding reference edge is determined based on the proximity relationship between its contour points and each side of the reference rectangular region;

[0027] Based on the directional relationship between each instance and its corresponding reference edge, the image region where the instance is located is rotated and transformed to align it with the standard viewing direction.

[0028] Based on the aligned image regions, a standardized single-view image is obtained by cropping.

[0029] Furthermore, the step of separating the enhanced multi-view synthesized image into multiple corresponding single-view images includes:

[0030] The normalized corner points of the reference rectangular region are extracted from the enhanced multi-view synthetic image and then denormalized to obtain the vertex set of the reference rectangular region in the image coordinate system. The geometric center of the reference rectangular region is determined based on the vertex set.

[0031] Target detection and instance segmentation are performed on the enhanced multi-view synthetic image to obtain an instance mask set and corresponding category labels, confidence scores and bounding box information. Based on the spatial position relationship of each instance in the reference rectangular region, the instance mask is divided into a side view instance set and a top view instance set.

[0032] Viewpoint separation operations are performed on the instances in the side-view instance set and the top-view instance set respectively to obtain a set of single-view images indexed by viewpoint. Each element in the single-view image set corresponds to the separation result of one viewpoint in the multi-view composite image. The viewpoint separation operation is as follows: extract the contour point set based on the instance mask, and determine the mapping relationship between the side-view instance and the corresponding edge of the reference rectangular region according to the distance relationship or coverage relationship between the contour points and each side of the reference rectangular region; perform rotation transformation on the multi-view composite image and the corresponding instance mask based on the bounding box center coordinates corresponding to the instance mask and the mapped reference edge direction information to achieve alignment of instances from different viewpoints under a unified viewpoint direction; obtain a standardized single-view image and the corresponding instance mask based on the rotated instance mask.

[0033] Furthermore, the step of performing defect detection on each single-view image to obtain corresponding single-view defect detection results, and generating the final overall detection result based on the single-view defect detection results, includes:

[0034] Each single-view image is input into the defect detection model to obtain the defect detection results under the corresponding viewpoint;

[0035] If the defect detection result from any viewpoint is that a defect exists, then the overall detection result is that the target under test has a defect. If the defect detection results from all viewpoints are that no defect exists, then the overall detection result is that the target under test does not have a defect.

[0036] Furthermore, the training process for the defect detection model includes:

[0037] Obtain normal sample images and construct a normal sample dataset;

[0038] Image-level anomalous samples are generated by synthesizing anomalous samples based on a normal sample dataset.

[0039] Based on normal sample images and image-level abnormal samples, a defect detection model is trained to obtain a defect detection model for single-view images.

[0040] This invention provides a surface defect detection device based on multi-view synthetic imaging, the device comprising the following modules:

[0041] The multi-view synthetic imaging module is used to acquire multi-view synthetic images of the target under test;

[0042] The imaging sampling module is used to acquire multi-view synthesized images from the multi-view synthetic imaging module and to evaluate the quality of the multi-view synthesized images to obtain multi-view synthesized images without blind spots.

[0043] The image enhancement module is used to suppress highlights and enhance shadows in multi-view composite images to obtain enhanced multi-view composite images.

[0044] The defect detection module is used to separate the enhanced multi-view composite image into multiple corresponding single-view images, perform defect detection on each single-view image, obtain the corresponding single-view defect detection results, and generate the final overall detection result based on the single-view defect detection results.

[0045] The present invention also provides a device.

[0046] The present invention also provides a computer-readable storage medium.

[0047] Compared with the prior art, the present invention has the following beneficial effects and advantages:

[0048] (1) This invention can obtain multiple perspective information of the target object in a single imaging by multi-view synthetic imaging and separation, avoiding the inefficiency and detection blind spots caused by traditional multiple shooting and mechanical switching.

[0049] (2) The present invention ensures the effectiveness and stability of the acquired images and improves the reliability of the detection by using an image observation quality assessment method.

[0050] (3) The present invention can effectively suppress the interference of high light spots caused by specular reflection and enhance the details of shadow areas by using a high reflective surface image enhancement method.

[0051] (4) This invention achieves comprehensive coverage of highly reflective surfaces without blind spots by using a single inspection to determine defects from multiple perspectives. It has the advantages of high detection accuracy, strong robustness and good real-time performance, and is suitable for industrial application scenarios such as intelligent manufacturing, precision workpiece inspection and packaging inspection.

[0052] (5) The present invention can introduce real abnormal samples and labels in defect detection training, reduce the impact of the unrealistic nature of synthetic samples on detection performance, and improve the generalization ability of the model. Attached Figure Description

[0053] Figure 1 This is a flowchart illustrating the surface defect detection method using multi-view synthetic imaging in an embodiment of the present invention.

[0054] Figure 2 This is a schematic diagram of the multi-view synthetic imaging surface defect detection device in an embodiment of the present invention.

[0055] Figure 3 This is a schematic diagram of the composition of a device according to an embodiment of the present invention. Detailed Implementation

[0056] The specific embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0057] Please see Figure 1 The present invention provides a surface defect detection method based on multi-view synthetic imaging, comprising the following steps:

[0058] Step S1: Obtain a multi-view composite image of the target under test without blind spots after quality assessment.

[0059] This step includes the following sub-steps:

[0060] Step S1.1: Acquire multi-view synthetic images of the target under test, and perform target detection and instance segmentation on the multi-view synthetic images to obtain instance segmentation masks corresponding to each view. The instance segmentation masks are used to indicate the spatial position and visible area of ​​the target under test in the multi-view synthetic images.

[0061] In one embodiment, the object detection and instance segmentation method is implemented based on an end-to-end deep learning instance segmentation network, preferably using the YOLO11x-seg model, and its specific implementation process is as follows:

[0062] S1.1.1: Obtain the input image.

[0063] The input image to be detected is obtained as the model input. The input image is a multi-view synthesized image, and size normalization and pixel value standardization are performed according to the model input requirements.

[0064] S1.1.2: Extract image features.

[0065] The input image is fed into the backbone network of the YOLO11x-seg model, and multi-scale semantic features are extracted through multi-layer convolution and feature pyramid structure to characterize the spatial structure and semantic information of the target being tested.

[0066] S1.1.3: Predict the category information, bounding box position, and mask coefficient of the candidate target.

[0067] Based on the feature extraction results, the detection head of the YOLO11x-seg model simultaneously outputs the class probabilities of multiple candidate targets, the corresponding bounding box parameters, and the instance mask coefficients, wherein the instance mask coefficients are used to characterize the linear combination relationship between each candidate target and the prototype mask.

[0068] S1.1.4: Generate an instance mask based on the instance mask coefficients and the prototype mask.

[0069] The instance mask coefficients are weighted and combined with the prototype mask output by the YOLO11x-seg model, and then subjected to activation function and size mapping operations to generate instance segmentation masks that correspond one-to-one with each candidate target.

[0070] S1.1.5: Output the detection and segmentation results of the candidate targets.

[0071] Candidate targets are subjected to confidence screening and non-maximum suppression processing. Finally, the category label, bounding box position, and corresponding instance segmentation mask of the tested target are output as the result of target detection and instance segmentation.

[0072] Step S1.2: Combine the instance segmentation mask and use the image observation quality assessment method to assess the quality of the multi-view composite image. Determine whether the acquired multi-view composite image meets the preset quality conditions. When the acquired multi-view composite image of the current frame meets the preset quality conditions, the multi-view composite image of the current frame is used as the multi-view composite image without blind spots after quality assessment.

[0073] The image observation quality assessment method is used to perform real-time quality judgment on the acquired multi-view composite images during the acquisition process. Only when the multi-view composite image meets preset quality conditions is it saved as a valid sample image, and the current imaging sampling process ends. The preset quality conditions require the multi-view composite image to simultaneously meet preset geometric center constraints and edge coverage constraints. Specifically, the image observation quality assessment method performs quality judgment based on the following two criteria:

[0074] Geometric center constraint:

[0075] During the imaging process, the mask region on the upper surface of the target under test is first extracted, and the mask is segmented based on instances on the upper surface of the target under test. Calculate instance segmentation mask The position of the center of the corresponding smallest circumcircle , These represent the x and y coordinates of the center of the smallest circumcircle calculated from the mask area on the upper surface of the target object in the image coordinate system, respectively. Beforehand, during the calibration phase, the position and size parameters of the circle in the image pixel coordinate system are manually drawn and recorded on the image based on the initial running image, thus determining the reference circle. .in, For the radius of the reference circle, As the center of the reference circle, , These represent the x and y coordinates of the reference circle center (true value circle center) obtained during the calibration phase in the image coordinate system, respectively. The center position of the smallest circumcircle is compared with the reference circle. If the center position of the smallest circumcircle is within the range of the reference circle, that is, if the following condition is met, then the geometric center constraint is considered valid:

[0076]

[0077] in This indicates Euclidean distance.

[0078] Edge coverage constraint:

[0079] Define the reference rectangular area of ​​the target to be measured. Reference rectangular area The four sides are , length is ;

[0080] Based on the instance mask contour point set of the target to be tested Calculate the contour points on each side. Coverage on Specifically, it includes:

[0081] Extracting the contour point set from the instance segmentation mask of the target object. , Indicates the first Calculate the x and y coordinates of each contour point and the distance from each contour point to the edge. minimum distance For a given threshold Defined at a given threshold Set of valid points within for:

[0082]

[0083] in, The index of the contour point is used to identify the first... A contour point in the contour point set The position in the middle; This represents the total number of contour points, i.e., the number of contour points extracted from the instance segmentation mask.

[0084] Project these contour points onto the edge Get the number of covered points on the edge. This leads to the edge coverage rate. Edge coverage The expression is:

[0085]

[0086] in This represents the number of pixels corresponding to the side length.

[0087] For each edge, set an allowable coverage tolerance threshold. Edge coverage Reference coverage of the upper surface of the target under test on the corresponding edge The comparison is performed when the coverage differences on each side do not exceed the preset coverage tolerance threshold. When, that is, if the following conditions are met:

[0088]

[0089] Then the edge coverage constraint is determined to be valid.

[0090] in, Indicates the upper surface of the measured target relative to the edge The baseline coverage.

[0091] Image quality assessment is performed based on the geometric center constraint determination results and the edge coverage constraint determination results.

[0092] If the geometric center constraint is not satisfied, the multi-view composite image of the current frame is determined to be insufficient and invalid, and the sampling loop continues. If the geometric center constraint is satisfied but the edge coverage constraint is not satisfied, a prompt signal is output, waiting for manual confirmation to terminate the acquisition or continue the loop. If both the geometric center constraint and the edge coverage constraint are satisfied, that is, there are no blind spots in the target in the multi-view composite image of the current frame, the multi-view composite image of the current frame is automatically acquired and saved as a valid sample, and the loop ends.

[0093] Step S2: Perform highlight suppression and shadow enhancement on the multi-view composite image to obtain the enhanced multi-view composite image.

[0094] This step uses a strong reflective surface image enhancement method to suppress highlights and enhance shadows in the multi-view composite image.

[0095] The method for enhancing images of highly reflective surfaces includes the following steps:

[0096] Step S2.1: Perform luminance extraction and normalization, and normalize the pixel values ​​to obtain the normalized luminance components.

[0097] Multi-view composite image (This can be an RGB image or other color format) converted to a luminance component, in one embodiment, by extracting... The luminance component is extracted by either using channels or calculating a grayscale image; then, the luminance component is linearly normalized so that the pixel values ​​are mapped to the interval [0,1].

[0098]

[0099] in, This represents the normalized luminance component. These represent the minimum and maximum values ​​of the image brightness, respectively. For small, stable terms, Represents a multi-view synthesized image from the input. The original luminance component is extracted from it.

[0100] Step S2.2: Construct a low-frequency brightness guide map.

[0101] Through smoothing operators (e.g., with scale parameters) Gaussian blur) is used to perform low-pass filtering on the normalized luminance components to obtain the low-frequency luminance map. :

[0102]

[0103] in, For having scale parameters Smoothing operators (e.g., Gaussian blur). Optionally, for low-frequency brightness maps... Apply gamma mapping To compress the dynamic range of the extremely bright area, This is a low-frequency brightness guide map after gamma transformation, used to suppress extremely bright areas and improve brightness distribution.

[0104] Step S2.3: Perform edge-preserving joint guided filtering, i.e., using the low-frequency brightness map As a guide map, the original luminance map, i.e., the luminance components after normalization, is used. As input, joint guided filtering is performed to obtain an edge-preserving base map.

[0105] Low-frequency brightness diagram As a guide map, the normalized luminance components are used. As input images, each indexed by pixels Centered on, with radius Local neighborhood window Within this framework, it is assumed that there is a locally linear relationship between the output brightness and the guided brightness, where, Indicates the position index of the center pixel of the window. Represented in pixels A local neighborhood window (e.g., a square region of fixed size) is constructed around a central point. Within this local neighborhood window... Internal calculation of linear model parameters:

[0106]

[0107] in, Indicates a local neighborhood window Inside, the normalized luminance component of the input. Low-frequency brightness diagram used for guidance The local linear relationship coefficient between them This represents the bias term used to compensate for local brightness levels. Represents low-frequency brightness diagram With normalized luminance components In local neighborhood window Covariance within, Represents low-frequency brightness diagram In local neighborhood window within variance, and Representing local neighborhood windows respectively Internal guide map, i.e., low-frequency brightness map With normalized luminance components The local mean, It is a numerically stable term.

[0108] For each local neighborhood window The results are averaged. This step utilizes prior knowledge of low-frequency brightness to maintain smooth edges, effectively suppressing specular artifacts caused by strong reflections while preserving target boundaries. Output pixels are averaged using a window.

[0109]

[0110] in, Represents brightness image At pixel position The brightness value at that location, i.e., the guided brightness under local neighborhood constraints. The output brightness of a single pixel obtained from linear reconstruction. To cover pixels Each local neighborhood window On average, Represents pixels The guide image pixel values ​​at the location are usually low-frequency brightness maps (guide brightness), which are used to constrain the local linear structure of the output results.

[0111] Step S2.4: Perform percentile adaptive S-shaped dynamic range remapping to enhance the dark areas and compress the bright areas in the image, resulting in an enhanced multi-view composite image.

[0112] Step S2.4.1: Determine the endpoints of the protected area based on percentiles:

[0113]

[0114] Step S2.4.2: Perform linear normalization on the brightness values:

[0115]

[0116] in, Represents the position of all pixels place The combined output brightness image is used for subsequent brightness statistics and dynamic range remapping processing. This indicates the lower limit of brightness normalization. Indicates the upper limit of brightness normalization; Indicates the use of calculation percentile parameter, Represents brightness image The lower percentile value is used as the lower bound for normalization; Indicates the use of calculation percentile parameter, Represents brightness image The highest percentile value is used as the upper limit of normalization; This represents the brightness value after percentile normalization, corresponding to a pixel. The remapping results; It is a small, stable term.

[0117] Step S2.4.3: Construct a piecewise sigmoid function Exponential boosting (gamma ≤ 1) is applied to dark areas, and exponential compression (gamma > 1) is applied to bright areas.

[0118]

[0119] in, This represents the brightness value after percentile normalization, used as the input to the piecewise S-shaped function; This represents the normalized brightness value. The output brightness value obtained after S-type dynamic range remapping is used to enhance the brightness of dark areas and compress the brightness of bright areas while maintaining the overall brightness continuity. This represents the gamma factor (less than or equal to 1) used to enhance the brightness of dark areas; This represents the gamma factor (greater than or equal to 1) used for bright areas, which is used to compress the brightness range of bright areas. In one embodiment, the brightness threshold is represented. The value is 0.5.

[0120] Step S2.4.4: Map the results back to the original dynamic range, while preserving the protection of the extremely dark and extremely bright regions:

[0121]

[0122] in, The enhanced brightness map shows that shadow areas are brightened, highlight reflections are suppressed, and edges are preserved.

[0123] Enhanced brightness map As the output of the image enhancement module, or by combining it with the original multi-view composite image. After channel replacement or fusion, an enhanced multi-view synthetic image is output for subsequent identification by the defect detection module.

[0124] In one embodiment, the enhanced brightness map obtained through step S2 It can be directly used in step S3, or it can replace the brightness channel of the original image, or it can be fused with the original image to obtain an enhanced multi-view composite image. This enhanced multi-view composite image is used in the subsequent step S3 for defect identification and judgment.

[0125] In this way, the image enhancement module can effectively suppress specular highlight reflection areas in high reflectivity surface detection scenarios, while enhancing texture details in low brightness areas, thereby improving the overall image observation quality.

[0126] Step S3: Separate the enhanced multi-view composite image into multiple corresponding single-view images (sample images), perform defect detection on each single-view image to obtain the corresponding single-view defect detection results, and generate the final overall detection result based on the single-view defect detection results.

[0127] The enhanced multi-view composite image is separated into multiple corresponding single-view images using a multi-view separation method, including the following steps:

[0128] Step S3.1: Acquire an enhanced multi-view composite image containing multiple viewpoints of the target under test. ;

[0129] From enhanced multi-view synthetic images Extract the reference rectangular region Normalized corner point The normalized angle is denormalized based on the image size to obtain the reference rectangular region. The set of quadrilateral vertices in the image coordinate system Meanwhile, the reference rectangular region is determined by the diagonal points of the quadrilateral vertex set. Reference geometric center :

[0130]

[0131]

[0132] Among them, variables Indicates the first Normalized corner coordinates from each viewpoint and These represent the horizontal and vertical components of the corner point, respectively. and These are the width and height of the multi-view synthesized image, used to normalize the corner coordinates within the image size range; The viewpoint number indicates the first viewpoint in the multi-view composite image. Image; The channel index of the reference image is used to unify the sampling benchmark; and Normalized reference rectangles The coordinates of the top-left and bottom-right vertices in the region. and Normalized reference rectangles The coordinates of the top-right and bottom-left vertices in the region. These variables together constitute the input elements for the unified sampling and coordinate inverse normalization process.

[0133] Step S3.2: Process the multi-view synthesized image using the object detection and instance segmentation methods described above. Perform reasoning to obtain the instance mask set. and the corresponding category tags Confidence level and bounding box information Based on each example, within the reference rectangular area The spatial relationships within the instance mask set divide the instance mask set into top-view instance sets. and side view instance set Two categories.

[0134]

[0135] ,

[0136] Indicates the first Binary mask image for each instance, and These represent the height and width of the mask image, respectively. This represents the total number of instances. and They represent the first The x and y coordinates of the top-left corner of each instance bounding box. and They represent the first The x and y coordinates of the bottom right corner of the instance bounding box.

[0137] Step S3.3: For the set of side-view instances Binary mask image for each instance Extract its outer contour point set, and then base the results on the contour points and the edges of the reference rectangular region. Determine the mapping relationship between the side view instance and the corresponding edge of the reference rectangular area by determining the distance or coverage relationship between them;

[0138] Given reference rectangular region The four sides are as follows: For each contour point in the outer contour point set, calculate its distance to the edge. The minimum distance, combined with the minimum average distance or maximum coverage criterion, determines the instance mask and the reference rectangular region. The correspondence between edges is used to establish the mapping relationship between the instance mask and the edges.

[0139] Calculate the center coordinates of the bounding box for each instance mask. , The x-coordinate of the bounding box center in the image coordinate system. Let the ordinate of the bounding box center in the image coordinate system be y, and combined with the direction angle of the corresponding edge, determine the rotation angle of the instance. Establish a rotation matrix around the origin. Perform rotational transformations on the enhanced multi-view composite image and the corresponding instance mask:

[0140]

[0141] If the centroid is located below the edge after rotation, then take the rotation angle. , Let π represent the mathematical constant pi, from which we obtain an affine matrix containing only rotations.

[0142] In one embodiment, to avoid rotational truncation, the output size is expanded to the diagonal length of the image, and the rotation matrix is ​​adjusted by center translation to obtain an equivalent secure affine matrix around the image center. Bilinear interpolation is applied to the image, and nearest-neighbor interpolation is applied to the instance mask to obtain the rotated image and the instance mask.

[0143] Step S3.4: Extract the foreground region from the rotated instance mask, calculate the minimum bounding rectangle, and perform compact cropping. The cropped result is then placed into a canvas of a preset size (in one embodiment, a square canvas) using center filling, with the canvas background being a preset color (e.g., white), to obtain a standardized single-view sample image. and corresponding instance masks .

[0144] Step S3.5: Perform steps S3.3 to S3.4 on the top view instance as well, or directly perform square cropping and filling based on the instance mask.

[0145] Step S3.6: Output a set of single-view images indexed by viewpoint. Single-view image collection Each element in the array corresponds to the separation result of one viewpoint in the multi-view composite image.

[0146]

[0147] in, Represents a set of instances of side-view perspective. It is a collection of top-view instances.

[0148] A single-view defect detection method is used to perform defect detection processing on each of the separated single-view images to obtain the corresponding single-view defect detection results. The single-view defect detection method is as follows:

[0149] In obtaining a collection of single-view surface images Then, each single-view image is processed separately. Input a defect detection model to identify defects, and output the defect detection results from the corresponding perspective. Defective, No Defect The defect detection model can output binary results (defective / no defect), or it can output probability or confidence level. and through a preset defect threshold Binarization is performed. Let the defect detection results from each viewpoint be... The joint decision rule is defined as follows:

[0150]

[0151] This represents the joint judgment result from multiple perspectives, where the defect detection result from any one perspective... (i.e., when defects exist) A value of 1 indicates that the joint judgment result is defective; a value of 1 is only valid if the defect detection results from all perspectives are 0. A value of 0 indicates that the joint determination result is defect-free.

[0152] Finally, the final defect detection result of the tested target is output. .

[0153] In one embodiment, let the binary mapping Defective , No defects ,but:

[0154]

[0155] Using the above method, it is possible to detect surface defects in separated samples using multi-view synthetic images acquired in a single acquisition, and to ensure the reliability and comprehensive coverage of the overall detection results through joint judgment.

[0156] The defect detection model is a deep learning model. In one embodiment, the defect detection model specifically adopts the GLASS defect detection model. The GLASS defect detection model is used to synthesize abnormal samples and perform defect detection. The training process and model architecture of this GLASS defect detection model are known, as in the literature [GLASS, A Unified Anomaly Synthesis Strategy with Gradient Ascent for Industrial Anomaly Detection and Localization].

[0157] The training process of the GLASS defect detection model includes: first, acquiring normal sample images and constructing a normal sample dataset; then, performing GLASS-based anomalous sample synthesis based on the normal sample dataset to generate image-level anomalous samples; next, using the normal samples and image-level anomalous samples together to train the GLASS defect detection model; finally, applying the trained GLASS defect detection model to the image to be detected and outputting the defect detection results.

[0158] In generating image-level anomaly samples, real anomaly samples collected in actual operation and their corresponding defect labels are introduced. These samples, together with the image-level anomaly samples, constitute an anomaly sample dataset, which is used for supervised training of the GLASS defect detection model.

[0159] The GLASS defect detection model incorporates real anomaly samples based on anomaly synthesis training to improve the authenticity and robustness of the detection.

[0160] To address the issue that highly reflective surfaces are prone to high-gloss spots, shadow interference, and limited viewing angles under complex lighting conditions, which leads to the need for multiple observations in traditional detection methods and the potential for blind spots, this invention provides a surface defect detection method based on multi-view synthetic imaging. By jointly judging the detection results of each single viewpoint, a multi-view joint defect detection strategy is adopted: "If a defect exists in any viewpoint, the whole surface is judged as a defect; if no defect exists in any viewpoint, the whole surface is considered defect-free." This allows for comprehensive coverage and defect judgment of highly reflective surfaces without blind spots in a single detection process.

[0161] In one embodiment, a surface defect detection device based on multi-view synthetic imaging is provided to implement the method provided in the foregoing embodiments. Please refer to [link to previous embodiment]. Figure 2 The device includes the following modules:

[0162] The multi-view synthetic imaging module is used to acquire multi-view synthetic images of the target under test;

[0163] The imaging sampling module is used to acquire multi-view synthesized images from the multi-view synthetic imaging module and to evaluate the quality of the multi-view synthesized images to obtain multi-view synthesized images without blind spots.

[0164] The image enhancement module is used to suppress highlights and enhance shadows in multi-view composite images to obtain enhanced multi-view composite images.

[0165] The defect detection module is used to separate the enhanced multi-view composite image into multiple corresponding single-view images, perform defect detection on each single-view image, obtain the corresponding single-view defect detection results, and generate the final overall detection result based on the single-view defect detection results.

[0166] The multi-view synthetic imaging module includes the following units:

[0167] The multi-mirror unit 2 is used to generate multiple mirror views of the same target by multiple mirrors arranged according to a predetermined geometric relationship, and to make the multiple mirror views and the direct view of the target fall into the field of view of the camera unit.

[0168] Camera unit 1 is used to acquire raw image frames containing direct viewpoints and multiple mirror viewpoints in one or more exposures, that is, to acquire multi-view composite images of the target under test.

[0169] In one embodiment, an auxiliary light source unit may also be included to adjust the lighting conditions, thereby further reducing the interference of highly reflective surfaces on image quality.

[0170] The imaging sampling module performs target detection and instance segmentation on the multi-view synthesized image based on target detection and instance segmentation methods, obtaining instance segmentation masks corresponding to each viewpoint. These masks indicate the spatial location and visible area of ​​the target object in the multi-view synthesized image. Further, the imaging sampling module invokes an image observation quality assessment method, combining the instance segmentation masks to calculate whether the multi-view synthesized image meets preset quality conditions. Additionally, the instance segmentation masks serve as input constraints for subsequent defect detection methods, limiting the effective area for defect detection and improving the accuracy and stability of the detection results.

[0171] After receiving the multi-view composite image acquired by the imaging sampling module, the image enhancement module processes the multi-view composite image using a strong reflective surface image enhancement method to generate an enhanced multi-view composite image that suppresses highlights and enhances shadows. The processing includes removing the highlight reflection areas in the multi-view composite image and enhancing the shadows in the low-brightness areas to generate the suppressed image.

[0172] The defect detection module receives the enhanced multi-view synthetic image after processing by the image enhancement module, and performs multi-view separation and single-view defect detection on the enhanced multi-view synthetic image based on the multi-view synthetic image combined defect detection method. Finally, the detection results of multiple views are integrated into the overall detection result.

[0173] The multi-view synthetic image joint defect detection method includes a multi-view separation method and a single-view defect detection method. The multi-view separation method is used to parse and separate the input multi-view synthetic image into multiple corresponding single-view images. The single-view defect detection method is used to perform defect detection processing on each of the separated single-view images to obtain the corresponding single-view defect detection results. The final overall detection result is generated based on the single-view defect detection results.

[0174] In one embodiment, a device is provided; see [link to device]. Figure 3 The device includes a processor unit 3 and a memory unit 4. The memory unit 4 stores instructions or computer programs, and the processor unit 3 executes the instructions or computer programs in the memory unit 4 to cause the device to perform the steps of the method described in the foregoing embodiments. The memory unit also stores raw image data acquired by the camera unit and detection results generated by the processor unit, and provides data caching for the operation of the processor unit.

[0175] In one embodiment, an output unit is further included for outputting the defect detection results generated by the processor unit. The output unit includes a display component and a storage component. The display component is used to visually present the detection images and result information on the display screen, and the storage component is used to write the detection results and related image data to a disk or other storage medium for subsequent querying, statistics and traceability.

[0176] In one embodiment, a computer-readable storage medium is provided, the computer-readable storage medium storing instructions that, when executed on a device, cause the device to perform the steps of the method described in the foregoing embodiments.

[0177] In one embodiment, camera unit 1 includes a 4K high-precision UVC camera with a resolution of 3496x4656, and camera unit 1 includes a stable light source. Multi-mirror unit 2 is composed of multiple ESR silver plane mirrors and mirror fixing components, used to acquire composite images containing multiple viewpoints.

[0178] In one embodiment, the processor unit is an embedded industrial computer and a graphics processing unit (GPU).

[0179] In one embodiment, the table compares the performance of different defect detection methods on a sealant dataset, where Img- The index is used to evaluate whether the method can correctly distinguish between defective and normal images at the image level. Pix- The metrics are used to evaluate the method's pixel-level accuracy in locating defect regions. Specifically, AUROC measures the method's overall discriminative power; a higher value indicates a stronger ability to distinguish between defects and normal samples. Recall represents the detection rate of true defects; a higher value indicates fewer missed detections. FPR (False Positive Rate) represents the false alarm rate; a lower value indicates fewer instances of normal samples being misclassified as defects. As shown in Table 1, the method of this invention achieves the highest or near-highest values ​​for both image-level and pixel-level AUROC and Recall metrics, while maintaining a low level for the FPR metric. This indicates that the method not only distinguishes between defective and normal samples more accurately but also provides more complete defect region localization with fewer false alarms. Compared to existing unsupervised, supervised, and other semi-supervised methods, the method of this invention achieves a better balance between detection accuracy and stability, and its overall defect detection performance is significantly superior to the comparative methods.

[0180] Table 1. Comparison of indicators between the method of the present invention and the comparative method.

[0181]

[0182] Those skilled in the art will understand that various modifications and variations can be made to the above embodiments without departing from the spirit and scope of the present invention, and all such modifications and variations should fall within the scope of protection of the present invention.

Claims

1. A surface defect detection method based on multi-view synthetic imaging, characterized in that, Includes the following steps: The multi-view composite image of the target under test is acquired by a multi-view composite imaging module, which includes a camera unit and a multi-mirror unit. The multi-mirror unit is used to generate multiple mirror views of the same target under test from multiple mirrors arranged according to a predetermined geometric relationship, and to make the multiple mirror views and the direct view of the target under test fall into the field of view of the camera unit. The camera unit is used to acquire original image frames containing the direct view and multiple mirror views in one or more exposures. Target detection and instance segmentation are performed on multi-view synthetic images to obtain instance segmentation masks corresponding to each view. The instance segmentation masks are used to indicate the spatial location and visible area of ​​the target in the multi-view synthetic image. For the acquired multi-view composite images, an image observation quality assessment method is used to evaluate the quality of the multi-view composite images to determine whether the acquired multi-view composite images meet the preset quality conditions. When the acquired multi-view composite images of the current frame meet the preset quality conditions, the multi-view composite images of the current frame are used as the multi-view composite images without blind spots after quality assessment. The preset quality conditions are that the multi-view composite images simultaneously meet geometric center constraints and edge coverage constraints. The geometric center constraint is that the center of the minimum circumcircle corresponding to the instance segmentation mask of the target surface is located within the reference circle predetermined in the calibration stage. The edge coverage constraint is that the difference between the coverage of the instance mask contour points of the target on each side of the preset reference rectangular area and the reference coverage of the target surface on the corresponding side does not exceed the preset coverage tolerance threshold. The multi-view composite image is enhanced by suppressing highlights and enhancing shadows using a strong reflective surface image enhancement method. The high-reflectivity surface image enhancement method includes: extracting the luminance component from the multi-view composite image and normalizing the pixel values ​​to obtain the normalized luminance component; performing low-pass filtering on the normalized luminance component to obtain a low-frequency luminance guide map; using the low-frequency luminance map as the guide map and the original luminance map as input, performing joint guide filtering to obtain an edge-preserving base map; and performing percentile-adaptive S-shaped dynamic range remapping to enhance the dark areas and compress the bright areas in the image to obtain the enhanced multi-view composite image. The enhanced multi-view composite image is separated into multiple corresponding single-view images. Defect detection is performed on each single-view image to obtain the corresponding single-view defect detection results. The final overall detection result is generated based on the single-view defect detection results.

2. The surface defect detection method based on multi-view synthetic imaging according to claim 1, characterized in that, The step of separating the enhanced multi-view synthesized image into multiple corresponding single-view images includes: Obtain a preset reference rectangular region in the enhanced multi-view synthesized image; The enhanced multi-view synthetic image is subjected to target detection and instance segmentation to obtain instance segmentation masks. Based on the relative position of each instance segmentation mask and the reference rectangular region, it is divided into top view instances and side view instances. For each side view instance, its corresponding reference edge is determined based on the proximity relationship between its contour points and each side of the reference rectangular region; Based on the directional relationship between each instance and its corresponding reference edge, the image region where the instance is located is rotated and transformed to align it with the standard viewing direction. Based on the aligned image regions, a standardized single-view image is obtained by cropping.

3. A surface defect detection method based on multi-view synthetic imaging according to any one of claims 1-2, characterized in that, The process involves performing defect detection on each single-view image to obtain corresponding single-view defect detection results, and generating a final overall detection result based on these results, including: Each single-view image is input into the defect detection model to obtain the defect detection results under the corresponding viewpoint; If the defect detection result from any viewpoint is that a defect exists, then the overall detection result is that the target under test has a defect. If the defect detection results from all viewpoints are that no defect exists, then the overall detection result is that the target under test does not have a defect.

4. A surface defect detection device based on multi-view synthetic imaging, characterized in that, The apparatus for implementing the method according to any one of claims 1-3, comprising the following modules: The multi-view synthetic imaging module is used to acquire multi-view synthetic images of the target under test; The imaging sampling module is used to acquire multi-view synthesized images from the multi-view synthetic imaging module and to evaluate the quality of the multi-view synthesized images to obtain multi-view synthesized images without blind spots. The image enhancement module is used to suppress highlights and enhance shadows in multi-view composite images to obtain enhanced multi-view composite images. The defect detection module is used to separate the enhanced multi-view composite image into multiple corresponding single-view images, perform defect detection on each single-view image, obtain the corresponding single-view defect detection results, and generate the final overall detection result based on the single-view defect detection results.

5. A device for surface defect detection based on multi-view synthetic imaging, characterized in that, The device includes a processor unit and a memory unit, the memory unit being used to store instructions or computer programs, and the processor unit being used to execute the instructions or computer programs in the memory unit to cause the device to perform the steps of the method according to any one of claims 1-3.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions that, when executed on the device, cause the device to perform the steps of the method according to any one of claims 1-3.

Images