Detection method and device for intelligent visual detection of parts and storage medium

Abstract

The invention discloses a detection method and device for intelligent visual detection of parts and a storage medium, belongs to the technical field of industrial quality detection, and aims to solve the problems that surface and internal defects of complex parts are difficult to identify synchronously and the identification precision is low. The method comprises the following steps: acquiring multi-modal image data obtained through structured light three-dimensional imaging and laser ultrasonic scanning; performing geometric registration and scale normalization on the image to generate a fused image; carrying out image preprocessing and edge analysis, and extracting a region of interest; dividing the candidate region of interest into a plurality of image blocks, and inputting the image blocks into an anomaly detection network; calculating a reconstruction error between the original image block and the reconstructed image block, and generating an abnormal scoring graph; and finally, extracting a defect area through image post-processing, and outputting information such as a defect type, a spatial position, a geometric dimension and a severity level. According to the method, the robustness and accuracy of multi-modal defect identification are improved, and the method is suitable for an online visual inspection task of an industrial production line.

Description

Technical Field

[0001] The present invention relates to the field of industrial quality inspection technology, and in particular to the field of component defect inspection. Background Art

[0002] With the continuous advancement of intelligent manufacturing and industrial automation, component quality inspection, a critical step in the manufacturing process, has become increasingly important. Its accuracy and efficiency are directly linked to the quality of vehicle assembly and operational safety. In particular, the automotive, aviation, and precision machinery sectors are placing higher demands on surface and internal defect detection for complex components like stampings and structural parts. Currently, mainstream inspection methods rely on manual visual inspection or inspection systems based on traditional visual algorithms. While this has improved inspection efficiency to a certain extent, it still presents significant deficiencies in detecting defects such as complex morphologies and structures and fine cracks.

[0003] For surface defect detection, existing technologies often rely on structured light-based 3D vision inspection systems, using DMDs (digital micromirror devices) to generate phase-shifted fringes for 3D reconstruction. However, core DMD chip technology has long been subject to foreign control, resulting in unstable supply, high costs, and difficulties in system integration. Furthermore, traditional structured light systems based on static light fields are sensitive to depth of field and noise, making them difficult to adapt to complex industrial environments such as frequent vibration and electromagnetic interference, leading to large detection errors.

[0004] In terms of internal defect detection, although existing methods such as X-ray imaging, industrial CT and ultrasonic testing can achieve a certain degree of defect identification, they have problems such as slow detection speed, long imaging time, high radiation safety risks or low system integration, and it is difficult to meet the technical requirements of modern production lines for high-speed, online and non-contact detection.

[0005] In terms of defect recognition algorithms, traditional supervised learning-based image recognition methods rely on large amounts of high-quality, manually annotated data. These methods suffer from weak generalization and robustness in the small sample sizes, large intra-class variability, and small inter-class variability commonly found in industry. Furthermore, most existing recognition models are unable to effectively integrate multimodal data (such as surface 3D structure and internal body wave images), resulting in low overall recognition accuracy for internal and external defects.

[0006] Based on the limitations of the above-mentioned existing technologies in terms of reliance on imported components for surface inspection, low internal inspection efficiency, poor adaptability of defect recognition algorithms to high-resolution and high-noise images, and reliance on manual labels for training data, the present invention proposes a detection method, device, and storage medium for intelligent visual inspection of parts.

[0007] This solution utilizes a DMD-free diffraction optical surface 3D imaging module, a non-contact laser ultrasonic internal imaging module, and a weakly supervised anomaly detection algorithm based on SoftPatch to achieve unified acquisition, intelligent identification, and visualization of both surface and internal information. In particular, the SoftPatch algorithm, through feature region weighting and noise filtering, improves the accuracy and stability of defect recognition in scenarios with weak features and mixed interference, effectively addressing the shortcomings of existing technologies in detection accuracy, system reliability, and integrated intelligence. Summary of the Invention

[0008] The embodiments of the present invention aim to provide a complex component defect identification method based on multimodal image fusion and weakly supervised reconstruction mechanism. The method combines a DMD-free structured light acquisition system, non-contact laser ultrasonic technology and a SoftPatch image reconstruction and recognition network to address the shortcomings of existing methods in terms of equipment dependence, recognition accuracy, abnormal robustness and label data dependence, thereby achieving automatic, accurate and rapid identification of surface and internal defects.

[0009] To achieve the above objectives, an embodiment of the present invention proposes a method for automatic multi-dimensional defect detection of complex surface parts based on weakly supervised image reconstruction, comprising the following steps:

[0010] S101: Acquire multimodal image data after fusion of surface structured light and internal laser ultrasound imaging, perform geometric alignment and noise suppression on the images, generate a standardized registration image set, unify the scale and align the coordinate system, and construct a fused feature image set that can be used for subsequent region of interest extraction and reconstruction analysis.

[0011] S102: Perform edge detection and connected domain analysis on the fused image to extract regions of interest suspected of containing defects. Combined with the gradient amplitude map and the regional compactness index, the regions of interest are screened to remove background and pseudo-boundary regions, retaining sub-regions with obvious structural features, and cropping them into a set of image blocks of equal size as detection input.

[0012] S103: Each input image patch is input into the SoftPatch anomaly detection network based on the weakly supervised image reconstruction mechanism. The surface and internal features are fused through the feature encoder to extract the potential feature representation of the image patch. .

[0013] S104: Feature vectors in the embedding library constructed with normal samples Perform similarity matching and generate predicted image blocks from multiple candidate reconstruction patches by weighting through a soft-weighted matching mechanism The matching weight According to the cosine similarity between features and temperature modulation parameters Determine, the calculation formula is as follows:

[0014]

[0015] Compared with the traditional hard matching strategy, this soft weighted reconstruction method can improve the recognition ability of edge blur and weak texture defect areas, and enhance the robustness of the algorithm under structural perturbations.

[0016] S105: Calculate the original image block Instead of reconstructing the image The pixel error is used as the local anomaly score to construct the anomaly score map , which is defined as follows:

[0017]

[0018] S106: The generated anomaly score map A(x, y) is smoothed using Gaussian filtering and segmented using thresholding to extract a potential defect area mask. The masked area is then mapped back to the original image coordinate system. Combining the surface 3D structure map with the internal point cloud data, the geometric dimensions and position coordinates of the defect in a unified space are calculated. The final output includes multi-dimensional attribute information such as defect type, spatial location, size boundary, and severity level, enabling visual defect recognition and structured detection results for industrial applications.

[0019] Furthermore, in order to enhance the model's sensitivity to texture and structural level differences, the present invention also constructs a joint loss function for model training, which includes two parts: pixel error and perceptual error, and its form is:

[0020]

[0021] in, represents the intermediate layer representation extracted by the pre-trained perceptual feature extraction network, , are the weighting coefficients of pixel error and perceptual error, respectively.

[0022] The model of the present invention only uses normal sample images in the training stage, and learns the normal structural distribution of the component surface and interior by minimizing the reconstruction error; in the inference stage, the abnormal area cannot be accurately reconstructed due to structural deviation, and thus appears as a significant abnormal response area on the abnormality score map.

[0023] After Gaussian filtering and threshold segmentation of the abnormal score image, the defect mask image is extracted. Combined with the spatial geometric information in the original image, each abnormal area is located, measured and graded, realizing multi-modal and multi-scale automated defect recognition and quality assessment of complex surface parts.

[0024] Compared with the prior art, the present invention has the following significant advantages:

[0025] Weakly supervised modeling without labeling solves the problem of traditional detection algorithms relying on defect labels;

[0026] Supports multimodal image fusion and robust recognition, and can handle complex surface structures and low-contrast images;

[0027] The abnormality scoring mechanism based on weighted reconstruction can accurately locate the structural mutation area;

[0028] The computing overhead is controllable, adaptable to industrial edge deployment environments, and suitable for online real-time detection scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 This is a flow chart of the method for detecting surface defects of complex-profile parts according to an embodiment of the present invention;

[0030] Figure 2 This is a flow chart for three-dimensional imaging detection of internal defects of complex surface parts according to an embodiment of the present invention;

[0031] Figure 3 This is a flowchart of anomaly identification based on SoftPatch according to an embodiment of the present invention;

[0032] Figure 4 This is a detailed flow chart of the method for extracting a region of interest according to an embodiment of the present invention. DETAILED DESCRIPTION

[0033] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0034] The terms "first," "second," and so on, used in the specification and claims of this invention are intended solely to distinguish different technical features and do not imply an actual order or hierarchy. The term "and / or" used in the specification and claims refers to a combination of any one or more of the listed items. Unless expressly provided otherwise, the terms "include," "comprising," and so on, as used herein, should be understood to encompass the stated content, but do not exclude the inclusion of additional content.

[0035] like Figures 1 to 4As shown, this embodiment provides a method and apparatus for intelligent visual inspection of parts. This method can achieve simultaneous acquisition, registration fusion, region identification, reconstruction detection, and multi-attribute output evaluation of surface and internal defects of complex parts. It has the advantages of lightweight structure, high accuracy, and strong robustness. The method includes the following operations:

[0036] First, the part to be inspected is placed at the center of the imaging system's workstation, and image acquisition is initiated via the vision control system. External 3D surface imaging utilizes a DMD-free structured light encoding module based on diffraction optical phase modulation, replacing the DMD chip relied upon in traditional structured light systems. This module generates multi-frequency virtual phase-shifted fringes through off-axis interferometry and reconstructs these fringes using a trained neural network model.

[0037] This module implements wavefront modulation based on diffraction grating, and its phase function is expressed as:

[0038]

[0039] in, are the amplitudes of different diffraction orders, is the corresponding spatial frequency, is the initial phase;

[0040] Multi-level interference superposition realizes spatial phase modulation, adapts to the curvature of different components, and realizes structured light encoding without DMD conditions.

[0041] To enhance the imaging effect, the system uses a dual-frequency phase-shift projection structure, where the first frequency component is used for detail recovery and the second frequency component is used for global contour restoration. The fringe phase of the system projection pattern satisfies the following relationship:

[0042]

[0043] in, , are the amplitude coefficients of the dual-frequency components respectively; is the spatial frequency (unit: rad / mm), satisfying > ; is the phase shift angle, uniformly set to ;

[0044] After fringe projection, a high-frame-rate industrial camera captures the image, which is then subjected to image distortion correction and input into a neural network module to generate a phase shift map. The network model uses a lightweight multi-scale upsampling structure and is optimized using a structure-aware loss. The loss function is defined as follows:

[0045]

[0046] in, is the pixel-level reconstruction error; is the structural gradient difference; is the experience weight;

[0047] Simultaneously, the component's internal information is non-contact scanned using laser excitation and a surface-wave laser ultrasonic sensor. During the scanning process, the laser pulse excites the metal surface, generating a thermoelastic acoustic wave signal. The excitation wave propagates within the component, and the laser interferometer sensor receives the echo signal. This process can be modeled as a wave propagation problem in an elastic medium, where the propagation of the acoustic pressure P(x, t) satisfies the one-dimensional wave equation:

[0048]

[0049] in:

[0050]

[0051] For internal defects of three-dimensional components, the three-dimensional wave equation in elastic medium is used to model the sound field propagation.

[0052] After the echo signal is amplified by the pre-amplifier and sampled by the A / D converter, a three-dimensional internal point cloud image is reconstructed through weighted superposition to identify structural defects such as internal pores and unfused structures.

[0053] To improve subsequent defect detection efficiency and reduce computing resource consumption, the system introduces a region of interest (ROI) extraction strategy after image preprocessing. This strategy analyzes the edge gradient distribution and geometric feature aggregation of multimodal images to extract subregions that may contain abnormal structures or material changes, which serve as the primary detection targets of the SoftPatch network. The ROI extraction process includes the following steps:

[0054] Use Sobel operator to extract edge map , calculate the gradient magnitude map:

[0055]

[0056] right Perform morphological opening operation to remove isolated pixels and mark connected domains;

[0057] Calculate the area A and perimeter P of each connected region and introduce the compactness index:

[0058]

[0059] Where A represents the area of ​​the connected region and P represents its perimeter;

[0060] Set area threshold and the compactness threshold , filter out background interference areas and obtain a set of regions of interest , each region was used for subsequent patch analysis.

[0061] After image correction and scale normalization, the surface structured light 3D reconstruction and the internal laser ultrasonic point cloud are mapped to the same spatial coordinate system. To ensure point cloud data consistency, this embodiment uses a rigid iterative closest point (ICP) algorithm based on singular value decomposition (SVD) to perform point cloud registration.

[0062] There are no predefined matching point pairs between multimodal point clouds, so the iterative closest point (ICP) algorithm is used for registration. The steps are as follows:

[0063] 1. For each source point p_i, search for the nearest neighbor q_i in the target point cloud Q;

[0064] 2. Based on the current matching point pair, solve the optimal rigid transformation matrix (R, t) that satisfies:

[0065]

[0066] 3. Apply the transformation to update the source point cloud, repeating steps 1 and 2 until convergence;

[0067] The rigid transformation matrix of each iteration is solved by SVD decomposition.

[0068] The centralization matrices of the source and target point clouds are decomposed using singular value decomposition (SVD) to determine the optimal rotation matrix R and translation vector t. Applying this rigid transformation to the source point cloud data allows for precise alignment of the multimodal point cloud data within the same coordinate system.

[0069] After point cloud registration is completed, the surface defect area and the internal defect voxel map can be fused on a unified spatial structure to realize an intelligent defect map that integrates structure and function, providing spatial context information support for subsequent identification.

[0070] After completing image preprocessing, region of interest extraction, and spatial alignment, a recognition method based on the SoftPatch anomaly detection network is proposed to achieve high-precision defect recognition and classification. This method uses weakly supervised learning, relying only on normal samples for training. It detects abnormal regions through reconstruction errors, and exhibits high robustness and generalizability.

[0071] The SoftPatch network consists of four stages: feature encoding, patch library construction, soft-weighted reconstruction, and anomaly score map generation. Feature encoding can utilize a ResNet-18 or lightweight Unified Network architecture. The candidate patch library is constructed from regions of interest (ROIs) of normal samples in the training set, and feature matching is measured using cosine similarity. Compared to traditional AutoEncoder and GAN reconstruction methods, SoftPatch utilizes a multi-target soft matching strategy, improving robustness and detection accuracy for regions with blurred boundaries and weak intra-class variation.

[0072] Optional reconstruction network structures include but are not limited to: AutoEncoder (AE), Convolutional Autoencoder (CAE), AnoGAN based on generative adversarial network, Patch-SVDD, MemAE, etc.

[0073] Among them, the SoftPatch algorithm is preferred because it integrates diverse representations through the Patch library, improves the response capability to slight morphological changes, and is particularly suitable for scenarios in industrial practice where data is scarce and defect changes are complex.

[0074] In the patch characterization stage, the system builds a candidate patch library, randomly crops image patches from the region of interest of normal samples, and extracts feature vectors through the encoder. And store to form a normal feature dictionary.

[0075] A soft-weighted matching mechanism is introduced to select the region most similar to the input patch from the candidate library for reconstruction. The reconstruction strategy is defined as follows:

[0076]

[0077] in is the similarity weighting factor, defined as:

[0078]

[0079] in, is the vectorized feature of the input patch P_i, is the feature of the jth patch in the candidate patch library, is the temperature parameter, which is empirically set to 0.05.

[0080] The model loss function contains the pixel-level L2 error and the perceptual feature error:

[0081]

[0082] in, Extract network for perceptual features; is an empirical parameter;

[0083] In the inference phase, the system performs a patch reconstruction operation on each region of interest and records the reconstruction error of each patch to obtain an anomaly heat map. Through Gaussian smoothing and adaptive threshold segmentation, a binary defect mask is obtained.

[0084] The system maps the defect mask back to the original image coordinate system, combines the aforementioned structured light image and laser ultrasonic point cloud information, calculates the position coordinates, volume size, and boundary contour of the defect in a unified three-dimensional space, and performs severity analysis and grade scoring based on the reconstruction error distribution in the anomaly scoring map. It finally outputs the defect type classification, three-dimensional coordinates, geometric dimensions, and credibility level, achieving complete structured defect assessment and result visualization.

[0085] This method realizes the multimodal defect detection task of parts based on normal samples and has the following advantages:

[0086] Label-free training, adapting to data-scarce environments;

[0087] Supports highly robust detection of multi-scale, weak texture, and complex background;

[0088] The modular structure is lightweight and can be adapted to the deployment of edge industrial equipment;

[0089] It works in conjunction with the 3D reconstruction system to achieve integrated defect detection and visualization.

[0090] The SoftPatch anomaly detection module, one of the core recognition algorithms of this invention, uses a soft-weighted matching mechanism to effectively improve the recognition of hidden defects such as blurred edges and slight morphological disturbances. It also has better regional focus and model convergence than traditional reconstruction networks.

[0091] Through the description of the above embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform. Of course, they can also be implemented by hardware, but in many cases the former is a better embodiment. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product. The computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal (which can be a mobile phone, computer, server or network device, etc.) to execute the methods of the various embodiments of the present application.

[0092] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of this application, ordinary technicians in this field can also make many forms without departing from the purpose of this application and the scope of protection of the claims, all of which are protected by this application.

Claims

1. A detection method for intelligent visual detection of parts, characterized in that: The detection method comprises: S101: Acquire multimodal image data of a target component, including surface image data acquired by diffraction optical structured light imaging and internal structure image data acquired by laser ultrasonic scanning; S102: performing geometric registration and scale normalization processing on the multimodal image data to generate fused image data in a unified coordinate system; S103: performing image preprocessing and edge analysis on the fused image data to extract a region of interest; S104: Divide the region of interest into multiple image patches and input them into a SoftPatch anomaly detection network built based on a weakly supervised image reconstruction mechanism; S105: Perform feature extraction and matching reconstruction operations on each image block through the SoftPatch network, calculate the reconstruction error between the original image block and the reconstructed image block, and generate an anomaly score map; S306: performing image processing operations on the abnormality score image, extracting the defect mask area, and combining the original image with the three-dimensional structure information to output the defect category, spatial position, geometric size and severity level.

2. The method according to claim 1, characterized in that In step S104, the anomaly detection network is a SoftPatch weakly supervised image reconstruction network, which includes an encoder for extracting feature vectors of image blocks and simultaneously receiving input features after the surface image and the internal image are fused.

3. The method according to claim 1, characterized in that In step S105, the image block reconstruction process includes a reconstruction operation based on weighted feature matching, which reconstructs the image block. Defined as: ； Among them, the weight The calculation method is: ； in, is the feature vector of the input image block, The first The feature vector of the image block, is the temperature parameter.

4. The method according to claim 1, wherein The anomaly score map The calculation method is: ； in is the original image pixel value, is the reconstructed image pixel value.

5. The method according to claim 1, wherein The training process of the SoftPatch anomaly detection network uses the following joint loss function: ； in, represents the perceptual feature extraction function, , is the loss weight parameter used in the training phase.

6. The method according to claim 1, characterized in that The step of extracting the region of interest comprises: Calculate the gradient magnitude map of the image ; right Perform connected domain analysis; The compactness C is calculated based on the area A of the connected domain and the perimeter P of the connected domain. The compactness C is defined as follows: ； Filter the region of interest by setting the area threshold and compactness threshold.

7. The method according to claim 1, characterized in that The geometric registration and scale normalization processing includes: Perform a rigid iterative closest point algorithm on the surface image and the internal structure image; Optimizing the Rigid Transformation Matrix Based on Minimum Mean Square Error ; Map point cloud data uniformly to the same coordinate system.

8. The method according to claim 1, characterized in that The defect assessment includes: performing an image processing operation based on the anomaly score map to extract a defect mask region; Map the defect mask area back to the original image coordinate system; Combined with the registration parameters of multimodal images, the position of defects in the three-dimensional space of the component is dynamically calculated; Severity assessment and credibility scoring are performed on each defect area based on reconstruction error distribution and structural feature indicators.

9. A device for intelligent visual inspection of parts, characterized in that: include: A data acquisition module for collecting surface images and internal structure images, for acquiring multimodal image data of target parts, including surface image data acquired by diffraction optical structured light imaging and internal structure image data acquired by laser ultrasonic scanning; A registration processing module for performing image registration and normalization processing, for performing geometric registration and scale normalization processing on the multimodal image data to generate fused image data in a unified coordinate system; An image analysis module for extracting candidate regions of interest, configured to perform image preprocessing and edge analysis on the fused image data to extract the regions of interest; The anomaly detection subsystem includes a SoftPatch network structure that performs image reconstruction and scoring. It is used to perform feature extraction and matching reconstruction operations on each image block through the SoftPatch network, calculate the reconstruction error between the original image block and the reconstructed image block, and generate an anomaly score map; The defect information evaluation module is used to perform image processing operations on the abnormality score map, extract the defect mask area, and output the defect category, spatial position, geometric size and severity level by combining the original image with the three-dimensional structure information.

10. A computer-readable storage medium having a computer program stored thereon, wherein the program is used to implement the method according to any one of claims 1 to 8 when executed by a processor.

Images