A method, system, product, and medium for defect detection of precision components.

Abstract

A method, system, product, and medium for defect detection of precision parts are disclosed, relating to the field of intelligent sensors. The aim is to address the difficulty in distinguishing between unstructured surface attachments and actual geometric damage in related 3D vision solutions. The method involves controlling a robotic arm to drive multi-source sensors to collaboratively acquire two-dimensional texture data and initial three-dimensional point cloud data, and automatically supplementing missing areas to generate complete three-dimensional point cloud data. Subsequently, two-dimensional feature vectors containing local texture information and three-dimensional feature vectors containing spatial geometric information are extracted and fused to generate a target fused feature vector possessing both texture and geometric attributes. Finally, the feature distance between this vector and a normal sample feature library is calculated as the deviation to determine the defect. This application utilizes semantic mutual verification of multimodal features to reduce the false alarm rate in complex scenarios, achieving high-precision and interference-resistant detection of subtle defects in precision parts.

Description

Technical Field

[0001] This application relates to the field of intelligent sensors, and in particular to a method, system, product, and medium for defect detection of precision parts. Background Technology

[0002] As the precision manufacturing industry moves towards miniaturization and complexity, the quality control requirements for injection molds and precision components (such as optical mold cores and mobile phone structural parts) are becoming increasingly stringent, often involving deep cavities, irregularly shaped curved surfaces, and high-gloss surfaces. During the production process, these workpieces are prone to defects such as microcracks, material shortages, and chipped corners due to the molding process and material properties. Traditional contact-based measurement is inefficient and easily scratches surfaces, while manual visual inspection is limited by visual fatigue and subjective standards, making it difficult to guarantee the stability and consistency of micron-level defect detection. Therefore, automated non-destructive testing technology based on machine vision has become the main means of quality control in this field.

[0003] Existing automated inspection solutions typically employ 3D vision systems based on line laser profilometers. Their working principle involves fixing a line laser projector and an industrial camera at a specific angle, and using a high-precision motorized translation stage to move the workpiece under test along a set direction for uniform linear scanning. Based on the principle of laser triangulation, the system calculates depth information by capturing the deformation of laser stripes on the workpiece surface, thereby reconstructing a 3D point cloud model of the workpiece surface. Subsequently, the inspection algorithm mainly uses template matching or geometric comparison methods to register and differencing the scanned point cloud data with a standard CAD model or a pre-scanned "gold sample" point cloud. By calculating the geometric deviations (such as height difference and volume difference) within a set threshold range, it determines whether the workpiece has dimensional errors or morphological defects.

[0004] However, related technical solutions primarily rely on the absolute deviation of spatial physical coordinates to characterize anomalies. When non-structural deposits (such as transparent oil stains or water stains) exist on the workpiece surface, or when false noise is generated by multiple reflections from the inner wall of deep cavities, these interfering factors often exhibit elevation fluctuations or geometric deviations in 3D data similar to real defects (such as minor dents or shallow scratches). Related technologies often struggle to effectively distinguish such non-structural anomalies from real geometric damage, thus increasing the false alarm rate in complex detection scenarios and reducing the accuracy of identifying subtle defects. Summary of the Invention

[0005] This application provides a method, system, product, and medium for defect detection of precision parts, which improves the accuracy and reliability of detecting complex defects in precision parts.

[0006] Firstly, this application provides a defect detection method for precision parts, applied to a defect detection system. The defect detection system includes at least a multi-source sensor, an industrial endoscope, and a robotic arm. The method includes: controlling the robotic arm to drive the multi-source sensor to collaboratively acquire data from the precision parts to be inspected, obtaining two-dimensional image data and initial three-dimensional point cloud data along a preset first scanning path. The two-dimensional image data includes an image representing the texture of the outer surface of the parts and an image representing the texture of the inner wall of the parts. The initial three-dimensional point cloud data is used for missing data detection. If missing areas exist, the robotic arm is controlled to perform supplementary measurements on the precision parts to be inspected to obtain complete three-dimensional point cloud data. The three-dimensional point cloud data includes a point cloud representing the overall shape of the parts and points obtained from supplementary measurements of deep holes or occluded areas. The process involves: extracting local point clouds from 2D image data; extracting 2D feature vectors containing local texture information from 2D image data; extracting 3D feature vectors containing spatial geometric information from 3D point cloud data; fusing the 2D and 3D feature vectors to generate a target fused feature vector, which possesses both texture and geometric attributes; calculating the feature distance between the target fused feature vector and a reference fused feature vector in the normal sample feature library, using the feature distance as the feature deviation; storing representative reference fused feature vectors extracted from defect-free samples and filtered; generating defect warning information when the feature deviation exceeds a preset anomaly judgment threshold, and determining the target defect region based on the mapping position of the target fused feature vector in the semantic space.

[0007] By adopting the above technical solution, the defect detection equipment first controls a robotic arm to drive multiple source sensors for collaborative data acquisition, breaking the data limitations of a single perspective or single modality, and acquiring two-dimensional images containing internal and external surface textures and an initial three-dimensional point cloud representing the overall morphology. Secondly, through missing detection and automatic supplementation, the defect detection equipment fills in blind spots caused by deep holes or occlusions, constructing a complete three-dimensional digital model, providing a comprehensive data foundation for subsequent analysis. Next, the defect detection equipment extracts and fuses texture and geometric features, utilizing the combined attributes of the target fusion feature vector to mutually verify texture appearance and spatial depth information, distinguishing between non-structural attachments on the surface (texture anomalies only) and real physical damage (texture and geometry anomalies). Finally, the defect detection equipment employs a metric learning method based on a normal sample feature library, using feature deviation as a criterion, enabling the identification of unknown anomalies without relying on a large number of defect samples for training. In summary, this solution improves the accuracy and reliability of complex defect detection in precision parts through deep fusion and completeness assurance of multimodal data.

[0008] In conjunction with some embodiments of the first aspect, in some embodiments, missing data detection is performed on the initial 3D point cloud data. If missing areas exist, the robotic arm is controlled to perform supplementary measurements on the precision parts to be inspected to obtain complete 3D point cloud data. Specifically, this includes: performing connectivity and density analysis on the initial 3D point cloud data to identify missing areas and recording the spatial coordinates of the missing areas in a unified coordinate system; generating a second scanning path based on the spatial coordinates; controlling the robotic arm to carry a point laser displacement sensor or an industrial endoscope to perform point-to-point supplementary measurements on the missing areas along the second scanning path to obtain locally completed data; and performing point cloud registration and stitching between the initial 3D point cloud data and the locally completed data to generate complete 3D point cloud data.

[0009] By employing the above technical solution, the defect detection equipment first performs connectivity and density analysis on the initial 3D point cloud data, accurately locating the missing areas in the point cloud caused by occlusion or deep holes from a mathematical and statistical perspective, avoiding blind global retesting. Secondly, based on the recorded spatial coordinates, the defect detection equipment generates a targeted second scanning path and controls a robotic arm carrying a point laser displacement sensor or industrial endoscope to perform point-to-point supplementary measurements, utilizing the high penetration of the point laser or the flexibility of the endoscope to obtain high-precision local completion data. Finally, the defect detection equipment rigorously registers and stitches the locally completed data with the initial 3D point cloud data, eliminating geometric errors at data seams. In summary, this solution, through the above closed-loop logic, solves the data void problem of traditional line scanning in complex structural components, improving the topological integrity and geometric accuracy of 3D point cloud data during defect detection.

[0010] In conjunction with some embodiments of the first aspect, in some embodiments, the image representing the texture of the outer surface of the component in the two-dimensional image data is a global RGB image, and the image representing the texture of the inner wall of the component is a local endoscope image; a two-dimensional feature vector containing local texture information is extracted from the two-dimensional image data, specifically including: constructing feature extraction branches for the global RGB image and the local endoscope image respectively, extracting a first feature map and a second feature map, and aligning them to the same channel dimension; calculating the channel importance weights of the first feature map and the second feature map respectively using a channel attention mechanism, dynamically exchanging channel data in a preset proportion according to the difference between the weights, and obtaining the first intermediate feature and the second intermediate feature after channel interaction; generating a spatial mask based on the feature response value of the second feature map, using the spatial mask to locate the effective region of the local endoscope image in global coordinates, and fusing the second intermediate feature into the effective region of the first intermediate feature, and obtaining a two-dimensional feature vector after dimensionality reduction processing.

[0011] By employing the aforementioned technical solution, the defect detection device first constructs feature extraction branches for both the global RGB image and the local endoscopic image, and establishes a unified dimension for cross-scale feature interaction through channel alignment. Secondly, the device utilizes a channel attention mechanism to calculate weight differences, driving dynamic channel exchange. This allows global macroscopic structural features and internal cavity microtexture features to complement each other at the feature channel level, enhancing the representation ability of hidden defects within the cavity. Finally, the device generates a spatial mask based on the feature response values ​​of the endoscopic image, precisely defining the effective range of the local image in the global coordinate system, and directionally fusing the second intermediate feature into this effective region. In summary, this solution, through a dual interaction mechanism of channels and space, solves the semantic fragmentation problem caused by varying scales in multi-view image stitching, improves the semantic integrity of the two-dimensional feature vector, and achieves a complete representation of the internal and external surface details of the precision parts to be inspected.

[0012] In conjunction with some embodiments of the first aspect, in some embodiments, the two-dimensional feature vector and the three-dimensional feature vector are fused to generate a target fused feature vector. Specifically, this includes: projecting the two-dimensional feature vector and the three-dimensional feature vector onto a semantic embedding space of the same dimension to obtain a two-dimensional embedding vector and a three-dimensional embedding vector; using the two-dimensional embedding vector as a query vector and the three-dimensional embedding vector as a key vector, calculating the attention response of the two-dimensional modality to the three-dimensional geometric information to generate a first fused component; using the three-dimensional embedding vector as a query vector and the two-dimensional embedding vector as a key vector, calculating the attention response of the three-dimensional modality to the two-dimensional texture information to generate a second fused component; and weighted concatenation of the first fused component and the second fused component to generate the target fused feature vector.

[0013] By employing the above technical solution, the defect detection device first projects heterogeneous 2D texture features and 3D geometric features into the same semantic embedding space, eliminating the dimensional gap between modalities. Secondly, using the 2D embedding vector as the query vector, the device retrieves the corresponding spatial structure in the 3D geometric information to generate the first fusion component, endowing the planar texture with depth perception capabilities. Simultaneously, it uses the 3D embedding vector as the query vector to associate with 2D texture details, generating the second fusion component, endowing the geometric shape with material context. Finally, the device concatenates the fusion components from both directions. In summary, this solution establishes a deep semantic mapping relationship between texture information and geometric information through a bidirectional cross-attention mechanism, ensuring that the generated feature vectors not only contain independent information of their respective modalities but also include correlation information for mutual verification between modalities, thus improving the robustness of the feature fusion process.

[0014] In conjunction with some embodiments of the first aspect, in some embodiments, the first fusion component and the second fusion component are weighted and concatenated to generate a target fusion feature vector. Specifically, this includes: performing element-wise multiplication on the first fusion component and the second fusion component to obtain a mutual activation vector representing the semantic consistency of texture and geometric features; mapping the mutual activation vector to a consistency weight matrix with values ​​within a preset interval using a preset activation function; performing element-wise weighted operations on the first fusion component and the second fusion component with the consistency weight matrix respectively to obtain a weighted first fusion component and a weighted second fusion component; and concatenating the weighted first fusion component and the weighted second fusion component to generate the target fusion feature vector.

[0015] By employing the above technical solution, the defect detection device first performs element-wise multiplication on the first and second fusion components. Utilizing the gating property of multiplication, it highlights regions in the mutual activation vector that match the texture and geometric feature responses (i.e., the true defect regions), while suppressing regions with conflicting responses (such as water stains or reflections). Next, the defect detection device maps the mutual activation vector to a consistency weight matrix using a preset activation function, quantifying the reliability of different feature dimensions. Finally, the defect detection device uses this weight matrix to adaptively weight and concatenate the two fusion components. In summary, this solution, through consistency verification logic, dynamically suppresses noise interference in a single modality, improving the representation of common defect features across multiple modalities in the target fusion feature vector.

[0016] In conjunction with some embodiments of the first aspect, in some embodiments, the target defect region is determined based on the mapping position of the target fused feature vector in the semantic space. Specifically, this includes: reconstructing the feature deviation into an initial anomaly scoring matrix based on the mapping position; determining the spatial neighborhood range with any feature point in the initial anomaly scoring map as the center, and performing Gaussian weighted average calculation on the feature deviation within the spatial neighborhood range to obtain a smooth anomaly score, which is then processed by row normalization; when the smooth anomaly score is greater than a preset defect segmentation threshold, marking the corresponding position as a defect identifier value to generate a binary defect mask; extracting the connected regions composed of defect identifier values ​​in the defect mask, and determining the range covered by the connected regions as the target defect region.

[0017] By employing the above technical solution, the defect detection device first utilizes the mapping position of the target fusion feature vector in the semantic space to inversely reconstruct the abstract high-dimensional feature deviation into an intuitive initial anomaly scoring matrix, establishing a mapping from the feature space to the physical space. Secondly, the defect detection device introduces a Gaussian weighted average of the spatial neighborhood range, using the correlation between neighboring pixels to smooth isolated noise responses and improve the continuity of the scoring map. Finally, the defect detection device generates a binary mask based on the defect segmentation threshold and uses connected component extraction technology to filter out fragmented artifacts, locking down the true defect range with physical dimensions. In summary, this solution infers spatial location from semantic anomalies and combines it with morphological processing to achieve accurate segmentation and localization of minute defects under unsupervised conditions.

[0018] In conjunction with some embodiments of the first aspect, in some embodiments, after calculating the feature distance between the target fused feature vector and the reference fused feature vector in the normal sample feature library, the method further includes: when the feature distance is less than a preset drift update threshold, performing a weighted update operation on the reference fused feature vector based on the currently generated target fused feature vector; the value of the drift update threshold is less than the anomaly determination threshold; the weighted update operation specifically includes: using a preset time decay coefficient to perform a weighted average calculation on the target fused feature vector and the reference fused feature vector to obtain the updated reference fused feature vector, and using the updated reference fused feature vector to replace the reference fused feature vector in the normal sample feature library.

[0019] By employing the above technical solution, the defect detection equipment calculates the feature distance after each detection and sets a drift update threshold that is less than the anomaly judgment threshold, thereby filtering out samples that are normal but on the verge of change. Then, for these samples, the defect detection equipment performs a weighted update operation on the normal sample feature library using a time decay coefficient, allowing the reference feature vector to slowly approximate the data distribution under the current production environment. Finally, the defect detection equipment replaces the original reference vector with the updated vector. In summary, this solution enables the normal sample feature library to adapt to the slow concept drift of data distribution caused by injection mold wear, light source attenuation, or changes in ambient temperature, avoiding an increase in false alarm rate due to subtle environmental changes and ensuring the long-term stability of the system.

[0020] In a second aspect, this application provides a defect detection system, which includes: one or more processors and a memory; the memory is coupled to one or more processors, and the memory is used to store computer program code, which includes computer instructions, and the one or more processors call the computer instructions to cause the defect detection system to perform the method described in the first aspect and any possible implementation thereof.

[0021] Thirdly, this application provides a computer-readable storage medium storing computer instructions that, when executed on a defect detection system, cause the defect detection system to perform the method described in the first aspect and any possible implementation thereof.

[0022] Fourthly, this application provides a computer program product, including a computer program / instructions that, when run on a defect detection system, cause the defect detection system to perform the method described in the first aspect and any possible implementation thereof.

[0023] Understandably, the defect detection system provided in the second aspect, the computer-readable storage medium provided in the third aspect, and the computer program product provided in the fourth aspect are all used to execute the methods provided in the embodiments of this application. Therefore, the beneficial effects they can achieve can be referred to the beneficial effects in the corresponding methods, and will not be repeated here.

[0024] One or more technical solutions provided in the embodiments of this application have at least the following technical effects or advantages:

[0025] 1. By employing a technical solution that fuses two-dimensional and three-dimensional feature vectors to generate a target fused feature vector with both texture and geometric attributes, and calculating the feature distance between this target fused feature vector and the reference fused feature vector in the normal sample feature library as the feature deviation, the surface texture information and spatial depth information of the parts can mutually corroborate and complement each other at the feature semantic level. This effectively solves the problem of high false alarm rate caused by relying solely on three-dimensional geometric deviation in related technologies, which makes it difficult to distinguish between non-structural attachments (such as water stains and oil stains) and real physical damage (such as micro-scratches and shallow dents). Thus, it achieves high-precision and high-robust recognition of minor defects in precision parts under complex interference scenarios.

[0026] 2. By adopting a channel attention-based mechanism for dynamic exchange of channel data and using spatial masks to fuse local endoscopic image features into the global effective region, the precise complementarity and seamless embedding of microscopic texture details of the cavity on key channels with global macroscopic structural features are achieved. This effectively solves the problem of semantic fragmentation or loss of key hidden area information caused by resolution differences and spatial misalignment in multi-scale and multi-view images in related technologies. As a result, it enables the generation of two-dimensional feature vectors that contain complete details of the inner and outer surfaces and have anti-interference capabilities.

[0027] 3. By adopting a technical solution that projects two-dimensional and three-dimensional feature vectors into a unified semantic embedding space and performs bidirectional cross-attention calculation (calculating attention responses with each other as key-value vectors), a one-to-one mapping relationship between heterogeneous modal data in the semantic layer is established. This enables texture features to actively perceive the corresponding spatial structure and geometric features to be associated with the corresponding appearance material. This effectively solves the problems of physical meaning mismatch and insufficient information interaction between modalities caused by directly splicing heterogeneous features in related technologies. As a result, it realizes the generation of target fusion feature vectors with both texture and geometric attributes that have high consistency and strong representation capabilities. Attached Figure Description

[0028] Figure 1 This is a flowchart illustrating a defect detection method for precision parts in an embodiment of this application;

[0029] Figure 2 This is another flowchart illustrating a defect detection method for precision parts in an embodiment of this application;

[0030] Figure 3 This is a schematic diagram of the physical device structure of a defect detection system in the embodiments of this application. Detailed Implementation

[0031] The terminology used in the following embodiments of this application is for the purpose of describing particular embodiments only and is not intended to be limiting of this application. As used in the specification of this application, the singular expressions “a,” “an,” “the,” “the,” and “this” are intended to include the plural expressions as well, unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in this application refers to any or all possible combinations including one or more of the listed items.

[0032] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as implying or suggesting relative importance or implicitly indicating the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature, and in the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more.

[0033] Target Fusion Feature Vector: This refers to the core feature representation of this application, a high-dimensional numerical vector generated by a deep learning network that combines texture and geometry attributes. It is not a simple linear concatenation of two-dimensional image features and three-dimensional point cloud features, but rather generated through semantic interaction using a multimodal fusion algorithm. Each dimension of this vector contains information about the appearance (e.g., color, pattern) and structure (e.g., depth, curvature) of the component at a specific location. Its main function is to serve as a unique "fingerprint" for defect determination, distinguishing between defects that are abnormal only in texture (e.g., oil stains) but normal in geometry, and defects that are abnormal in both texture and geometry (e.g., scratches, dents).

[0034] The Normal Sample Feature Library (MSL) is a database used to store representative features of defect-free samples (i.e., "good products"), and it forms the basis for unsupervised or few-sample defect detection in this application. Unlike traditional classification networks that require a large number of defect samples for training, this application adopts a metric learning approach, pre-extracting and storing the most representative fused feature vectors from a large amount of good product data through a selection algorithm (such as core-set sampling). During the detection phase, it acts as a "ruler," and the system determines whether an anomaly is present by calculating the distance between the features of the workpiece to be tested and the features in the library, thereby enabling it to identify novel, unseen defects.

[0035] Synergistic Acquisition & Point-specific Rescanning: This refers to the data acquisition strategy unique to this application. Synergistic acquisition involves controlling a robotic arm to work in conjunction with an RGB camera, a 3D laser profilometer, and an endoscope within a unified coordinate system, simultaneously acquiring both external and internal data. Point-specific rescanning addresses the common "blind spot" problem in 3D scanning (point cloud gaps caused by deep holes or occlusions). The system automatically analyzes point cloud density, intelligently plans a path, and controls the robotic arm carrying a point laser or endoscope to perform a secondary high-precision scan of the missing area. This mechanism ensures the integrity of the input data and avoids missed detections due to missing data.

[0036] For ease of understanding, the method provided in this implementation is described in process below. Please refer to [link / reference]. Figure 1 This is a flowchart illustrating a defect detection method for precision parts in an embodiment of this application.

[0037] S101. Control the robotic arm to drive the multi-source sensors to collect data in a coordinated manner on the precision parts to be inspected. Acquire two-dimensional image data and initial three-dimensional point cloud data along the preset first scanning path. The two-dimensional image data includes images representing the texture of the outer surface of the parts and images representing the texture of the inner cavity wall of the parts.

[0038] The robotic arm refers to an industrial robotic arm used to perform sensor positioning, path movement, and scanning actions. Through its multi-degree-of-freedom joint control, it enables precise movement and attitude adjustment of sensors in three-dimensional space. Multi-source sensors refer to a collection of various sensing devices, including RGB industrial cameras, 3D laser contour sensors, laser displacement sensors, and industrial endoscopes. These devices achieve complementary data acquisition of precision parts to be inspected through different physical imaging principles. The precision parts to be inspected refer to plastic injection mold parts that require defect detection. Their structures typically include complex geometric features such as outer surfaces, deep holes, internal cavities, and obstructed areas. Collaborative acquisition refers to the process by which multiple source sensors synchronously or sequentially acquire different modal data according to a predetermined time sequence and spatial position relationship within a unified coordinate system, ensuring that the acquired data has spatial correspondence and temporal consistency. The preset first scanning path refers to the sensor motion trajectory pre-planned based on the 3D model or structural features of the precision parts to be inspected. This path must cover all visible surfaces and main inspection areas of the parts to ensure the integrity of the data acquisition. Two-dimensional image data refers to planar image data containing pixel grayscale values ​​or color information, acquired through RGB industrial cameras and industrial endoscopes, used to characterize surface features such as texture, color, and cracks in parts. Initial three-dimensional point cloud data refers to a set of three-dimensional discrete points containing spatial coordinates (x, y, z), acquired line by line along the scanning path by a 3D laser contour sensor, used to characterize the overall external shape of parts, but data may be missing in areas such as deep holes and occlusions.

[0039] After the defect detection process is initiated, the defect detection equipment first needs to acquire complete multimodal raw data of the precision component to be inspected. Specifically, the defect detection equipment controls a robotic arm to fix the precision component to be inspected onto a fixture or rotating platform, establishing a unified workpiece coordinate system so that the data collected by all subsequent sensors can establish a correspondence under the same spatial reference system. Subsequently, the defect detection equipment loads the first scanning path parameters pre-planned based on the component's CAD model or historical scanning experience. This path comprehensively considers factors such as the component's geometric complexity, sensor field of view, and occlusion avoidance strategies. During path execution, the defect detection equipment controls the robotic arm to drive multiple source sensors to collect data collaboratively according to the set speed and posture parameters: an RGB industrial camera takes multi-angle pictures of the component's outer surface from the top or side, acquiring high-resolution color images containing complete appearance information; a 3D laser contour sensor uses a linear array scanning method, collecting height data line by line as the robotic arm moves laterally, accumulating to generate initial three-dimensional point cloud data covering the overall external contour of the component; an industrial endoscope is guided by the robotic arm into areas of the component such as deep holes and cavities that cannot be directly observed by external cameras, acquiring visible light images of the internal walls.

[0040] S102. Perform missing detection on the initial three-dimensional point cloud data. If there are missing areas, control the robotic arm to perform supplementary measurements on the precision parts to be inspected in order to obtain complete three-dimensional point cloud data. The three-dimensional point cloud data includes point clouds that characterize the overall shape of the parts and local point clouds obtained by supplementary measurements on deep holes or occluded areas.

[0041] Among them, supplementary measurement refers to the process of controlling a robotic arm to carry a laser displacement sensor or other supplementary measurement equipment to perform high-precision point-by-point or line-by-line measurements on the identified missing areas according to a targeted local scanning path, so as to obtain the three-dimensional coordinate information of the missing areas.

[0042] After initial data acquisition is completed in step S101, due to the limitations of the imaging principle of the 3D laser contour sensor, the defect detection equipment needs to assess the integrity of the initial 3D point cloud data and perform targeted supplementary measurements for missing parts. Specifically, the defect detection equipment first executes a missing detection algorithm on the initial 3D point cloud data acquired in step S101, including: dividing the point cloud into regular grids or constructing local neighborhoods based on KNN, and calculating the point cloud density within each grid or neighborhood; comparing the actual density with the theoretical density calculated based on scanning parameters or the average density statistically analyzed from normal samples; when the actual density is lower than a set threshold (e.g., 30% of the theoretical density), the area is marked as a missing area; simultaneously, combined with the CAD model information of the parts, areas that are geometrically prone to occlusion, such as deep holes and slits, are identified. If the missing detection result indicates the existence of a missing area, the defect detection equipment enters the supplementary measurement process: based on the spatial location and geometric characteristics of the missing area, a supplementary measurement path is planned, which needs to ensure that the ranging beam of the laser displacement sensor can illuminate the surface of the missing area perpendicularly or at a suitable angle; the robotic arm is controlled to carry the laser displacement sensor to scan point by point according to the supplementary measurement path. After the supplementary measurements are completed, the defect detection equipment performs coordinate registration and fusion between the supplementary local point cloud data and the initial 3D point cloud data. The registration process is based on the kinematic parameters of the robotic arm and the pre-calibrated sensor coordinate system transformation relationship to ensure that the two parts of the point cloud are accurately aligned in a unified coordinate system. Ultimately, the defect detection equipment obtains complete 3D point cloud data, including the overall shape of the component and the supplementary local point cloud data for deep holes or occluded areas. This data does not contain any critical missing areas that would affect subsequent defect detection in terms of spatial coverage.

[0043] S103. Extract a two-dimensional feature vector containing local texture information from the two-dimensional image data, and extract a three-dimensional feature vector containing spatial geometric information from the three-dimensional point cloud data.

[0044] Among them, the two-dimensional feature vector refers to the high-dimensional numerical vector extracted by deep learning network encoding of two-dimensional image data. Local texture information refers to the detailed features of local areas in the two-dimensional image, such as texture patterns, edge gradients, color distribution, and crack morphology. This information can characterize the microscopic defects of the surface and inner wall of the parts. The three-dimensional feature vector refers to the high-dimensional numerical vector extracted by deep learning network encoding of three-dimensional point cloud data. Spatial geometric information refers to the geometric attributes contained in the three-dimensional point cloud data, such as curvature, normal vector, local concavity and convexity, and overall contour shape. This information can characterize the geometric defects of the parts, such as structural deformation, excess material, and missing material.

[0045] After obtaining the complete multimodal raw data in step S102, the defect detection device needs to convert the original two-dimensional image data and three-dimensional point cloud data into high-level semantic feature representations suitable for feature comparison and defect determination. Specifically, the defect detection device first extracts features from the two-dimensional image data: the image representing the texture of the outer surface of the component is input into a pre-trained DINOv2-ViT-S / 14 visual Transformer network. Of course, other self-supervised learning models or convolutional neural network models capable of extracting general visual representations can also be used. This network learns general visual feature representation capabilities on large-scale data through self-supervised learning, enabling it to capture the shape boundaries, overall contours, and texture distribution of images. Images representing the texture of the inner wall of components are input into a lightweight ViT network, which focuses on extracting local high-frequency detail features, including microscopic defects such as cracks, burrs, and missing material edges. Two branches output feature maps of shape C×H×W. After channel alignment via 1×1 convolution, a channel-space bidirectional interactive fusion module (CSIF) is used for cross-modal information interaction. This module achieves semantic independence of key channels and complementarity of non-key channels through channel-level adaptive interaction, and performs feature fusion in high-response regions through spatial-level region selection interaction. Finally, after refinement operations such as adaptive batch normalization and channel compression, a two-dimensional feature vector with both global structural and local detail representation capabilities is obtained. In parallel, the defect detection device performs feature extraction on the 3D point cloud data: the complete 3D point cloud data obtained in step S102 is input into the Point-MAE network based on the mask autoencoder structure. This network divides the point cloud into multiple local patches using the FPS and KNN algorithms. After randomly masking 60%-80% of the patches, only the visible patches are encoded using Transformer. The network learns the global structural relationship and local geometric details of the point cloud through the reconstruction task, making the network more sensitive to geometric defects such as missing material, depressions, and protrusions. The high-dimensional feature vector output by Point-MAE serves as a 3D feature vector containing spatial geometric information.

[0046] Optionally, in some embodiments, the defect detection device can perform pseudo-depth enhancement processing on the local endoscopic image characterizing the texture of the internal cavity wall of the component. Since monocular endoscopes lack depth information, the defect detection device utilizes the principles of photometric stereo vision or a deep learning-based monocular depth estimation network to recover the relative depth map of the internal cavity wall from the shadows and brightness variations in the endoscopic image. This relative depth map is then stitched as an additional feature channel to the RGB channels of the local endoscopic image, forming RGB-D four-channel input data.

[0047] S104. Fuse the two-dimensional feature vector and the three-dimensional feature vector to generate the target fused feature vector, which has both texture and geometric attributes.

[0048] Among them, the target fusion feature vector refers to the comprehensive high-dimensional numerical vector generated after processing by the multimodal fusion strategy. This vector serves as the final feature representation of the precision parts to be inspected and is used for subsequent comparison with normal samples and defect determination.

[0049] After obtaining the two-dimensional and three-dimensional feature vectors in step S103, the defect detection device needs to effectively fuse the features of these two heterogeneous modalities. Specifically, the defect detection device first performs unified semantic encoding on the two-dimensional and three-dimensional feature vectors, and projects them onto a unified feature space of the same dimension through two lightweight mapping encoders to ensure the feasibility of subsequent cross-modal interaction. Subsequently, the defect detection device uses a cross-modal Transformer interaction fusion module (CMTF) for bidirectional information interaction: in the first interaction direction, the two-dimensional feature vector is used as the query and the three-dimensional feature vector is used as the key / value, and the association between texture features and geometric context is learned through a cross-attention mechanism, enabling the two-dimensional features to perceive the corresponding spatial structure information; in the second interaction direction, the three-dimensional feature vector is used as the query and the two-dimensional feature vector is used as the key / value, enabling the geometric features to be associated with the corresponding texture appearance information; each interaction direction also includes a self-attention layer to strengthen the internal correlation of each modality.

[0050] After obtaining the two branch features after interaction, the defect detection device adaptively calculates the importance weights of the two-dimensional and three-dimensional modalities through a dynamic weight prediction network. These weights are dynamically adjusted based on the reliability of texture and geometric information in the current detection task; for example, the weight of geometric attributes is increased when the point cloud quality is high, and the weight of texture attributes is increased when the texture contrast is significant. Finally, the defect detection device performs a weighted summation of the two branch features according to the predicted dynamic weights, generating a target fusion feature vector that simultaneously possesses both texture and geometric attributes.

[0051] Optionally, in some embodiments, to further enhance the fusion effect, the defect detection device introduces a contrastive learning loss in the form of InfoNCE, which makes the two-dimensional features and three-dimensional features of the same local region closer in semantic space, and the features of different regions more separated, thereby improving the model's ability to focus on defect regions.

[0052] Optionally, in some embodiments, the defect detection device may incorporate a feature refinement process based on a multi-scale feature pyramid (FPN). Before generating the target fused feature vector, the two-dimensional and three-dimensional feature vectors are mapped to multi-level pyramid structures, corresponding to the macroscopic contour, mesoscopic components, and microscopic texture of the parts, respectively. Cross-modal interaction is performed at each level of the pyramid, and the fusion results from each level are upsampled and stitched together. This ensures that the target fused feature vector not only contains high-level semantics but also retains the low-level high-resolution detail features used to identify minute scratches.

[0053] S105. Calculate the feature distance between the target fusion feature vector and the reference fusion feature vector in the normal sample feature library, and use the feature distance as the feature deviation. The normal sample feature library stores representative reference fusion feature vectors extracted from defect-free samples and screened.

[0054] Feature distance refers to the similarity measure between the target fused feature vector and the reference fused feature vector in a high-dimensional feature space, typically using Euclidean distance or cosine distance. Feature deviation refers to the degree of deviation of the target fused feature vector of the precision component under inspection from the feature distribution of normal samples, using feature distance as a quantitative indicator. The larger the value of this indicator, the more likely the precision component under inspection is to have anomalies or defects. The normal sample feature library is a database structure that stores representative feature vectors of normal, defect-free samples. This library is built on a Memory Bank architecture and is used to characterize the distribution range and typical patterns of normal samples in the feature space. The reference fused feature vector is the set of fused feature vectors stored in the normal sample feature library. These vectors are extracted from a large number of defect-free samples and filtered using a Core-set Sampling strategy to ensure maximum coverage of the feature distribution space of normal samples with a smaller number of features.

[0055] After generating the target fusion feature vector in step S104, the defect detection equipment needs to quantify the degree of abnormality of the precision parts to be inspected by comparing it with the established normal sample feature distribution. Specifically, the defect detection equipment first retrieves a set of reference fused feature vectors from a pre-built normal sample feature library, which was established during the training phase before system deployment. A large number of defect-free samples are collected, and for each sample, steps S101 to S104 are executed to extract its fused feature vector. These feature vectors are then uniformly normalized using L2 to eliminate differences in feature amplitude. For the extracted large-scale feature vector set, the Core-set Sampling algorithm is applied for filtering. This algorithm first randomly selects an initial feature point to add to the feature library, and then iteratively executes the following process: calculating the minimum distance from all unselected feature points to the set of selected feature points, selecting the feature point with the largest minimum distance to add to the feature library, and repeating until the preset feature library size is reached (usually 10%-30% of the original number of features). This ensures that the selected reference fused feature vectors are evenly distributed and representative in the feature space. The completed normal sample feature library is organized using the FAISS high-dimensional vector index structure, supporting fast nearest neighbor retrieval operations.

[0056] During the detection phase, the defect detection equipment performs L2 normalization on the target fused feature vector generated in step S104, and then performs a K-nearest neighbor search operation (K typically ranges from 1 to 5) in the normal sample feature library to find the K reference fused feature vectors most similar to the target fused feature vector. The defect detection equipment calculates the feature distance between the target fused feature vector and the K nearest neighbor reference fused feature vectors using the Euclidean distance formula, and takes the average or minimum value of the K distance values ​​as the final feature distance. The defect detection equipment directly uses this feature distance as a feature deviation index: when the precision part to be inspected is in a normal, defect-free state, its target fused feature vector should fall within the feature distribution range represented by the normal sample feature library, with a small distance to the nearest neighbor reference fused feature vector and a low feature deviation value; when the precision part to be inspected has defects, its target fused feature vector will deviate from the normal feature distribution, with a large distance to all reference fused feature vectors and a significantly increased feature deviation value.

[0057] Optionally, in some embodiments, the defect detection device may employ a hierarchical clustering feature library index structure to accelerate retrieval. Specifically, the defect detection device first uses the K-Means clustering algorithm to divide the feature vectors in the normal sample feature library into multiple cluster centroids. When calculating feature distances, the defect detection device first calculates the distance between the target fused feature vector and each cluster centroid, quickly locating the nearest clusters, and then performs a fine-grained KNN search only within these clusters. Furthermore, the defect detection device sets up an "outlier buffer," where samples with feature deviations near the critical value are not directly involved in distance calculations but are used as a secondary reference set to prevent edge noise in the normal sample distribution from interfering with the detection results.

[0058] S106. When the feature deviation is greater than the preset anomaly judgment threshold, generate defect prompt information and determine the target defect area according to the mapping position of the target fused feature vector in the semantic space.

[0059] The preset anomaly threshold is a critical value for distinguishing between normal and defective samples based on feature deviation. When the feature deviation exceeds this threshold, a defect is identified. The threshold is typically set by: statistically analyzing the feature deviation distribution of normal samples on the validation set; determining the corresponding quantile as the threshold based on the desired detection recall (e.g., 95% or 99%); or selecting the threshold point that maximizes the ratio of true positive rate to false positive rate through ROC curve analysis. A typical value range is the mean of normal sample feature deviation plus 2 to 3 times the standard deviation. Defect alert information refers to the alert signal or report generated by the defect detection equipment when a defect is detected. This information includes the defect detection status, anomaly score, and possible defect type, used to notify operators or trigger subsequent processing. The target defect area refers to the actual spatial location range of the defect on the precision component to be inspected, specifically including the two-dimensional pixel coordinates or three-dimensional point cloud coordinates of the defect on or inside the component's surface.

[0060] After calculating the feature deviation in step S105, the defect detection device needs to make a final defect determination decision based on this quantitative index and further locate the specific spatial position of the defect to complete the entire defect detection process. Specifically, the defect detection device first compares the feature deviation value calculated in step S105 with a preset anomaly determination threshold. When the feature deviation is greater than the preset anomaly determination threshold, it indicates that the target fused feature vector of the precision component to be detected has significantly deviated from the normal feature distribution range represented by the normal sample feature library. The defect detection device determines that the precision component to be detected has a defect and generates a defect warning message. This message includes a defect detection flag, the specific value of the feature deviation as a quantitative score of the severity of the anomaly, and a detection timestamp, among other basic information. Further, the target defect region is determined based on the mapping position of the target fused feature vector in the semantic space corresponding to the feature deviation greater than the anomaly determination threshold.

[0061] When the feature deviation is less than or equal to the preset anomaly judgment threshold, the defect detection equipment determines that the precision component to be inspected is a qualified product and does not generate a defect warning message. Through this complete judgment and location process, the defect detection equipment can not only accurately identify whether there is a defect in the precision component to be inspected, but also accurately point out the specific location and degree of abnormality of the defect, providing detailed decision-making basis for subsequent quality analysis, defect classification or rework.

[0062] In this embodiment, the robotic arm drives multiple sensors to collaboratively acquire data and supplement missing areas to obtain complete 3D point cloud data. Simultaneously, the extracted 2D and 3D feature vectors are fused to generate a target fusion feature vector that combines texture and geometric attributes. Anomaly detection is then performed based on feature deviation calculated from a normal sample feature library. Therefore, it can achieve full-coverage measurement of complex structures such as deep holes and obstructions in precision parts to be inspected. Furthermore, it allows for semantic verification between appearance texture information and spatial geometric information, thereby filtering out interference from non-structural attachments (such as planar oil stains) using geometric depth information. Simultaneously, it captures minute, shallow damage using texture details. This effectively solves the problems of blind spots in scanning data of complex irregular workpieces and high false alarm rates due to the inability to accurately distinguish between surface contamination and real physical defects in related technologies. Thus, it achieves high-precision automatic detection and positioning of weak defects in precision parts without blind spots or interference, without requiring extensive defect sample training, improving the accuracy and reliability of detecting complex defects in precision parts.

[0063] Based on the above embodiments, the method provided in this embodiment will be described in further detail below. Please refer to... Figure 2 This is another flowchart illustrating a defect detection method for precision parts in an embodiment of this application.

[0064] S201. Control the robotic arm to drive the multi-source sensors to collect data in a coordinated manner on the precision parts to be inspected. Acquire two-dimensional image data and initial three-dimensional point cloud data along the preset first scanning path. The two-dimensional image data includes images representing the texture of the outer surface of the parts and images representing the texture of the inner cavity wall of the parts.

[0065] This step is similar to the description of step S101 in the above embodiment, and will not be repeated here.

[0066] S202. Perform missing detection on the initial 3D point cloud data. If there are missing regions, perform connectivity and density analysis on the initial 3D point cloud data to identify the missing regions of the point cloud and record the spatial coordinates of the missing regions of the point cloud in a unified coordinate system.

[0067] The initial three-dimensional point cloud data refers to the initial three-dimensional point cloud data acquired by the 3D laser contour sensor along the first scanning path. This point cloud mainly covers the outer surface area of ​​the precision parts to be inspected.

[0068] After initial data acquisition, when the 3D laser contour sensor produces incomplete point cloud data due to factors such as deep holes, occlusion, or reflection, the defect detection equipment needs to accurately identify the missing locations and record their coordinate information. Specifically, the defect detection equipment first performs connectivity analysis on the initial 3D point cloud data, using a region growing algorithm or a connected component labeling algorithm to divide the point cloud into multiple connected components. If isolated small-scale point clusters or large-area point cloud breakage regions exist, they are preliminarily identified as potential missing regions. Subsequently, the defect detection equipment performs density analysis on the initial 3D point cloud data, dividing the point cloud space into a 3D voxel grid or a 2D raster, counting the number of points in each grid, and calculating the local point cloud density value. The defect detection equipment compares the calculated actual density with the theoretical density, which is calculated based on the sensor's scanning parameters and the expected geometric characteristics of the workpiece surface. When the actual density of a mesh is lower than a preset threshold of the theoretical density (usually set to 30%-50%, this threshold is intended to tolerate a certain degree of noise and normal fluctuations, avoiding misjudging reflectance changes as missing regions; its setting is determined based on point cloud quality statistics on the validation set), the defect detection device marks the mesh as a missing point cloud region. For all marked missing point cloud regions, the defect detection device calculates the spatial extent of each missing region and extracts its boundary contour or center position.

[0069] S203. Generate a second scanning path based on spatial coordinates, and control the robotic arm to carry a point laser displacement sensor or industrial endoscope to perform fixed-point supplementation of the missing area of ​​the point cloud along the second scanning path to obtain local completion data.

[0070] The second scanning path refers to a supplementary scanning trajectory specifically planned for the missing point cloud regions identified and recorded in step S202. This path differs from the first scanning path, focusing on covering the missing regions rather than the entire surface. The point laser displacement sensor is a sensor employing the principle of point-based laser ranging, capable of high-precision distance measurement of a single point, suitable for point-by-point scanning of local areas such as the bottom of deep holes and narrow grooves. The local completion data refers to the three-dimensional coordinate dataset for the missing point cloud regions obtained through fixed-point supplementary measurements. This data is used to fill in the blank areas of the initial three-dimensional point cloud data.

[0071] After identifying the missing point cloud region and recording its spatial coordinates in step S202, the defect detection device needs to perform targeted supplementary measurements to obtain the geometric data of the missing region. Specifically, the defect detection device generates a second scanning path based on the spatial coordinates recorded in step S202, combined with the geometric features of the missing point cloud region (such as aperture size, depth, occlusion angle, etc.). The path planning algorithm comprehensively considers the following factors: the sensor's field of view and ranging capability to ensure effective illumination of the missing region; the kinematic constraints of the robotic arm to avoid joint over-limits and collisions; and the incident angle of the sensor relative to the surface of the missing region. For deep hole regions, a vertical or small-angle incident posture needs to be planned to obtain effective reflected signals. After generating the second scanning path, the defect detection device selects a suitable supplementary sensor according to the type of missing region: for deep hole regions with large depths, the robotic arm is controlled to carry a point laser displacement sensor, whose fine beam can penetrate into narrow spaces for point-by-point measurement; for complex internal structures, an industrial endoscope can be selected in conjunction with structured light or binocular vision for local 3D reconstruction. The defect detection equipment controls a robotic arm to move along a second scanning path. At each measurement point, sensors collect distance or depth information. Combined with the robotic arm's real-time pose, the measurement data is converted into three-dimensional coordinates in a unified coordinate system. During supplementary measurements, the defect detection equipment may need to adjust the sensor orientation or measure the same area from multiple angles to ensure data integrity and accuracy. Finally, the defect detection equipment aggregates the three-dimensional coordinates of all supplementary measurement points to form a locally complete dataset.

[0072] Optionally, in some embodiments, the defect detection device can introduce a path safety constraint mechanism based on bounding box collision detection when generating the second scanning path. Specifically, a real-time bounding volume hierarchy of the precision component to be inspected is constructed based on the initial 3D point cloud data, and the geometric models of the robotic arm links and sensors are used as dynamic obstacles. When planning the supplementary measurement path for deep holes or grooves, not only is the optimal observation angle of the sensor calculated, but it is also necessary to verify whether the path trajectory points interfere with the bounding volume hierarchy. If there is a risk of interference, the defect detection device automatically adjusts the redundant degrees of freedom (null-space motion) of the robotic arm, changing the elbow posture while keeping the sensor end-effector posture unchanged, to avoid the outer wall of the workpiece.

[0073] S204. Perform point cloud registration and stitching on the initial 3D point cloud data and the local completion data to generate complete 3D point cloud data;

[0074] After acquiring the local completion data in step S203, the defect detection device needs to integrate this data with the initial 3D point cloud data to form a spatially continuous and complete 3D point cloud data. Specifically, the defect detection device first performs point cloud registration between the initial 3D point cloud data and the local completion data. Since both sets of data are collected by sensors carried by the robotic arm in a unified coordinate system, they should theoretically be naturally aligned. However, in practice, there may be coordinate offsets caused by repeated positioning errors of the robotic arm, sensor calibration deviations, or slight movements of the workpiece. The defect detection device extracts point cloud fragments of the missing region boundaries in the initial 3D point cloud data as references and performs feature matching with the corresponding boundary regions in the local completion data to calculate the coordinate transformation relationship between the two. If the deviation is within the allowable range (usually less than 0.1mm), the theoretical transformation provided by the robotic arm kinematics is directly adopted; if the deviation is large, the ICP algorithm is used for fine registration, and the optimal transformation matrix is ​​iteratively calculated to minimize the error in the overlapping area of ​​the two sets of point clouds. After registration, the defect detection equipment performs point cloud stitching: it applies the transformation matrix obtained from registration to the locally completed data, transforming it to the same coordinate system as the initial 3D point cloud data; it merges the two sets of point cloud data, filtering out or fusing redundant points in overlapping areas, and using voxelization downsampling to maintain uniform density; it checks whether the stitched point cloud has effectively filled the original missing areas, and if small holes still exist, they are marked for further processing. Finally, the defect detection equipment generates complete 3D point cloud data with sufficient coverage and density in all key detection areas.

[0075] S205. Extract a two-dimensional feature vector containing local texture information from the two-dimensional image data. The image representing the texture of the outer surface of the component in the two-dimensional image data is a global RGB image, and the image representing the texture of the inner wall of the component is a local image of the endoscope.

[0076] This step specifically includes:

[0077] Feature extraction branches are constructed for the global RGB image and the local endoscopic image, respectively, to extract the first feature map and the second feature map, and then the two are aligned to the same channel dimension.

[0078] The channel importance weights of the first feature map and the second feature map are calculated using the channel attention mechanism. The channel data are dynamically exchanged in a preset ratio according to the difference between the weights to obtain the first intermediate feature and the second intermediate feature after channel interaction.

[0079] A spatial mask is generated based on the feature response value of the second feature map. The spatial mask is used to locate the effective region of the local image of the endoscope in the global coordinate system. The second intermediate feature is then fused into the effective region of the first intermediate feature. After dimensionality reduction, a two-dimensional feature vector is obtained.

[0080] The global RGB image refers to a color image captured by an RGB industrial camera, covering the complete outer surface of a precision component under inspection, containing the overall structure and large-scale texture information of the component. The endoscopic local image refers to an image of a local area such as a deep hole or cavity acquired through an industrial endoscope; it has a small field of view but can capture wall details in hidden areas. The first feature map refers to the multi-channel feature representation output after the feature extraction branch of the global RGB image; its shape is typically C1×H×W, where C1 is the number of channels, and H and W are the spatial height and width. The second feature map refers to the multi-channel feature representation output after the feature extraction branch of the endoscopic local image; its shape is C2×H'×W'. The preset ratio refers to the proportion of dynamically exchanged channels to the total number of channels. This parameter controls the intensity of information interaction between modalities, avoiding excessive exchange that could lead to semantic confusion. Its setting is typically 5%-20%, with the specific value determined through validation set evaluation. The first intermediate feature refers to the feature representation of the first feature map after channel interaction, where some channel data is replaced or fused with the corresponding channel of the second feature map. The second intermediate feature refers to the feature representation of the second feature map after channel interaction. The feature response value refers to the feature activation intensity at each spatial location in the second feature map, obtained by performing max pooling or summation operations on the channel dimension to obtain a two-dimensional response map. The spatial mask is a binary or continuous-value mask matrix generated based on the feature response values ​​through thresholding or normalization, used to identify information-rich effective regions in endoscopic images.

[0081] After obtaining the complete 3D point cloud data in step S204, the defect detection device needs to extract a 2D feature vector that fuses global appearance and local cavity information from the 2D image data. Specifically, the defect detection device constructs feature extraction branches for the global RGB image and the local endoscopic image, respectively. The global branch uses a DINOv2-ViT-S / 14 network to extract large-scale semantic features, while the endoscopic branch uses a lightweight ViT network to extract local detail features, outputting a first feature map and a second feature map, respectively. The defect detection device aligns the two to the same channel dimension C using a 1×1 convolution. Subsequently, the defect detection device applies a channel attention mechanism to the two feature maps, calculates their respective channel importance weights, and compares the difference between the two sets of weights. It then selects the channel with the largest difference and performs a bidirectional swap, replacing the low-importance channel in the first feature map with the corresponding channel in the second feature map, and vice versa, to obtain the first and second intermediate features after channel interaction. Next, the defect detection device performs channel dimension max pooling on the second feature map to obtain feature response values, and generates a spatial mask by thresholding to identify the effective region of the endoscopic image. The defect detection device maps the spatial mask to the global coordinate system based on the sensor calibration relationship, locates the corresponding position of the endoscopic field of view in the global image, and fuses the second intermediate feature to the corresponding position of the first intermediate feature according to the effective region. Finally, the defect detection device performs dimensionality reduction processing on the fused features, compressing the number of channels and flattening the spatial dimension through 1×1 convolution, to obtain a two-dimensional feature vector containing global external surface texture information and local internal cavity detail information.

[0082] S206. Extract a three-dimensional feature vector containing spatial geometric information from the three-dimensional point cloud data;

[0083] This step is similar to step S103 in the above embodiment, and will not be repeated here.

[0084] S207. Project the two-dimensional feature vector and the three-dimensional feature vector onto the semantic embedding space of the same dimension respectively to obtain the two-dimensional embedding vector and the three-dimensional embedding vector.

[0085] In this context, a two-dimensional embedding vector refers to the representation of a two-dimensional feature vector in the semantic embedding space after projection, with dimensions typically set to 256 or 512. A three-dimensional embedding vector refers to the representation of a three-dimensional feature vector in the semantic embedding space after projection, and has the same dimensions as the two-dimensional embedding vector.

[0086] After extracting the two-dimensional and three-dimensional feature vectors in steps S205 and S206 respectively, since they come from different modalities and their feature dimensions may differ, the defect detection device needs to map them to a unified semantic space to achieve subsequent cross-modal interaction. Specifically, the defect detection device applies independent projection layers to the two-dimensional and three-dimensional feature vectors respectively. The two-dimensional feature vector is converted into a two-dimensional embedding vector by a two-dimensional projection encoder (usually a 1-2 layer MLP), and the three-dimensional feature vector is converted into a three-dimensional embedding vector by a three-dimensional projection encoder. The output dimensions of the two encoders are set to the same value to ensure dimension alignment, thereby obtaining feature representations that can be compared and interacted with in a unified semantic embedding space.

[0087] S208. Using the two-dimensional embedding vector as the query vector and the three-dimensional embedding vector as the key vector, calculate the attention response of the two-dimensional modality to the three-dimensional geometric information to generate the first fusion component.

[0088] In this context, the query vector refers to the feature vector used in the attention mechanism to retrieve relevant information, determining which content is focused on. The key-value vector refers to the feature vector being queried in the attention mechanism, providing information for matching and extraction. The attention response is the fused feature representation obtained by weighted aggregation of the key-value vectors by calculating the similarity between the query vector and the key-value vector. The first fusion component refers to the fused feature component generated after associating the two-dimensional embedding vector with three-dimensional geometric information, enabling the two-dimensional features to perceive the corresponding spatial structure information.

[0089] After semantic space alignment is completed in step S207, the defect detection device needs to perform information querying and fusion from two-dimensional texture features to three-dimensional geometric features. Specifically, the defect detection device uses the two-dimensional embedding vector as the query vector and the three-dimensional embedding vector as the key vector. It calculates the similarity between the two-dimensional and three-dimensional embedding vectors through a multi-head attention mechanism to obtain an attention weight matrix. Then, it uses this weight matrix to perform a weighted summation on the three-dimensional embedding vectors to achieve the attention response of the two-dimensional modality to the three-dimensional geometric information. This enables the two-dimensional texture features to be associated with the geometric structure information of the corresponding position, generating a first fusion component that combines texture and geometric semantics.

[0090] S209. Using the three-dimensional embedding vector as the query vector and the two-dimensional embedding vector as the key vector, calculate the attention response of the three-dimensional modality to the two-dimensional texture information to generate the second fusion component.

[0091] The second fusion component refers to the fusion feature component generated after the three-dimensional embedding vector is associated with the two-dimensional texture information, which enables the three-dimensional geometric features to perceive the corresponding texture appearance information.

[0092] After completing the first interaction direction in step S208, the defect detection device needs to perform a reverse information query to achieve information fusion from three-dimensional geometric features to two-dimensional texture features. Specifically, the defect detection device uses the three-dimensional embedding vector as the query vector and the two-dimensional embedding vector as the key vector. It calculates the similarity between the three-dimensional and two-dimensional embedding vectors through a multi-head attention mechanism to obtain an attention weight matrix. This weight matrix is ​​then used to perform a weighted summation of the two-dimensional embedding vectors to achieve the attention response of the three-dimensional modality to the two-dimensional texture information. This enables the three-dimensional geometric features to be associated with the texture appearance information at the corresponding location, generating a second fusion component that combines geometric and texture semantics. Together with the first fusion component, this second fusion component constitutes a bidirectional fused feature representation.

[0093] S210. Weighted concatenation of the first fusion component and the second fusion component is performed to generate the target fusion feature vector, specifically including:

[0094] Element-wise multiplication of the first fusion component and the second fusion component yields a mutual activation vector representing the semantic consistency of texture and geometric features.

[0095] The mutual activation vectors are mapped to a consistent weight matrix with values ​​within a preset range using a preset activation function.

[0096] The first fusion component and the second fusion component are respectively weighted element by element with the consistency weight matrix to obtain the weighted first fusion component and the weighted second fusion component.

[0097] The weighted first fusion component and the weighted second fusion component are concatenated to generate the target fusion feature vector.

[0098] The mutual activation vector is a vector obtained through element-wise multiplication, representing the semantic consistency of texture and geometric features. A larger value indicates greater consistency in the semantic responses of the two features at that location. The preset activation function is the function used for non-linear mapping, typically the Sigmoid function. Its purpose is to compress the arbitrary numerical range of the mutual activation vector to the [0, 1] interval, making it usable as normalized weight coefficients. The standard Sigmoid function σ(x) = 1 / (1 + e^(-x)) is used directly. The preset interval refers to the numerical range of the consistency weight matrix, usually set to [0, 1]. This range ensures that the weight values ​​have probabilistic meaning and are numerically stable. The consistency weight matrix is ​​the matrix obtained after mapping the mutual activation vectors using the activation function, used to adaptively adjust the contribution of the two fusion components.

[0099] After generating the first and second fusion components in steps S208 and S209 respectively, the defect detection device needs to fuse them into a unified target fusion feature vector through an adaptive weighting mechanism. Specifically, the defect detection device performs element-wise multiplication on the first and second fusion components to obtain a mutual activation vector, which has a larger value at positions where the texture and geometric feature responses are consistent. The defect detection device inputs the mutual activation vector into a preset activation function (Sigmoid function) for mapping to obtain a consistency weight matrix with values ​​in the preset interval [0, 1]. The defect detection device performs element-wise weighting on the first fusion component and the consistency weight matrix to obtain a weighted first fusion component, and performs element-wise weighting on the second fusion component and the consistency weight matrix to obtain a weighted second fusion component, thus enhancing the features at positions with larger weights. Finally, the defect detection device concatenates the weighted first and weighted second fusion components along the channel dimension to generate a target fusion feature vector that simultaneously contains texture and geometric attributes.

[0100] S211. Calculate the feature distance between the target fusion feature vector and the reference fusion feature vector in the normal sample feature library, and use the feature distance as the feature deviation. The normal sample feature library stores representative reference fusion feature vectors extracted from defect-free samples and screened.

[0101] This step is similar to step S105 in the above embodiments, and will not be repeated here.

[0102] S212. When the feature distance is less than the preset drift update threshold, a weighted update operation is performed on the reference fusion feature vector based on the currently generated target fusion feature vector; the value of the drift update threshold is less than the anomaly detection threshold.

[0103] The weighted update operation specifically includes: using a preset time decay coefficient, performing a weighted average calculation on the target fused feature vector and the reference fused feature vector to obtain the updated reference fused feature vector, and using the updated reference fused feature vector to replace the reference fused feature vector in the normal sample feature library.

[0104] The preset drift update threshold is a critical feature distance value used to determine whether to adaptively update the normal sample feature library. This threshold identifies samples close to a normal distribution to update the feature library and adapt to the slow drift during the production process. Its value is less than the anomaly detection threshold, and is typically set at 50%-70% of the anomaly detection threshold to ensure only normal samples participate in the update. The weighted update operation refers to the dynamic adjustment of the reference fusion feature vector in the normal sample feature library based on the current target fusion feature vector, enabling the feature library to adapt to slow changes in process parameters. The preset time decay coefficient is a coefficient that controls the relative weights of historical features and current features in the weighted average calculation. This coefficient balances the stability and adaptability of the feature library, and is typically set between 0.9 and 0.99; a larger value indicates a higher weight for historical features.

[0105] After calculating the feature distance in step S211, if the feature distance is less than the preset drift update threshold, it indicates that the current precision component to be inspected is a normal sample and the feature representation is slightly offset. The defect detection equipment needs to update the normal sample feature library to adapt to the slow changes in production conditions. Specifically, the defect detection equipment uses a preset time decay coefficient α (e.g., 0.95) and performs a weighted average calculation according to the formula: Updated reference fusion feature vector = α × Reference fusion feature vector + (1-α) × Target fusion feature vector, so that the original reference fusion feature vector is slightly adjusted towards the current target fusion feature vector. The defect detection equipment replaces the corresponding reference fusion feature vector in the normal sample feature library with the updated reference fusion feature vector, completing the online update of the feature library.

[0106] Optionally, in some embodiments, the defect detection device can introduce a model drift anti-contamination mechanism before performing the weighted update operation. Specifically, the system sets up a pending update queue, temporarily storing samples that meet the drift update threshold in this queue instead of directly updating the feature library. When the queue length reaches a preset batch size (e.g., 50 samples), the vector angle between the mean of the features of this batch of samples and the mean of the original feature library is calculated. Weighted updates are only performed in batches when the vector angle is less than a safety threshold (e.g., 5 degrees); if the angle is too large, an alarm is issued prompting manual intervention for review.

[0107] S213. When the feature deviation is greater than the preset anomaly judgment threshold, generate defect prompt information and determine the target defect area according to the mapping position of the target fused feature vector in the semantic space.

[0108] This step specifically includes:

[0109] The feature deviation is reconstructed into an initial anomaly scoring matrix based on the mapping location;

[0110] Taking any feature point in the initial anomaly score map as the center, the spatial neighborhood range is determined, and the Gaussian weighted average of the feature deviation within the spatial neighborhood range is calculated to obtain the smooth anomaly score. The smooth anomaly score is then processed by row normalization.

[0111] When the smooth anomaly score is greater than the preset defect segmentation threshold, the corresponding position is marked as a defect identifier value, and a binary defect mask is generated.

[0112] Extract the connected regions composed of defect identifier values ​​from the defect mask, and determine the range covered by the connected regions as the target defect region.

[0113] The initial anomaly scoring matrix refers to a two-dimensional or three-dimensional matrix structure reconstructed from the feature deviations according to their mapping positions in semantic space. Each element in the matrix corresponds to the degree of anomaly at a spatial location. A feature point refers to the position of a single element in the initial anomaly scoring matrix. The spatial neighborhood refers to the local region centered on a particular feature point. The smoothed anomaly score is the smoothed anomaly score obtained after Gaussian weighted averaging, eliminating the influence of isolated noise points. The preset defect segmentation threshold is the threshold used to binarize the smoothed anomaly score. This threshold separates defective regions from normal regions; it is typically set to the 75%-90% quantile of the smoothed anomaly score distribution, or automatically determined using the Otsu algorithm. The defect identifier value is the pixel value marked as a defect after binarization, usually set to 1. The defect mask is a binary matrix after binarization, where the defect identifier value marks the defective region.

[0114] After determining in step S211 that the feature deviation is greater than the preset anomaly threshold, the defect detection device needs to accurately locate the spatial position of the defect. Specifically, the defect detection device reconstructs the feature deviation into an initial anomaly scoring matrix based on the mapping position of the target fused feature vector in the semantic space, establishing a correspondence between the degree of anomaly and the spatial position. The defect detection device performs Gaussian filtering to smooth the initial anomaly scoring matrix, determines the spatial neighborhood range with each feature point as the center, calculates the smoothed anomaly score by Gaussian weighted average of the feature deviation within the neighborhood, and then maps it to the [0, 1] interval after row normalization. The defect detection device compares the smoothed anomaly score with the preset defect segmentation threshold. When the smoothed anomaly score is greater than the threshold, the corresponding position is marked as a defect identifier value of 1; otherwise, it is marked as 0, generating a binary defect mask. The defect detection device extracts the connected regions composed of defect identifier values ​​in the defect mask, maps the range covered by the connected regions back to the original two-dimensional image or three-dimensional point cloud coordinate system, determines it as the target defect region, and generates defect prompt information containing information such as defect position, area, and anomaly score.

[0115] In this embodiment, by performing connectivity and density analysis on the initial 3D point cloud data and controlling the robotic arm to perform point-to-point supplementation of missing point cloud regions along the second scanning path to generate complete 3D point cloud data, a channel attention mechanism and spatial mask are used to fuse global and endoscopic image features. A target fused feature vector is generated through bidirectional attention response and consistency weight matrix, and a weighted update operation is performed on the reference fused feature vector in the normal sample feature library based on the drift update threshold. Therefore, high-precision data completion for blind spots such as deep holes is achieved, constructing a two-dimensional representation that complements global structure and local micro-texture, and verifying the texture... By dynamically suppressing unstructured noise (such as oil stains) interference through the consistency of geometric response in the semantic dimension, and enabling the detection model to adaptively follow the slow drift of the production environment (such as light attenuation and mold wear), it effectively solves the problems of missing scanning data for complex irregular parts, mismatch between physical meanings of heterogeneous modal feature fusion leading to false defect false alarms, and fixed models being unable to adapt to dynamic changes in the production environment, resulting in a decline in long-term system stability. Thus, it achieves highly reliable automatic detection and positioning of subtle defects in precision parts with full coverage, high anti-interference ability, and self-evolution capability in dynamic and ever-changing industrial scenarios.

[0116] The defect detection system in the embodiments of this invention is described below from the perspective of hardware processing. Please refer to [link / reference]. Figure 3 This is a schematic diagram of the physical device structure of a defect detection system in an embodiment of this application.

[0117] It should be noted that, Figure 3 The structure of the defect detection system shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0118] like Figure 3 As shown, the defect detection system includes a CPU 301, which can perform various appropriate actions and processes based on a program stored in the read-only memory ROM 302 or a program loaded from the storage section 308 into the random access memory RAM 303, such as performing the methods described in the above embodiments. The RAM 303 also stores various programs and data required for system operation. The CPU 301, ROM 302, and RAM 303 are interconnected via a bus 304. An I / O interface 305 is also connected to the bus 304.

[0119] The following components are connected to I / O interface 305: input section 306 including audio input devices, push-button switches, etc.; output section 307 including a liquid crystal display (LCD) and audio output devices, indicator lights, etc.; storage section 308 including a hard disk, etc.; and communication section 309 including a network interface card such as a LAN (Local Area Network) card, modem, etc. Communication section 309 performs communication processing via a network such as the Internet. Drive 310 is also connected to I / O interface 305 as needed. Removable media 311, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 310 as needed so that computer programs read from them can be installed into storage section 308 as needed.

[0120] In particular, according to embodiments of the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present invention include a computer program product comprising a computer program / instructions carried on a computer-readable medium, the computer program / instructions containing computer program / instructions for performing the methods shown in the flowcharts. In such embodiments, the computer program / instructions can be downloaded and installed from a network via communication section 309, and / or installed from removable medium 311. When the computer program is executed by CPU 301, it performs the various functions defined in the present invention.

[0121] It should be noted that specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0122] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. Each block in a flowchart or block diagram may represent a module, program segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those shown in the drawings.

[0123] Specifically, the defect detection system of this embodiment includes a processor and a memory. The memory stores a computer program. When the computer program is executed by the processor, it implements the defect detection method for precision parts provided in the above embodiment.

[0124] In another aspect, the present invention also provides a computer-readable storage medium, which may be included in the defect detection system described in the above embodiments; or it may exist independently and not assembled into the defect detection system. The storage medium carries one or more computer programs, which, when executed by a processor of the defect detection system, cause the defect detection system to implement a defect detection method for precision parts provided in the above embodiments.

[0125] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit it. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

[0126] As used in the above embodiments, depending on the context, the term "when..." can be interpreted as meaning "if...", "after...", "in response to determining...", or "in response to detecting...". Similarly, depending on the context, the phrase "when determining..." or "if (the stated condition or event) is interpreted as meaning "if determining...", "in response to determining...", "when (the stated condition or event) is detected", or "in response to detecting (the stated condition or event)".

[0127] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This program can be stored in a computer-readable storage medium, and when executed, it can include the processes described in the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM or random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A defect detection method for precision parts, applied to a defect detection system, characterized in that, The defect detection system includes at least a multi-source sensor, an industrial endoscope, and a robotic arm, and the method includes: The robotic arm is controlled to drive multiple source sensors to collaboratively collect data on the precision parts to be inspected. Two-dimensional image data and initial three-dimensional point cloud data are acquired along a preset first scanning path. The two-dimensional image data includes images representing the texture of the outer surface of the parts and images representing the texture of the inner cavity wall of the parts. The initial three-dimensional point cloud data is subjected to missing detection. If there is a missing area, the robotic arm is controlled to perform supplementary measurement on the precision parts to be tested in order to obtain complete three-dimensional point cloud data. The three-dimensional point cloud data includes point cloud that characterizes the overall shape of the parts and local point cloud obtained by supplementary measurement of deep holes or occluded areas. A two-dimensional feature vector containing local texture information is extracted from the two-dimensional image data, and a three-dimensional feature vector containing spatial geometric information is extracted from the three-dimensional point cloud data. The two-dimensional feature vector and the three-dimensional feature vector are fused to generate a target fused feature vector, which has both texture and geometric attributes; Calculate the feature distance between the target fused feature vector and the reference fused feature vector in the normal sample feature library, and use the feature distance as the feature deviation. The normal sample feature library stores representative reference fused feature vectors extracted from defect-free samples and screened. When the feature deviation is greater than the preset anomaly detection threshold, a defect prompt message is generated, and the target defect region is determined according to the mapping position of the target fused feature vector in the semantic space.

2. The method according to claim 1, characterized in that, The process of performing missing detection on the initial 3D point cloud data, and controlling the robotic arm to perform supplementary detection on the precision parts to be inspected if missing areas exist, to obtain complete 3D point cloud data, specifically includes: Connectivity and density analysis are performed on the initial three-dimensional point cloud data to identify missing point cloud regions, and the spatial coordinates of the missing point cloud regions in a unified coordinate system are recorded. A second scanning path is generated based on the spatial coordinates. The robotic arm carrying a point laser displacement sensor or the industrial endoscope is controlled to perform fixed-point supplementation measurement of the missing area of ​​the point cloud along the second scanning path to obtain local completion data. The initial 3D point cloud data and the locally completed data are registered and stitched together to generate complete 3D point cloud data.

3. The method according to claim 2, characterized in that, The image representing the texture of the outer surface of the component in the two-dimensional image data is a global RGB image, and the image representing the texture of the inner wall of the component is a local image of the endoscope. The step of extracting a two-dimensional feature vector containing local texture information from the two-dimensional image data specifically includes: Feature extraction branches are constructed for the global RGB image and the local endoscopic image, respectively, to extract a first feature map and a second feature map, and then the two are aligned to the same channel dimension. The channel importance weights of the first feature map and the second feature map are calculated using a channel attention mechanism. Based on the difference between the weights, a preset ratio of channel data is dynamically exchanged to obtain the first intermediate feature and the second intermediate feature after channel interaction. A spatial mask is generated based on the feature response value of the second feature map. The spatial mask is used to locate the effective region of the local image of the endoscope in global coordinates. The second intermediate feature is fused into the effective region of the first intermediate feature. After dimensionality reduction, a two-dimensional feature vector is obtained.

4. The method according to claim 3, characterized in that, The step of fusing the two-dimensional feature vector and the three-dimensional feature vector to generate the target fused feature vector specifically includes: The two-dimensional feature vector and the three-dimensional feature vector are projected onto a semantic embedding space of the same dimension to obtain a two-dimensional embedding vector and a three-dimensional embedding vector, respectively. Using the two-dimensional embedding vector as the query vector and the three-dimensional embedding vector as the key vector, the attention response of the two-dimensional modality to the three-dimensional geometric information is calculated to generate the first fusion component; Using the three-dimensional embedding vector as the query vector and the two-dimensional embedding vector as the key vector, the attention response of the three-dimensional modality to the two-dimensional texture information is calculated to generate the second fusion component. The first fusion component and the second fusion component are weighted and concatenated to generate the target fusion feature vector.

5. The method according to claim 4, characterized in that, The step of weighted concatenation of the first fusion component and the second fusion component to generate the target fusion feature vector specifically includes: Element-wise multiplication is performed on the first fusion component and the second fusion component to obtain a mutual activation vector representing the consistency of semantic response between texture and geometric features. The mutual activation vectors are mapped to a consistent weight matrix with values ​​within a preset range using a preset activation function. The first fusion component and the second fusion component are respectively weighted element-wise with the consistency weight matrix to obtain the weighted first fusion component and the weighted second fusion component; The weighted first fusion component and the weighted second fusion component are concatenated to generate the target fusion feature vector.

6. The method according to claim 1, characterized in that, Determining the target defect region based on the mapping position of the target fused feature vector in the semantic space specifically includes: The feature deviation is reconstructed into an initial anomaly scoring matrix based on the mapping position; Taking any feature point in the initial anomaly score map as the center, a spatial neighborhood range is determined, and the feature deviation within the spatial neighborhood range is calculated by Gaussian weighted average to obtain a smooth anomaly score, which is then processed by row normalization. When the smooth anomaly score is greater than the preset defect segmentation threshold, the corresponding position is marked as a defect identifier value, and a binary defect mask is generated. Extract the connected regions composed of the defect identifier values ​​from the defect mask, and determine the range covered by the connected regions as the target defect region.

7. The method according to claim 1, characterized in that, After calculating the feature distance between the target fused feature vector and the reference fused feature vector in the normal sample feature library, the method further includes: When the feature distance is less than a preset drift update threshold, a weighted update operation is performed on the reference fusion feature vector based on the currently generated target fusion feature vector; the value of the drift update threshold is less than the anomaly determination threshold. The weighted update operation specifically includes: using a preset time decay coefficient, performing a weighted average calculation on the target fused feature vector and the reference fused feature vector to obtain an updated reference fused feature vector, and using the updated reference fused feature vector to replace the reference fused feature vector in the normal sample feature library.

8. A defect detection system, characterized in that, The defect detection system includes: one or more processors and a memory; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code including computer instructions, and the one or more processors call the computer instructions to cause the defect detection system to perform the method as described in any one of claims 1-7.

9. A computer-readable storage medium storing computer instructions, characterized in that, When the computer instructions are executed on the defect detection system, the defect detection system performs the method as described in any one of claims 1-7.

10. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are run on the defect detection system, the defect detection system performs the method as described in any one of claims 1-7.

Images