Machine vision dynamic defect detection method and device for precise structural part

Abstract

The invention relates to a method and a device for detecting machine vision dynamic defects of a precise structural part. The method comprises the following steps: scanning a sample precise structural member to obtain an original image sequence; inputting the original image sequence into a GPU for parallel processing to obtain a preprocessed image sequence and a target feature point set; performing image matching to obtain a camera pose transformation matrix; dynamically correcting the preprocessed image sequence to obtain high-precision three-dimensional point cloud data; inputting the high-precision three-dimensional point cloud data into an improved U-Net deep learning network containing a geometric perception module for training and transfer learning to obtain a curved surface adaptive defect detection model; and based on the curved surface adaptive defect detection model, carrying out full-matrix sampling analysis and geometric model compensation on target three-dimensional point cloud data scanned by the precision structural member to be detected in real time to obtain a target defect detection result. According to the method, the accuracy and robustness of defect identification of the precision structural part are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of machine vision, and in particular to a method and device for detecting dynamic defects of precision structural parts using machine vision. Background Art

[0002] As the manufacturing industry's requirements for surface defect detection of precision structural parts continue to increase, traditional manual detection methods can no longer meet the needs of modern production lines, so machine vision dynamic defect detection technology has emerged. Existing machine vision detection methods still face many challenges when facing complex surfaces, high reflectivity materials, and dynamic detection environments.

[0003] The surfaces of precision structural parts have complex geometric shapes and diverse material properties, which makes it difficult for traditional 2D image processing methods to accurately identify and locate defects. Secondly, when performing real-time dynamic inspection on the production line, camera motion and structural part movement will cause image blur and distortion, affecting inspection accuracy. Precision structural parts made of different materials and processing techniques exhibit different surface features and defect types, which requires the inspection system to have strong adaptability and generalization capabilities. Existing inspection algorithms are often optimized for specific types of defects and are difficult to adapt to diverse inspection needs. Summary of the invention

[0004] The main purpose of the present invention is to provide a method and device for dynamic defect detection of precision structural parts by machine vision, so as to improve the accuracy and robustness of defect recognition of precision structural parts.

[0005] To achieve the above object, the present invention provides a method for dynamic defect detection of precision structural parts by machine vision, comprising the following steps: Perform multi-angle synchronous scanning on the sample's precision structural parts to obtain the original image sequence; Inputting the original image sequence into the GPU for parallel processing and extracting feature points to obtain a preprocessed image sequence and a target feature point set; Image matching is performed based on the target feature point set, and erroneous matching points are eliminated using the RANSAC algorithm, and the camera pose transformation matrix is ​​obtained through the consistency check of the interior point neighborhood space; Dynamically correcting the preprocessed image sequence according to the camera posture transformation matrix, using a structured light stereo vision algorithm combined with projection coded grating pattern analysis to obtain high-precision three-dimensional point cloud data; Inputting the high-precision three-dimensional point cloud data into an improved U-Net deep learning network training and transfer learning including a geometric perception module to obtain a surface adaptive defect detection model; Based on the surface adaptive defect detection model, full matrix sampling analysis and geometric model compensation are performed on the target three-dimensional point cloud data scanned in real time for the precision structural parts to be tested, so as to obtain the target defect detection results.

[0006] The present invention also provides a precision structural part machine vision dynamic defect detection device, comprising: A scanning module is used to perform multi-angle synchronous scanning on the sample precision structural parts to obtain the original image sequence; A processing module, used for inputting the original image sequence into the GPU for parallel processing and extracting feature points to obtain a preprocessed image sequence and a target feature point set; A matching module is used to perform image matching based on the target feature point set, and to eliminate erroneous matching points using the RANSAC algorithm, and to obtain a camera pose transformation matrix through an inlier neighborhood space consistency check; A correction module is used to dynamically correct the pre-processed image sequence according to the camera posture transformation matrix, using a structured light stereo vision algorithm combined with projection coded grating pattern analysis to obtain high-precision three-dimensional point cloud data; A training module, used for inputting the high-precision three-dimensional point cloud data into an improved U-Net deep learning network training and transfer learning containing a geometric perception module to obtain a surface adaptive defect detection model; The detection module is used to perform full matrix sampling analysis and geometric model compensation on the target three-dimensional point cloud data scanned in real time on the precision structural parts to be tested based on the surface adaptive defect detection model to obtain the target defect detection result.

[0007] The present invention also provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of any one of the above methods when executing the computer program.

[0008] The present invention also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned methods are implemented.

[0009] In summary, the multi-angle synchronous scanning technology of the technical solution provided by the present invention combines the vertical reciprocating motion of the high-speed camera and the horizontal rotation of the precision rotating platform to realize the all-round and efficient data collection of the surface of the precision structural parts, overcome the limitations of the traditional single-angle scanning method, and improve the detection coverage of complex surface defects. The GPU parallel processing and improved ORB algorithm are used to extract feature points, which significantly improves the speed of image preprocessing and feature extraction, and lays the foundation for real-time dynamic defect detection. The introduction of layered spatial consistency constraint strategy and RANSAC algorithm for image matching and error matching point elimination improves the accuracy of camera pose estimation, and provides a reliable foundation for subsequent dynamic correction and three-dimensional reconstruction. Combined with structured light stereo vision algorithm and projection coded grating pattern analysis, the acquisition of high-precision three-dimensional point cloud data is realized, effectively solving the problem of 3D reconstruction of complex surfaces and high reflectivity material surfaces. An improved U-Net deep learning network with a geometric perception module is designed, combined with transfer learning technology, to improve the adaptability of the model to different materials and processes, and realize surface adaptive defect detection. The use of full-matrix sampling analysis and geometric model compensation technology effectively reduces the interference of surface shape on defect detection and improves the accuracy and robustness of defect identification. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 Schematic diagram of the steps of a method for dynamic defect detection of precision structural parts by machine vision in one embodiment of the present invention; Figure 2 It is a structural block diagram of a device for dynamic defect detection of precision structural parts by machine vision in one embodiment of the present invention; Figure 3 It is a schematic block diagram of the structure of a computer device according to an embodiment of the present invention.

[0011] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0012] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0013] Reference Figure 1 This embodiment provides a method for dynamic defect detection of precision structural parts by machine vision, comprising the following steps: S1, perform multi-angle synchronous scanning on the sample precision structure to obtain the original image sequence; Among them, the sample precision structural parts are fixed on the precision rotating platform, and a preset angular velocity ω is applied to the precision rotating platform, so that the sample precision structural parts keep rotating at a constant speed in the horizontal plane, obtain the complete appearance information of the structural parts at various angles, and generate accurate rotational motion parameters. At the same time, in order to realize the synchronous dynamic scanning of the precision structural parts, the high-speed camera is installed on the vertical guide rail, and the preset reciprocating speed v and the preset reciprocating amplitude A are applied to the high-speed camera, so that the high-speed camera reciprocates in the vertical direction, and the vertical motion parameters of the camera in the vertical direction are obtained. Based on the rotational motion parameters and the vertical motion parameters, the three-dimensional space trajectory equation S(t) of the high-speed camera is calculated, which is used to describe the three-dimensional space position change of the high-speed camera under the combined action of the rotating platform motion and the vertical guide rail reciprocating motion. The camera's motion trajectory data is obtained by calculating the camera's motion trajectory equation. According to the camera's motion trajectory data, the trigger frequency f of the high-speed camera is set so that the camera's motion trajectory data meets the requirements of the Nyquist sampling theorem, ensuring that the sampling frequency is high enough when the image is collected during the camera's movement to avoid information loss caused by insufficient sampling, and obtain accurate camera trigger timing. Based on the camera trigger timing, the high-speed camera is controlled to collect images during its reciprocating motion, and the collected images constitute the original image set. During the acquisition process, each frame of the image is timestamped, and each frame of the image is associated with its acquisition time t to obtain a time-series marked image set. Based on the time-series marked image set, the theoretical overlap rate O(t) between adjacent image frames is calculated. The theoretical overlap rate is a measure of the proportion of the same area covered by adjacent image frames in space. By calculating the theoretical overlap rate O(t), the overlap degree between each frame of the camera during the motion process is obtained. According to the image overlap rate data, the time-series marked image set is processed by removing redundant frames and supplementing frames to ensure that the final original image sequence has reasonable spatial coverage and time distribution, reducing redundant information while retaining necessary image information. By removing redundant frames, images with high repetition are removed, thereby reducing the amount of data and improving processing efficiency. For parts with missing information, the missing image frames are supplemented by supplementing frames to ensure the integrity and continuity of the original image sequence, and finally the original image sequence is obtained.

[0014] S2, input the original image sequence into the GPU for parallel processing and extract feature points to obtain the preprocessed image sequence and the target feature point set; Specifically, the original image sequence is divided into multiple image blocks, each image block is assigned to a CUDA thread, and these CUDA threads together form a parallel processing unit. In this way, the parallel computing power of the GPU is utilized so that each image block can be processed quickly and independently. In each parallel processing unit, each image block is subjected to Gaussian filtering to eliminate noise in the image and reduce the interference of noise on feature extraction. The image blocks after Gaussian filtering are subjected to histogram equalization processing to enhance the contrast of the image and make the features of the image more prominent, thereby obtaining enhanced image blocks. Corner point detection is performed on the enhanced image blocks to extract candidate feature points with significant features. Corner points are an important image feature located at positions with high gradient changes, and these positions have strong recognition in the image. Through corner point detection, an initial feature point set is obtained, which contains rich image information. Based on the initial feature point set, the BRIEF descriptor of each feature point is calculated. The BRIEF descriptor is a lightweight binary descriptor that describes the local information of the feature point in a compact manner by encoding the brightness difference of the local area of ​​the image, thereby obtaining a feature description subset. The main direction of the feature description subset is estimated, and the rotation invariance of the feature point is obtained by calculating the main direction of each feature point. Rotation invariance means that the feature point can still maintain a similar description under different rotation angles. Through the main direction estimation, a rotation invariant feature point set is obtained. These feature points have high robustness and can be stably detected under multiple angles and rotation changes. Based on the rotation invariant feature point set, an image pyramid is constructed to extract feature points at different scales. The image pyramid is a structure that gradually reduces the image. By extracting feature points at different scales, a multi-scale feature point set is obtained, which makes the feature points scale invariant and can adapt to changes in images of different sizes and resolutions, thereby enhancing the robustness of detection. Feature point verification is performed on the multi-scale feature point set to eliminate noise and false detection points and obtain the target feature point set. At the same time, in the process of extracting and verifying feature points of the image block, the enhanced image block is reorganized into a complete image to ensure that the processed image has the same structure and content as the original image. The complete image after enhancement processing and feature point extraction can be associated with the target feature point set to obtain a preprocessed image sequence.

[0015] S3, image matching is performed based on the target feature point set, and the RANSAC algorithm is used to eliminate incorrect matching points. The camera pose transformation matrix is ​​obtained through the consistency check of the inner point neighborhood space; It should be noted that the image space is divided into multiple levels of grids based on the target feature point set to obtain a multi-layer grid structure. The purpose of the multi-layer grid structure is to organize and manage feature points at different spatial resolutions so that the matching of feature points is performed in a finer-grained local area. Based on the multi-layer grid structure, local descriptor matching is performed on the feature points in each grid to obtain an initial matching point pair set. By matching in the local grid, the accuracy of matching is improved and the probability of false matching is reduced. Global consistency constraints are imposed on the initial matching point pair set. The spatial relationship between the matching point pairs is calculated, and the matching point pairs that do not conform to the global transformation model are eliminated by analyzing the geometric relationship between these point pairs to obtain an optimized matching point pair set. Based on the optimized matching point pair set, a graph structure is constructed, and the matching point pairs are used as nodes of the graph, and the spatial relationship between the nodes is calculated as the edge weight of the graph to obtain a matching relationship graph. The graph structure is used to describe the mutual relationship between the matching point pairs, and then the matching relationship graph is optimized by graph cut, and the optimal matching point pair set is selected by minimizing the energy function. Through graph cut optimization, point pairs that do not meet local consistency are effectively eliminated to obtain a set of locally consistent matching point pairs. These point pairs have good consistency in space and describe the correspondence between adjacent images. The set of locally consistent matching point pairs is input into the RANSAC algorithm, which further eliminates incorrect matching points through iterative sampling and verification. RANSAC randomly samples matching point pairs, calculates their transformation models, and verifies whether other point pairs meet the model, thereby eliminating those non-compliant external points and obtaining an inlier matching set. The inlier matching set is checked for neighborhood spatial consistency. By calculating the neighborhood transformation consistency score of each matching point pair, matching point pairs with high consistency are screened out to obtain an exact matching point pair set, ensuring that each matching point pair and its surrounding neighborhood area are consistent in spatial transformation. Based on the exact matching point pair set, the singular value decomposition algorithm is used to solve the rigid body transformation matrix that describes the camera motion between adjacent images to obtain the camera pose transformation matrix. The singular value decomposition algorithm is an effective linear algebra tool for solving the optimal rigid body transformation between two point sets. It can minimize the error between the transformed point set and the target point set, thereby accurately describing the spatial relationship between cameras and finally obtaining the camera pose transformation matrix that describes the camera motion between adjacent images.

[0016] S4, dynamically corrects the pre-processed image sequence according to the camera pose transformation matrix, uses the structured light stereo vision algorithm, and combines the projection coded grating pattern analysis to obtain high-precision three-dimensional point cloud data; Specifically, based on the camera pose transformation matrix, an affine transformation is performed on each frame of the preprocessed image sequence. Through the affine transformation, the geometric distortion in the image is corrected to make the spatial coordinates of the image more accurate, and the corrected image sequence is obtained. The corrected image sequence is decomposed by Gaussian pyramid, and a multi-scale representation of the image is constructed through multiple downsampling to form an image pyramid structure. Based on the image pyramid structure, a multi-frequency phase-shifted grating encoding mode is designed to generate a coded projection pattern for structured light projection. Multi-frequency phase-shifted grating encoding is a structured light encoding method that improves the extraction accuracy of phase information by phase shifting gratings of different frequencies. The coded projection pattern is projected onto the surface of the sample precision structural part, and the reflected grating pattern is collected by a high-speed camera to obtain a first reflected image sequence. The first reflected image sequence contains the deformation of the grating pattern caused by the surface morphology of the structural part. The first reflected image sequence is subjected to anti-distortion processing and denoising filtering. The purpose of anti-distortion processing is to eliminate image distortion caused by lens distortion or projection angle, thereby ensuring the geometric accuracy of the reflected image; while denoising filtering is used to reduce random noise in the image to improve the accuracy of phase extraction. The second reflected image sequence is obtained through anti-distortion processing and denoising filtering. Based on the second reflected image sequence, the phase unwrapping algorithm is used to extract the absolute phase information of the reflected pattern to obtain a phase map. The phase unwrapping algorithm converts the periodic phase information into continuous phase values ​​by processing it, and describes the phase state of each pixel under the coded grating. According to the phase map and the calibration parameters of the camera, the mapping relationship between the pixel and the three-dimensional space point is established to generate the initial three-dimensional point cloud data. The camera calibration parameters include the intrinsic and extrinsic parameters of the camera. These parameters describe the imaging model and spatial position relationship of the camera. Through these parameters, the phase value of each pixel is converted into the actual three-dimensional coordinate to obtain the initial three-dimensional point cloud data. The initial three-dimensional point cloud data is statistically outlier removed to eliminate noise points and outliers to obtain the filtered point cloud data. Based on the filtered point cloud data, the local plane fitting algorithm is applied to smooth the point cloud. The local plane fitting algorithm reduces the roughness of the point cloud by fitting a plane in the local area of ​​the point cloud to obtain smoother point cloud data. Smoothing helps eliminate small high-frequency fluctuations caused by noise and enhances the quality of point cloud data. In order to make the distribution of point clouds more uniform, the smoothed point cloud data is gridded and resampled. By adjusting the density distribution of point clouds, the point clouds are evenly distributed on the surface of the entire structure, reducing the complexity of data processing, and improving the accuracy and stability of 3D reconstruction, obtaining high-precision 3D point cloud data.

[0017] S5, inputting high-precision 3D point cloud data into the improved U-Net deep learning network training and transfer learning with geometric perception module to obtain the surface adaptive defect detection model; Among them, the high-precision 3D point cloud data is input into the geometric perception module for feature extraction. The geometric perception module contains 3 3D convolutional layers and 2 pooling layers. The 3D convolutional layer is used to extract local features of the point cloud data, while the pooling layer is used to reduce the resolution of the feature map layer by layer. Each 3D convolutional layer is followed by batch normalization and ReLU activation function to ensure the stability and nonlinear expression ability of the feature. Through this structure, a feature map containing local geometric information is obtained. Based on the extracted local geometric features, an improved U-Net deep learning network is constructed. The structure of the network is designed to include two parts: an encoder and a decoder. The encoder is responsible for downsampling and extracting features layer by layer, and the decoder is responsible for upsampling the encoded features layer by layer to restore them to the original resolution. In the encoder part, it contains 4 downsampling blocks, each of which contains two 3D convolutional layers, a maximum pooling layer and a skip connection. The 3D convolutional layer is used to extract deep features, while the maximum pooling layer is used to reduce the resolution while retaining the main information of the feature, and the skip connection is used to pass the encoder features directly to the decoder so that more detailed information can be integrated during the upsampling process. In the decoder part, there are 4 upsampling blocks, each of which includes a deconvolution layer, two 3D convolution layers and a skip connection. The deconvolution layer is used to upsample the feature map so that the resolution of the feature map is gradually restored, while the skip connection splices the features of the corresponding layer in the encoder with the features in the decoder to achieve effective fusion of multi-scale features. In the process of network forward propagation, the local geometric features are input into the encoder of the improved U-Net, and the multi-scale feature map is obtained through layer-by-layer downsampling and feature extraction. The multi-scale feature map can describe the defect feature information at different scales, which is helpful for better defect identification. The multi-scale feature map is upsampled and feature fused, and the jump connection features of the encoder are spliced ​​with the corresponding features in the decoder to obtain the fused feature map. In order to improve the detection accuracy of the model, an attention mechanism module is designed based on the fused feature map. This module contains two sub-modules: channel attention and spatial attention. The channel attention is used to weight each channel of the feature map and adaptively select more important channel features, while the spatial attention is used to weight different spatial positions of the feature map to highlight the area where the defect is located. Through the combined effect of channel attention and spatial attention, the features are adaptively weighted to obtain an enhanced feature map. The enhanced feature map is input into the segmentation head for classification. The segmentation head contains two 1×1×1 convolutional layers and a Softmax layer. Through the 1×1×1 convolutional layer, the features of each voxel are nonlinearly transformed, and the Softmax layer is used to predict the category of each voxel to obtain the training defect detection results. The design of the segmentation head enables the network to accurately classify each voxel, thereby locating and identifying defects on the surface of precision structural parts.In order to optimize the network parameters, a loss function is designed based on the defect detection results obtained by training. The loss function includes a weighted combination of cross entropy loss, Dice loss and boundary perception loss. Cross entropy loss is used to measure the classification error, Dice loss is used to measure the overlap between the predicted result and the real defect area, so as to ensure the accuracy of the predicted result, and boundary perception loss is used to emphasize the precise detection of the defect edge to improve the model's recognition ability of the defect edge. Through the weighted combination of these losses, the performance of the model is optimized globally, so that the model can better adapt to the defect detection needs of complex surfaces. During the training process, the network parameters are optimized by back propagation, and the transfer learning technology is used to enable the model to adapt to different materials and processes. The application of transfer learning effectively reduces the demand for large-scale labeled data. By migrating the parameters of the pre-trained model to the new task, the convergence of the model can be accelerated, and the detection ability of defects under different materials and process conditions can be improved. A defect detection model with surface adaptive ability is obtained, which can perform high-precision defect detection on precision structural parts with complex surfaces.

[0018] S6, based on the surface adaptive defect detection model, performs full matrix sampling analysis and geometric model compensation on the target three-dimensional point cloud data of the real-time scanning of the precision structural parts to be tested, and obtains the target defect detection results.

[0019] Specifically, the precision structural parts to be tested are scanned in real time to obtain their target three-dimensional point cloud data. The collected point cloud data is voxelized, and the target three-dimensional point cloud data is converted into a three-dimensional grid of a fixed size. Through grid representation, the point cloud data is organized into a structured voxelized tensor. Each grid unit contains the density, normal vector and curvature information of the point. This information constitutes the basic data for describing the surface geometric characteristics of the structural parts. The voxelized tensor is input into the tensor nuclear norm minimization algorithm, and iterative optimization is performed through the alternating direction multiplier method to decompose the tensor into a low-rank background tensor and a sparse foreground tensor. Tensor nuclear norm minimization decomposes complex point cloud data into a smooth background part and a sparse foreground part to better detect and compensate for defects. The low-rank background tensor describes the overall shape of the precision structural parts, while the sparse foreground tensor contains abnormal changes on the surface of the structural parts, that is, potential defects. The low-rank background tensor is subjected to principal curvature analysis, and the Gaussian curvature and mean curvature of each point are calculated to construct a curvature map. Gaussian curvature and mean curvature can describe the local geometric characteristics of the surface of the structural part. Gaussian curvature is used to describe the curvature degree of the surface in two main directions, while mean curvature is used to reflect the overall curvature of the local part. Based on the curvature map, an adaptive mesh subdivision algorithm is used to encrypt the sampling of high curvature areas to obtain a fine background model. High curvature areas are parts of the surface where the shape of the structure changes greatly. Encrypted sampling of these areas can more accurately describe the geometric details of the structure. The fine background model is used to geometrically compensate the sparse foreground tensor, and the distance field from each point to the fine background model is calculated. The distance field is used to quantify the geometric difference between the foreground point and the background model. The foreground data is segmented using a distance threshold, and the points with large differences from the background are marked as defect areas to obtain the compensated foreground data. The compensated foreground data is input into the surface adaptive defect detection model, and the model is used to perform multi-scale feature extraction and attention mechanism enhancement on the foreground data to generate a defect probability heat map. Multi-scale feature extraction is used to capture the feature information of defects at different scales, and the attention mechanism highlights important defect features by adaptively weighting the features, thereby improving the detection accuracy. The defect probability heat map is an image used to indicate the possibility that each point is a defect. The distribution of defect areas can be intuitively observed through the heat map. Based on the defect probability heat map, local adaptive threshold segmentation is performed, and the dynamic segmentation threshold is calculated by combining local curvature, material reflectivity and defect probability distribution. The most suitable threshold is set according to the specific characteristics of each area to achieve fine segmentation of the defect area and obtain the defect boundary. Morphological post-processing is performed on the defect boundary, and morphological post-processing includes weighted opening and closing operations and regional connectivity analysis. Weighted opening and closing operations are used to remove small noise areas to reduce false positives in detection. At the same time, adjacent defect areas are merged through regional connectivity analysis to ensure the integrity and continuity of the detection results. After morphological post-processing, the optimized defect area is obtained.Based on the optimized defect area, multi-dimensional feature vectors are extracted, including geometric features, shape features, and texture features, which are used to describe the specific morphology and distribution of defects. The extracted multi-dimensional feature vectors are used to identify the defect type and evaluate the severity, so that the defects can be classified and quantitatively analyzed in more detail, and the target defect detection results of the precision structural parts to be tested can be obtained. By identifying the defect type, it is determined which factor caused the defect, and by evaluating the severity, the degree of influence of the defect on the structural parts can be determined.

[0020] In one example, a multi-angle synchronous scan is performed on a sample precision structure to obtain a raw image sequence, including: The sample precision structure is fixed on a precision rotating platform, and a preset angular velocity ω is applied to the precision rotating platform to make the sample precision structure rotate at a constant speed in the horizontal plane, and the rotational motion parameters are obtained; and a high-speed camera is installed on a vertical guide rail, and a preset reciprocating velocity v and a preset reciprocating amplitude A are applied to the high-speed camera to make it reciprocate in the vertical direction, and the vertical motion parameters are obtained; Based on the rotational motion parameters and vertical motion parameters, the three-dimensional space trajectory equation S(t) of the high-speed camera is calculated to obtain the camera motion trajectory data. According to the camera motion trajectory data, the trigger frequency f of the high-speed camera is set so that the camera motion trajectory data satisfies the Nyquist sampling theorem and the camera trigger timing is obtained. Based on the camera trigger timing, the high-speed camera is controlled to collect images during the motion process to obtain the original image set, and the original image set is time-stamped, and each frame of the image is associated with the acquisition time t to obtain the time-stamped image set; Based on the time-series marked image set, the theoretical overlap rate O(t) between adjacent image frames is calculated to obtain image overlap rate data. According to the image overlap rate data, the time-series marked image set is processed by redundant frame removal and frame supplementation to obtain the original image sequence.

[0021] In this example, the sample precision structure is fixed on a precision rotating platform, and then a preset angular velocity is applied to the rotating platform. , so that the sample structure can maintain a uniform rotation in the horizontal plane. By controlling the preset angular velocity, the sample rotates around its central axis at a fixed speed so as to comprehensively collect the features of each surface of the sample. The rotational motion parameter is expressed as ,in represents the rotation angle, is the angular velocity of the platform, At the same time, in order to achieve synchronous dynamic scanning of the sample's precision structural parts, a high-speed camera is installed on a vertical rail and a preset reciprocating speed is applied to the camera. and preset reciprocating amplitude By setting the camera's movement, the camera is made to move back and forth in the vertical direction. The speed and amplitude of the movement are determined by and The high-speed camera can scan the sample at different heights and collect information from multiple angles. The vertical camera motion is expressed as ,in represents the vertical displacement of the camera, Represents the angular velocity related to the reciprocating speed of the camera. Through this setting, the vertical motion parameters of the camera are obtained. Based on the above rotational motion parameters and vertical motion parameters, the three-dimensional space trajectory equation of the high-speed camera is calculated. The trajectory equation is expressed as: ; in, represents the radius of the sample rotation platform, is the angular velocity of the platform, is the angular velocity of the camera in the vertical direction, and Represents the reciprocating motion amplitude of the camera in the vertical direction. Through the three-dimensional space trajectory equation , describe the motion trajectory of the high-speed camera around the rotating sample, and obtain the camera motion trajectory data. Based on the camera motion trajectory data, set the trigger frequency of the high-speed camera , so that the camera can collect images at appropriate time intervals. In order to ensure the accuracy of sampling, the trigger frequency needs to satisfy the Nyquist sampling theorem, that is, ,in The highest frequency motion component on the camera trajectory is used to avoid aliasing caused by insufficient sampling. In this way, the trigger timing of the camera is obtained, thereby controlling the image acquisition frequency of the camera during movement. Based on the camera trigger timing, the high-speed camera is controlled to acquire images at specific time points in its motion trajectory to obtain the original image set. During each acquisition process, the camera will capture a frame of image and associate this frame of image with the acquisition time point, adding a timestamp to each frame of image. Each frame of image in the original image set can be associated with a specific acquisition moment. Correspondingly, we get a set of time-series labeled images, denoted as . The theoretical overlap ratio between adjacent image frames is calculated to analyze the spatial coverage of the image. Theoretical overlap ratio It is used to describe the size of the overlapping area covered in space between adjacent frames, which depends on the speed of camera movement, trigger frequency and rotation speed of the sample. The theoretical overlap rate is expressed as: ; in, is the overlapping area between adjacent image frames, and is the total area of ​​a single frame image. , the overlap between adjacent images is obtained, so as to evaluate the quality of image acquisition and data redundancy. , redundant frames are removed and supplemented for the time-stamped image set to obtain a more efficient original image sequence. In the redundant frame removal process, if the overlap rate between adjacent image frames is too high, it means that these frames contain a large amount of repeated information. The redundant frames are removed to reduce the data storage and processing burden. For areas where information is missing due to insufficient sampling or other reasons, the missing image frames are supplemented by frame supplementation to ensure the integrity and spatiotemporal consistency of the original image sequence. By removing and supplementing redundant frames, an optimized original image sequence is obtained.

[0022] In one example, the original image sequence is input into the GPU for parallel processing, and feature points are extracted to obtain a preprocessed image sequence and a target feature point set, including: The original image sequence is divided into multiple image blocks, a CUDA thread is assigned to each image block to obtain a parallel processing unit, and based on the parallel processing unit, Gaussian filtering and histogram equalization processing are performed on each image block to obtain an enhanced image block; Perform corner detection on the enhanced image block, extract candidate feature points, obtain an initial feature point set, and calculate the BRIEF descriptor of each feature point based on the initial feature point set to obtain a feature description subset; The main direction of the feature description subset is estimated, and the rotation invariance of each feature point is calculated to obtain a rotation invariant feature point set. Based on the rotation invariant feature point set, an image pyramid is constructed to extract feature points at different scales to obtain a multi-scale feature point set. Feature point verification is performed on the multi-scale feature point set to obtain the target feature point set, and the enhanced image blocks are reorganized into a complete image and associated with the target feature point set to obtain a preprocessed image sequence.

[0023] In this example, the original image sequence is divided into multiple image blocks. The parallel computing capability of the graphics processing unit (GPU) is utilized to form a parallel processing unit by assigning each image block to a CUDA thread, and multiple image blocks are processed simultaneously. image blocks, each image block size is , each image block is assigned to a CUDA thread for processing. These CUDA threads can operate on their respective image blocks at the same time, greatly improving the efficiency of image processing. In the parallel processing unit, each image block is subjected to Gaussian filtering. Gaussian filtering reduces high-frequency noise in the image by smoothing the image. The mathematical expression of Gaussian filtering is: ; in, Indicates at location The Gaussian weight at Represents the standard deviation of the Gaussian filter, which is used to control the degree of smoothing. After applying Gaussian filtering to each image block, the random noise in the image is significantly reduced, thereby retaining important information in the image and obtaining an enhanced image with less noise. After completing the Gaussian filter, the image block is subjected to histogram equalization. Histogram equalization is an image enhancement technology that adjusts the grayscale value distribution of the image so that it covers the entire possible grayscale range, enhances the contrast, and makes the details clearer. Through processing, the image is more balanced in brightness and contrast, and the effect of feature extraction is improved. Corner detection is performed on each enhanced image block. Corner points refer to points in the image with significant gradient changes, usually located at the intersection of edges or in areas with rich texture details. Corner detection uses the Harris corner detection method, and its mathematical expression is: ; in, represents the autocorrelation matrix of the image gradient, is the determinant of the matrix, is the trace of the matrix, is an empirical constant, usually taken as Through corner point detection, candidate feature points in each image block are extracted to form an initial feature point set. The BRIEF descriptor of each feature point is calculated based on the initial feature point set. The BRIEF descriptor is a binary descriptor used for fast feature matching. By performing a comparison operation in the neighborhood of the feature point, the local features of each feature point are encoded into a compact binary string to obtain a feature description subset. Assume that the feature point The descriptor is ,in It is the comparison result of different pixel pairs in the neighborhood of the feature point. In this way, the feature points are effectively described. The main direction of the feature description subset is estimated, and the rotation invariance of each feature point is calculated. Rotational invariance refers to the ability of feature points to be correctly matched at different rotation angles. By calculating the gradient direction histogram in the neighborhood of the feature point, the main direction is found, and the descriptor of the feature point is rotationally aligned to obtain a rotation-invariant feature point set. Based on the rotation-invariant feature point set, an image pyramid is constructed to extract multi-scale features. The image pyramid is a method of hierarchical representation of images. By downsampling the image layer by layer, an image hierarchy of different resolutions is formed. At each scale, feature points are extracted to obtain a multi-scale feature point set. The construction of the image pyramid enables feature points to be extracted and matched at different scales, thereby achieving scale invariance. Assume that each layer of the pyramid image is ,in Represent the levels of the image pyramid, and extract features from each layer to obtain a multi-scale feature point set. Verify the multi-scale feature point set. By analyzing the spatial consistency of the feature points and the descriptor matching scores, remove the erroneous feature points caused by noise or false detection to obtain the target feature point set. Reassemble the enhanced image blocks into a complete image and associate them with the target feature point set to obtain the preprocessed image sequence.

[0024] In one example, image matching is performed based on the target feature point set, and the RANSAC algorithm is used to remove the wrong matching points. The camera pose transformation matrix is ​​obtained through the consistency check of the inner point neighborhood space, including: Based on the target feature point set, the image space is divided into multiple levels of grids to obtain a multi-layer grid structure. Based on the multi-layer grid structure, local descriptor matching is performed on the feature points in each grid to obtain an initial matching point pair set. Perform global consistency constraints on the initial matching point pair set, calculate the spatial relationship between the matching point pairs, eliminate the matching point pairs that do not conform to the global transformation model, and obtain the optimized matching point pair set; Based on the optimized matching point pair set, a graph structure is constructed. The matching point pairs are used as nodes of the graph. The spatial relationship between the nodes is calculated as the edge weight to obtain a matching relationship graph. The matching relationship graph is then optimized by graph cut to minimize the energy function and obtain a locally consistent matching point pair set. The local consistent matching point pair set is input into the RANSAC algorithm. Through iterative sampling and verification, the outliers are eliminated to obtain the inlier matching set. The inlier matching set is then checked for neighborhood space consistency. The neighborhood transformation consistency score of each matching point pair is calculated. The matching point pairs with high consistency are screened out to obtain the precise matching point pair set. Based on the set of exact matching point pairs, the singular value decomposition algorithm is used to solve the rigid body transformation matrix describing the camera motion between adjacent images to obtain the camera pose transformation matrix.

[0025] In this example, the target feature point set is gridded. The image space is divided into multiple levels of grids to organize and process feature points at different scales, forming a multi-layer grid structure. Assume that the spatial range of the image is , which is divided into The size of each grid is ,in , Through multi-level grid division, feature points at different scales are summarized into different grids, so that feature matching can be performed in finer-grained local areas. Based on the multi-layer grid structure, local descriptors are matched for feature points in each grid to obtain the initial matching point pair set. Suppose in a certain grid, there is a feature point set and ,in and They represent the feature points to be matched in the image. In order to find the best match, the Euclidean distance of the feature descriptor is used to measure the similarity of two feature points. The calculation formula is: ; in Representing feature points and The distance between the descriptors, is the dimension of the descriptor vector, and Respectively represent the first By calculating the distance between each feature point pair, we find the matching point pair with the smallest distance, and form the matching point pairs into an initial matching point pair set. We impose global consistency constraints on the initial matching point pair set, and find matching point pairs that conform to the overall transformation model by analyzing the spatial relationship between the matching point pairs. Assume that the initial matching point pair is , by calculating the relative spatial position relationship between these matching point pairs, point pairs that do not conform to the global transformation model are eliminated. For example, by estimating the affine transformation parameters between matching point pairs, point pairs that deviate from the global model are found and eliminated to obtain an optimized set of matching point pairs. Based on the optimized set of matching point pairs, a graph structure is constructed, each matching point pair is regarded as a node in the graph, and the spatial relationship between the nodes is calculated as the edge weight to obtain a matching relationship graph. Assume that the matching point pair set is , these point pairs are regarded as nodes of the graph, and the spatial relationship between nodes is represented by edge weights. Edge weights are represented by calculating the Euclidean distance between nodes. Assuming that the nodes and nodes The distance between ,but: ; in, For Node and nodes The edge weights between Representing feature points and The Euclidean distance between Representing feature points and The Euclidean distance between them. By constructing a matching relationship graph, the global spatial relationship between feature point pairs is effectively described. The matching relationship graph is optimized by graph cut to minimize the energy function and obtain a set of locally consistent matching point pairs. Graph cut optimization finds matching point pairs with good spatial consistency by minimizing the energy between nodes and edges in the graph. The energy function is designed as the sum of the edge weights between nodes. By minimizing the energy function, matching point pairs that do not meet the local consistency requirements are eliminated to obtain a set of locally consistent matching point pairs. The set of locally consistent matching point pairs is input into the RANSAC algorithm to eliminate incorrect matching points. The RANSAC (random sampling consensus) algorithm identifies and eliminates outliers through iterative sampling and verification to obtain an inlier matching set. In each iteration, RANSAC randomly extracts a small number of samples from the matching point pair set to estimate the transformation model and verify whether other point pairs conform to the model based on the model. After multiple iterations, an inlier matching set that conforms to the consistency model is obtained. The inlier matching set is checked for neighborhood spatial consistency to improve the accuracy of matching points. Calculate the neighborhood transformation consistency score of each matching point pair. The higher the score, the better the consistency of the point pair in the local space. By scoring all matching point pairs, filter out the matching point pairs with high consistency and obtain the set of exact matching point pairs. Based on the set of exact matching point pairs, use the singular value decomposition algorithm to solve the rigid body transformation matrix that describes the camera motion between adjacent images. Assume that the matching point pair is ,in and Represent the corresponding points in two adjacent images respectively. The camera posture transformation matrix is ​​calculated by the singular value decomposition algorithm. and , its mathematical expression is: ; in, represents the rotation matrix, represents the translation vector, Represents matching point pairs and The distance after transformation. By solving the minimization problem, the rigid body transformation matrix of the camera pose is obtained.

[0026] In one example, the pre-processed image sequence is dynamically corrected according to the camera pose transformation matrix, and the structured light stereo vision algorithm is used in combination with the projection coded grating pattern analysis to obtain high-precision 3D point cloud data, including: Based on the camera pose transformation matrix, each frame of the preprocessed image sequence is affine transformed to obtain a corrected image sequence, and the corrected image sequence is decomposed into a Gaussian pyramid to construct a multi-scale image representation to obtain an image pyramid structure. Based on the image pyramid structure, a multi-frequency phase-shift grating coding mode is designed to generate a structured light projection sequence to obtain a coded projection pattern, and the coded projection pattern is projected onto the surface of the sample precision structure, and the reflected image is collected to obtain a first reflected image sequence; Performing anti-distortion processing and denoising filtering on the first reflection image sequence to obtain a second reflection image sequence, and extracting absolute phase information based on the second reflection image sequence using a phase unwrapping algorithm to obtain a phase map; According to the phase map and camera calibration parameters, a mapping relationship between pixels and three-dimensional space points is established to obtain initial three-dimensional point cloud data, and statistical outlier removal is performed on the initial three-dimensional point cloud data to remove noise points and outliers to obtain filtered point cloud data; Based on the filtered point cloud data, a local plane fitting algorithm is applied to smooth the point cloud to obtain smoothed point cloud data. The smoothed point cloud data is gridded and resampled to adjust the point cloud density distribution to obtain high-precision three-dimensional point cloud data.

[0027] In this example, an affine transformation is performed on each frame in the preprocessed image sequence based on the camera pose transformation matrix. Affine transformation is an image transformation technique used to correct geometric distortion of an image so that the objects in the image maintain their proportions and shapes under a new perspective. Assume that the transformation matrix is , the coordinates of a pixel in the preprocessed image are , the new coordinates obtained after affine transformation It is expressed as: ; in, for The transformation matrix is ​​used to describe the translation, rotation and scaling of the camera. Through affine transformation, the rectified image sequence is obtained. The rectified image sequence is decomposed by Gaussian pyramid to construct a multi-scale image representation. Gaussian pyramid is a method for constructing a multi-scale representation of an image. By performing Gaussian smoothing and downsampling on the image layer by layer, a series of images with different resolutions are formed. Assume that the original image is , No. The layer pyramid image is , the image of each layer is obtained by the following formula: ; in, Represents the Gaussian smoothing operation. By smoothing and downsampling the previous layer of image, the image of the next layer is obtained, thereby obtaining images of different resolutions and forming an image pyramid structure. The multi-scale representation of the image pyramid can help capture structural details at different scales. Based on the image pyramid structure, a multi-frequency phase-shifted grating encoding mode is designed to generate a structured light projection sequence and obtain a coded projection pattern. Multi-frequency phase-shifted grating is a technology used to encode three-dimensional surfaces. By projecting gratings of different frequencies and phases, the accuracy of phase information is effectively improved. Let the grating function be ,in represents the amplitude of the grating, represents the spatial frequency of the grating, Indicates the phase shift. By changing The value of is used to generate a series of grating patterns with different phases, and these encoded projection patterns are projected onto the surface of the sample precision structure. At the same time, a high-speed camera is used for synchronous acquisition to obtain the first reflected image sequence. The reflected image contains grating deformation caused by the geometric shape of the sample surface. When processing the first reflected image sequence, anti-distortion processing is performed to eliminate the geometric distortion caused by the camera lens and projection angle. The reflected image is denoised and filtered to reduce the influence of noise on phase extraction to obtain the second reflected image sequence. Based on the second reflected image sequence, the phase unwrapping algorithm is used to extract the absolute phase information to obtain the phase map. The purpose of the phase unwrapping algorithm is to convert the phase in the reflected image from periodic changes to a continuous absolute phase. Assuming the phase is , then the absolute phase Solve by recursive formula: ; in, It is an integer multiple of the unfolded phase and is used to eliminate phase ambiguity. The phase map obtained by phase unfolding can accurately describe the surface morphology of the structural component. According to the phase map and the calibration parameters of the camera, the mapping relationship between pixels and three-dimensional space points is established to obtain the initial three-dimensional point cloud data. The camera calibration parameters include the intrinsic and extrinsic parameters of the camera, which are used to describe the imaging model and relative position of the camera. Through the camera intrinsic parameter matrix and the external parameter matrix , pixel coordinates Mapping to points in 3D space , the mapping relationship is expressed as: ; in, is the scale factor. By solving this mapping relationship, the three-dimensional coordinates corresponding to each pixel are obtained, thereby obtaining the initial three-dimensional point cloud data. The initial three-dimensional point cloud data contains some noise points and outliers due to the influence of measurement errors and noise. In order to improve the quality of point cloud data, statistical outlier removal is performed. By calculating the mean and standard deviation of the points in the neighborhood of each point, points that deviate from the normal distribution are identified and removed to obtain the filtered point cloud data. Assume that a point in the point cloud The point set in the neighborhood of , the average distance between a point and other points in its neighborhood is , the standard deviation is , then if the point The distance exceeds ( is a constant), the point is considered an outlier and is removed. Based on the filtered point cloud data, local plane fitting is performed to smooth the point cloud. The local plane fitting algorithm fits a plane in the neighborhood of each point to reduce the roughness and high-frequency noise of the point cloud and obtain smoothed point cloud data. Set point The neighborhood point set of , fit a plane equation by the least squares method: ; in, are the parameters of the plane equation, and these parameters are obtained by fitting to obtain the description point The optimal model of the neighborhood plane is obtained, and the point cloud data is smoothed using the plane. The smoothed point cloud data is grid-resampled to adjust the density distribution of the point cloud to obtain high-precision 3D point cloud data, so that the point cloud is evenly distributed on the entire surface of the structural part, and the key areas are properly encrypted to improve the accuracy of 3D reconstruction. By uniformly sampling the point cloud, the amount of calculation is reduced while maintaining the integrity and details of the reconstructed surface.

[0028] In one example, high-precision 3D point cloud data is input into the improved U-Net deep learning network training and transfer learning with a geometric perception module to obtain a surface adaptive defect detection model, including: The high-precision 3D point cloud data is input into the geometry perception module, which contains 3 3D convolutional layers and 2 pooling layers. Each 3D convolutional layer is followed by batch normalization and ReLU activation function to obtain local geometric features. Based on local geometric features, an improved U-Net deep learning network is constructed. The improved U-Net deep learning network includes an encoder and a decoder. The encoder contains 4 downsampling blocks, each of which contains two 3D convolutional layers, a maximum pooling layer and a skip connection; the decoder contains 4 upsampling blocks, each of which contains a deconvolution layer, two 3D convolutional layers and a skip connection; The local geometric features are input into the encoder of the improved U-Net deep learning network, and a multi-scale feature map is obtained through layer-by-layer downsampling and feature extraction. The multi-scale feature map is then upsampled and feature fused, and the jump connection features of the encoder are concatenated with the corresponding decoder features to obtain a fused feature map. Based on the fusion feature map, an attention mechanism module is designed, which includes two sub-modules: channel attention and spatial attention. The features are adaptively weighted to obtain an enhanced feature map, which is then input into the segmentation head. The segmentation head contains two 1×1×1 convolutional layers and a Softmax layer to predict the category of each voxel and obtain the training defect detection result. Based on the training defect detection results, a loss function is designed, including a weighted combination of cross entropy loss, Dice loss and boundary-aware loss. The network parameters are optimized through back propagation, and the transfer learning technology is used to adapt to different materials and processes to obtain a surface adaptive defect detection model.

[0029] In this example, high-precision 3D point cloud data is input into the geometry perception module, which contains 3 3D convolutional layers and 2 pooling layers. Each 3D convolutional layer is followed by batch normalization and ReLU activation function. The 3D convolutional layer is used to extract local geometric features from the point cloud data, and the pooling layer is used to reduce the resolution of the feature map layer by layer to make feature extraction more efficient. Batch normalization is used to standardize the distribution of each batch of data to help the network converge faster, and the ReLU activation function introduces nonlinearity to enable the neural network to express complex mapping relationships. Assume that the input 3D point cloud data is ,in , , Respectively represent the dimensions of the 3D point cloud in depth, height, and width, Represents the number of channels. Each 3D convolution operation is represented as: ; in, is the convolution kernel, is the bias term, * represents the convolution operation, is the ReLU activation function. After three convolutions and pooling, the features obtained are used to describe the local geometric features of the point cloud. The local geometric features contain the shape and topology information of the local area of ​​the surface of the structural part. Based on the extracted local geometric features, an improved U-Net deep learning network is constructed. U-Net is a classic architecture for image segmentation, which is improved to be suitable for three-dimensional point cloud data. The structure of the improved U-Net includes two parts: the encoder and the decoder. The encoder is used to extract the features of the layer-by-layer downsampling, and the decoder is used to gradually restore the resolution and fuse multi-scale information. In the encoder, there are 4 downsampling blocks, each of which contains two three-dimensional convolutional layers, a maximum pooling layer and a skip connection. The maximum pooling layer is used to reduce the size of the feature map, thereby aggregating more spatial information layer by layer; and the skip connection is used to directly pass the features in the encoder to the decoder, so that the decoder can fuse the detail information in the process of restoring the resolution. The decoder also contains 4 upsampling blocks, each of which includes a deconvolution layer, two three-dimensional convolutional layers and a skip connection. The deconvolution layer is used to upsample the feature map and restore its size to the same as the input for step-by-step reconstruction. In each upsampling block, the features of the corresponding jump connection in the encoder are concatenated with the features of the current decoder to achieve feature fusion. Assume that the features of the encoder are and the decoder is characterized by , the fused features are expressed as: ; Among them, Concat Represents the concatenation operation. In this way, the high-resolution features in the encoder are fused with the low-resolution features in the decoder to obtain a more refined feature representation. Based on the fused feature map, an attention mechanism module is designed to improve the representation ability of the feature. The attention mechanism module contains two sub-modules: channel attention and spatial attention. Channel attention is used to adaptively weight the channel dimension of the feature map, while spatial attention is used to weight the spatial dimension. The calculation of channel attention is expressed as: ; in, represents the channel attention weight, Represents the Sigmoid activation function, AvgPool and MaxPool Represent global average pooling and global maximum pooling respectively, and is the weight of the fully connected layer. By weighting each channel, the important features related to the defect are highlighted. Similarly, spatial attention weights the features by calculating the importance of each spatial position, so that attention is focused on the area with defects. The feature map enhanced by the attention mechanism is input into the segmentation head for classification. The segmentation head contains two Convolutional layers and a Softmax layer. The convolutional layer is used to perform nonlinear transformation on each voxel, and the Softmax layer is used to predict the category of each voxel, thereby obtaining the defect detection result of the training. Assume that the prediction result of the voxel is , represents the voxel The probability of belonging to the defect category is output through the Softmax layer to generate the defect probability map. During the training process, in order to optimize the parameters of the network, a loss function is designed to measure the difference between the predicted results and the true labels. The loss function is composed of a weighted combination of cross entropy loss, Dice loss, and boundary-aware loss. Cross entropy loss is used to measure the difference between the predicted probability and the true category, and its expression is: ; in, is the true label, Voxel The predicted probability of is the total number of voxels. Dice loss is used to measure the degree of overlap between the predicted results and the actual defect area, and is suitable for unbalanced data sets. Boundary-aware loss is used to enhance the network's sensitivity to defect boundaries, thereby improving the ability to accurately locate defect areas. Through the back-propagation algorithm, the network parameters are optimized based on the loss function, so that the network can gradually learn to identify and distinguish defect areas. In order to improve the adaptability of the model under different materials and process conditions, transfer learning technology is used. Transfer learning effectively reduces the need for a large amount of labeled data and accelerates the convergence of the model by utilizing model parameters that have been pre-trained on large-scale data sets. For example, the model parameters trained on the metal surface defect dataset are migrated to the surface defect detection of plastic products. The model is adapted to the new process conditions and material properties through fine-tuning to obtain a highly adaptable surface adaptive defect detection model.

[0030] The cross entropy loss is calculated for the training defect detection results and the real labeled data. The softmax function is used to convert the network output into a probability distribution. Then, the cross entropy between the predicted probability and the real label is calculated to obtain the classification loss value. The Dice coefficient is calculated based on the training defect detection results and the real labeled data. The predicted results and the real labels are regarded as sets. The intersection area is calculated and divided by the union area. The Dice coefficient is then converted into the Dice loss to obtain the area overlap loss value. The edge of the training defect detection results is extracted. The 3×3 Sobel operator is used to calculate the gradients in the x and y directions respectively. Then The square root of the sum of the gradients in two directions is taken to obtain the predicted boundary map; the edge of the real annotated data is extracted, and the Sobel edge extraction method that is the same as the predicted result is used to obtain the real boundary map; the boundary perception loss is calculated based on the predicted boundary map and the real boundary map, and the difference between the two boundary maps is calculated using the binary cross entropy loss function to obtain the boundary loss value; the classification loss value, the area overlap loss value, and the boundary loss value are weighted and combined to obtain the total loss function; based on the total loss function, the Adam optimizer is used to update the parameters, the learning rate is set to 0.001, and each mini-batch contains 32 samples. The training was performed for 100 epochs to obtain the optimized network parameters. A source domain dataset was selected, which contained sample precision structural parts of different materials and processes. The optimized network was pre-trained. An alternating training strategy was adopted. One material or process was randomly selected for training in each epoch to obtain a pre-trained model. A target domain dataset was selected, which contained samples of precision structural parts of specific materials and processes to be tested. The pre-trained model was fine-tuned to obtain a fine-tuned model. Based on the fine-tuned model, a domain adaptation layer was designed. A gradient reversal layer was added after the feature extraction layer. A domain classifier was constructed. An adversarial training strategy was adopted to reduce the feature separation between the source domain and the target domain. The domain adaptation model is obtained through the adversarial process of minimizing the feature extractor loss and maximizing the domain classifier loss; knowledge distillation is performed on the domain adaptation model, and the teacher network and the student network are designed. A combination of soft labels and hard labels with a temperature of 4 is used as the distillation target, and KL divergence is used as the distillation loss. The distillation training is performed for 50 epochs to obtain a lightweight model; the lightweight model is deployed to the edge computing device, 8-bit fixed-point quantization is used to convert floating-point numbers into integers, and a pruning algorithm is used to remove 90% of non-important connections. Finally, retraining is performed for 5 epochs to obtain a surface adaptive defect detection model.

[0031] In one example, based on the surface adaptive defect detection model, full matrix sampling analysis and geometric model compensation are performed on the target three-dimensional point cloud data of the real-time scanning of the precision structural parts to be tested, and the target defect detection results are obtained, including: Scan the target 3D point cloud data of the precision structural parts to be tested in real time, and voxelize the target 3D point cloud data to convert the target 3D point cloud data into a 3D grid of fixed size. Each grid unit contains the density, normal vector and curvature information of the point, and obtains the voxelized tensor. The voxelized tensor is input into the tensor nuclear norm minimization algorithm, and the tensor is decomposed into a low-rank background tensor and a sparse foreground tensor through iterative optimization using the alternating direction multiplier method; The principal curvature analysis is performed on the low-rank background tensor, the Gaussian curvature and mean curvature of each point are calculated, and a curvature map is constructed. Based on the curvature map, an adaptive grid subdivision algorithm is used to perform encrypted sampling on the high curvature area to obtain a refined background model. The sparse foreground tensor is geometrically compensated using a fine background model, the distance field from each point to the fine background model is calculated, and the foreground is segmented using a distance threshold to obtain compensated foreground data; The compensated foreground data is input into the surface adaptive defect detection model, and the defect probability heat map is generated through multi-scale feature extraction and attention mechanism enhancement; Based on the defect probability heat map, local adaptive threshold segmentation is performed. The dynamic segmentation threshold is calculated by combining the local curvature, material reflectivity and defect probability distribution, and the defect area is finely segmented to obtain the defect boundary. Morphological post-processing is performed on the defect boundary. Morphological post-processing includes weighted opening and closing operations and regional connectivity analysis to remove small area noise and merge adjacent defects to obtain the optimized defect area. Based on the optimized defect area, multidimensional feature vectors are extracted, including geometric features, shape features and texture features. The defect type and severity evaluation are performed on the multidimensional feature vectors to obtain the target defect detection results of the precision structural parts to be tested.

[0032] In this example, the target three-dimensional point cloud data is scanned in real time. The three-dimensional point cloud data of the precision structural parts to be tested is obtained through high-precision laser scanning or structured light technology. The point cloud data describes the coordinate position of each point on the surface of the structural part and the corresponding geometric characteristics. The point cloud data is voxelized. The target three-dimensional point cloud data is converted into a three-dimensional grid of a fixed size, in which each voxel (i.e., grid unit) represents a small volume area in the point cloud. The voxelized tensor is formed by counting the density, normal vector, and curvature information of the points in each voxel. ,in , , denote the dimensions of the voxelized tensor in depth, height, and width, respectively. Represents the dimensions of features contained in it, such as density, normal vector, and curvature. Input tensor nuclear norm minimization algorithm, iterative optimization by alternating direction multiplier method. The purpose of tensor nuclear norm minimization is to decompose the voxelized tensor into low-rank background tensor and sparse foreground tensor, so as to better separate normal structure and abnormal area. The objective function of tensor nuclear norm minimization is expressed as: ; in, is the low-rank background tensor, is the sparse foreground tensor, Representing a tensor The nuclear norm of Representing a tensor of norm, is a regularization parameter used to control the balance between low-rank terms and sparse terms. The tensor is effectively Decomposed into a low-rank background tensor that describes the overall morphology of the structure and a sparse foreground tensor containing the anomaly variation . For the low-rank background tensor Perform principal curvature analysis to calculate the Gaussian curvature and mean curvature at each point. Gaussian curvature and mean curvature It is an important parameter used to describe the surface geometry, which can reflect the local curvature of the surface of the structural part. For each point, its Gaussian curvature and mean curvature are calculated by the following formula: ; in, and are the principal curvatures of the surface in two main directions, respectively. Based on the calculated Gaussian curvature and mean curvature, a curvature map is constructed to intuitively represent the geometric characteristics of the surface of the structural component. Based on the curvature map, an adaptive mesh subdivision algorithm is used to perform dense sampling of high curvature areas. High curvature areas correspond to parts with complex surface shapes. By dense sampling these areas, their geometric details are more accurately described, and a fine background model is obtained. Using the fine background model, the sparse foreground tensor is Perform geometric compensation and calculate the distance field from each point to the refined background model. The distance field is used to quantify the geometric difference between the foreground point and the background model. Assume that the foreground point The minimum distance to the background model is , the distance field is used to describe the spatial distribution characteristics of foreground points. The foreground data is segmented using a preset distance threshold, and the points that are significantly different from the background model are marked as defect areas to obtain compensated foreground data. The compensated foreground data is input into the surface adaptive defect detection model, and the model is used to perform multi-scale feature extraction and attention mechanism enhancement on the foreground data to generate a defect probability heat map. Multi-scale feature extraction is used to capture the feature information of defects from different scales, and the attention mechanism enhances important features related to defects by adaptively weighting the features, making the detection results more accurate. The defect probability heat map is a probability distribution map used to indicate that each point is a defect. The defect location and range on the surface of the structural part can be intuitively identified through the heat map. Based on the defect probability heat map, local adaptive threshold segmentation is performed. In order to obtain an accurate defect boundary, the dynamic segmentation threshold is calculated by combining the local curvature, material reflectivity, and defect probability distribution. Let the dynamic threshold be , which is calculated by the following formula: ; in, represents the curvature of a point, Represents the reflectivity of the material. represents the defect probability, , , is the weight parameter. By combining this information, the segmentation threshold of each region is adaptively set to achieve fine segmentation of the defect area and obtain the defect boundary. Morphological post-processing is performed on the defect boundary to optimize the defect detection results. Morphological post-processing includes weighted opening and closing operations and regional connectivity analysis. The opening operation is used to remove small area noise, while the closing operation is used to fill small holes in the defect area. Through regional connectivity analysis, adjacent defect areas are merged to obtain a more coherent and complete defect area. Based on the optimized defect area, a multidimensional feature vector is extracted to describe the geometric features, shape features and texture features of the defect. The geometric features include the area and volume of the defect, the shape features include the aspect ratio and roundness of the defect, and the texture features include information such as the grayscale co-occurrence matrix. The extracted multidimensional feature vector is used to identify the defect type and evaluate the severity. The defects are classified into different types, such as cracks, pits or bubbles, using a machine learning classifier or a deep learning model, and their influence on the performance of the structural parts is evaluated to obtain the target defect detection results of the precision structural parts to be tested.

[0033] Reference Figure 2 This embodiment provides a precision structural part machine vision dynamic defect detection device, including: Scanning module 1, used to perform multi-angle synchronous scanning on the sample precision structure to obtain the original image sequence; Processing module 2, used for inputting the original image sequence into GPU for parallel processing and extracting feature points to obtain a pre-processed image sequence and a target feature point set; Matching module 3 is used to perform image matching based on the target feature point set, and use the RANSAC algorithm to remove incorrect matching points, and obtain the camera pose transformation matrix through the consistency check of the inner point neighborhood space; Correction module 4 is used to dynamically correct the pre-processed image sequence according to the camera posture transformation matrix, using the structured light stereo vision algorithm combined with the projection coded grating pattern analysis to obtain high-precision three-dimensional point cloud data; Training module 5, used for inputting high-precision 3D point cloud data into the improved U-Net deep learning network training and transfer learning containing the geometric perception module to obtain a surface adaptive defect detection model; The detection module 6 is used to perform full matrix sampling analysis and geometric model compensation on the target three-dimensional point cloud data scanned in real time on the precision structural parts to be tested based on the surface adaptive defect detection model, so as to obtain the target defect detection result.

[0034] In this embodiment, for the specific implementation of each unit in the above device embodiment, please refer to the above method embodiment, which will not be repeated here.

[0035] Reference Figure 3 In an embodiment of the present invention, a computer device is also provided. The computer device may be a server, and its internal structure may be as follows: Figure 3 As shown. The computer device includes a processor, a memory, a display screen, an input device, a network interface and a database connected through a system bus. Among them, the processor designed by the computer is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store the corresponding data in this embodiment. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, the above method is implemented.

[0036] Those skilled in the art will understand that Figure 3 The structure shown in the figure is merely a block diagram of a portion of the structure related to the solution of the present invention, and does not constitute a limitation on the computer device to which the solution of the present invention is applied.

[0037] An embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the above method is implemented. It can be understood that the computer-readable storage medium in this embodiment can be a volatile readable storage medium or a non-volatile readable storage medium.

[0038] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media provided by the present invention and used in the embodiments may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double-speed data rate SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM.

[0039] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, device, article or method including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, device, article or method. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the presence of other identical elements in the process, device, article or method including the element.

[0040] The above description is only a preferred embodiment of the present invention, and does not limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made by using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A method for dynamic defect detection of precision structural parts by machine vision, characterized in that: The following steps are involved: Perform multi-angle synchronous scanning on the sample's precision structural parts to obtain the original image sequence; Inputting the original image sequence into the GPU for parallel processing and extracting feature points to obtain a preprocessed image sequence and a target feature point set; Image matching is performed based on the target feature point set, and erroneous matching points are eliminated using the RANSAC algorithm, and the camera pose transformation matrix is ​​obtained through the consistency check of the interior point neighborhood space; Dynamically correcting the preprocessed image sequence according to the camera posture transformation matrix, using a structured light stereo vision algorithm combined with projection coded grating pattern analysis to obtain high-precision three-dimensional point cloud data; Inputting the high-precision three-dimensional point cloud data into an improved U-Net deep learning network training and transfer learning including a geometric perception module to obtain a surface adaptive defect detection model; Based on the surface adaptive defect detection model, full matrix sampling analysis and geometric model compensation are performed on the target three-dimensional point cloud data scanned in real time for the precision structural parts to be tested, so as to obtain the target defect detection results.

2. The method for dynamic defect detection of precision structural parts by machine vision according to claim 1, characterized in that: The method of performing multi-angle synchronous scanning on the sample precision structural part to obtain an original image sequence includes: Fix the sample precision structure on a precision rotating platform, apply a preset angular velocity ω to the precision rotating platform, make the sample precision structure rotate at a constant speed in a horizontal plane, and obtain the rotational motion parameters; install a high-speed camera on a vertical guide rail, apply a preset reciprocating velocity v and a preset reciprocating amplitude A to the high-speed camera, make it reciprocate in the vertical direction, and obtain the vertical motion parameters; Based on the rotational motion parameter and the vertical motion parameter, a three-dimensional space trajectory equation S(t) of the high-speed camera is calculated to obtain camera motion trajectory data, and according to the camera motion trajectory data, a trigger frequency f of the high-speed camera is set so that the camera motion trajectory data satisfies the Nyquist sampling theorem, and a camera trigger timing is obtained; Based on the camera trigger timing, the high-speed camera is controlled to collect images during the motion process to obtain an original image set, and the original image set is time-stamped, and each frame of the image is associated with the collection time t to obtain a time-stamped image set; Based on the time-series marked image set, the theoretical overlap rate O(t) between adjacent image frames is calculated to obtain image overlap rate data, and according to the image overlap rate data, redundant frames are eliminated and frames are supplemented on the time-series marked image set to obtain the original image sequence.

3. The method for dynamic defect detection of precision structural parts by machine vision according to claim 2, characterized in that: The step of inputting the original image sequence into the GPU for parallel processing and extracting feature points to obtain a preprocessed image sequence and a target feature point set includes: The original image sequence is divided into a plurality of image blocks, a CUDA thread is assigned to each image block to obtain a parallel processing unit, and based on the parallel processing unit, Gaussian filtering and histogram equalization processing are performed on each image block to obtain an enhanced image block; Performing corner point detection on the enhanced image block, extracting candidate feature points, obtaining an initial feature point set, and calculating the BRIEF descriptor of each feature point based on the initial feature point set to obtain a feature description subset; Performing main direction estimation on the feature description subset, calculating the rotation invariance of each feature point to obtain a rotation invariant feature point set, and constructing an image pyramid based on the rotation invariant feature point set, extracting feature points at different scales, and obtaining a multi-scale feature point set; Feature point verification is performed on the multi-scale feature point set to obtain a target feature point set, and the enhanced image blocks are reorganized into a complete image and associated with the target feature point set to obtain a preprocessed image sequence.

4. The method for dynamic defect detection of precision structural parts by machine vision according to claim 3, characterized in that: The image matching is performed based on the target feature point set, and the RANSAC algorithm is used to eliminate the wrong matching points, and the camera posture transformation matrix is ​​obtained through the internal point neighborhood space consistency check, including: Performing grid division based on the target feature point set to divide the image space into multiple levels of grids to obtain a multi-layer grid structure, and performing local descriptor matching on the feature points in each grid based on the multi-layer grid structure to obtain an initial matching point pair set; Performing global consistency constraints on the initial matching point pair set, calculating the spatial relationship between the matching point pairs, and eliminating the matching point pairs that do not conform to the global transformation model to obtain an optimized matching point pair set; Based on the optimized matching point pair set, a graph structure is constructed, the matching point pairs are used as nodes of the graph, the spatial relationship between the nodes is calculated as the edge weight, a matching relationship graph is obtained, and the matching relationship graph is optimized by graph cut to minimize the energy function to obtain a locally consistent matching point pair set; The local consistent matching point pair set is input into the RANSAC algorithm, and the outliers are removed through iterative sampling and verification to obtain the inlier matching set, and the neighborhood space consistency check is performed on the inlier matching set, and the neighborhood transformation consistency score of each matching point pair is calculated, and the high consistency matching point pairs are screened out to obtain the accurate matching point pair set; Based on the exact matching point pair set, a singular value decomposition algorithm is used to solve the rigid body transformation matrix describing the camera motion between adjacent images to obtain the camera posture transformation matrix.

5. The method for dynamic defect detection of precision structural parts by machine vision according to claim 4, characterized in that: The method of dynamically correcting the preprocessed image sequence according to the camera posture transformation matrix, using a structured light stereo vision algorithm, and combining projection coded grating pattern analysis to obtain high-precision three-dimensional point cloud data includes: Based on the camera pose transformation matrix, performing affine transformation on each frame of the preprocessed image sequence to obtain a corrected image sequence, and performing Gaussian pyramid decomposition on the corrected image sequence to construct a multi-scale image representation to obtain an image pyramid structure; Based on the image pyramid structure, a multi-frequency phase-shift grating coding mode is designed to generate a structured light projection sequence to obtain a coded projection pattern, and the coded projection pattern is projected onto the surface of the sample precision structure part, and a reflection image is collected to obtain a first reflection image sequence; Performing anti-distortion processing and denoising filtering on the first reflected image sequence to obtain a second reflected image sequence, and extracting absolute phase information using a phase unwrapping algorithm based on the second reflected image sequence to obtain a phase map; According to the phase map and the camera calibration parameters, a mapping relationship between pixels and three-dimensional space points is established to obtain initial three-dimensional point cloud data, and statistical outlier removal is performed on the initial three-dimensional point cloud data to remove noise points and outliers to obtain filtered point cloud data; Based on the filtered point cloud data, a local plane fitting algorithm is applied to perform point cloud smoothing processing to obtain smoothed point cloud data, and the smoothed point cloud data is grid-resampled to adjust the point cloud density distribution to obtain high-precision three-dimensional point cloud data.

6. The method for dynamic defect detection of precision structural parts by machine vision according to claim 5, characterized in that: The high-precision three-dimensional point cloud data is input into the improved U-Net deep learning network training and transfer learning containing the geometric perception module to obtain a surface adaptive defect detection model, including: Inputting the high-precision three-dimensional point cloud data into a geometric perception module, wherein the geometric perception module comprises three three-dimensional convolutional layers and two pooling layers, each of which is followed by batch normalization and a ReLU activation function to obtain local geometric features; Based on the local geometric features, an improved U-Net deep learning network is constructed, wherein the improved U-Net deep learning network includes an encoder and a decoder, wherein the encoder includes four downsampling blocks, each of which includes two three-dimensional convolutional layers, a maximum pooling layer, and a skip connection; and the decoder includes four upsampling blocks, each of which includes a deconvolutional layer, two three-dimensional convolutional layers, and a skip connection; Input the local geometric features into the encoder of the improved U-Net deep learning network, obtain a multi-scale feature map through layer-by-layer downsampling and feature extraction, upsample and feature fuse the multi-scale feature map, and splice the jump connection features of the encoder with the corresponding decoder features to obtain a fused feature map; Based on the fused feature map, an attention mechanism module is designed, which includes two submodules: channel attention and spatial attention. The features are adaptively weighted to obtain an enhanced feature map, and the enhanced feature map is input into a segmentation head. The segmentation head includes two 1×1×1 convolutional layers and a Softmax layer. The category of each voxel is predicted to obtain a training defect detection result. Based on the training defect detection results, a loss function is designed, including a weighted combination of cross entropy loss, Dice loss and boundary perception loss. The network parameters are optimized through back propagation, and the transfer learning technology is used to adapt to different materials and processes to obtain a surface adaptive defect detection model.

7. The method for dynamic defect detection of precision structural parts by machine vision according to claim 6, characterized in that: Based on the surface adaptive defect detection model, full matrix sampling analysis and geometric model compensation are performed on the target three-dimensional point cloud data scanned in real time for the precision structural part to be tested, and the target defect detection result is obtained, including: Scanning the target three-dimensional point cloud data of the precision structural part to be tested in real time, and voxelizing the target three-dimensional point cloud data to convert the target three-dimensional point cloud data into a three-dimensional grid of a fixed size, where each grid unit contains the density, normal vector and curvature information of the point, to obtain a voxelized tensor; Inputting the voxelized tensor into a tensor nuclear norm minimization algorithm, and iteratively optimizing the tensor by an alternating direction multiplier method to decompose the tensor into a low-rank background tensor and a sparse foreground tensor; Performing principal curvature analysis on the low-rank background tensor, calculating the Gaussian curvature and mean curvature of each point, constructing a curvature map, and based on the curvature map, using an adaptive grid subdivision algorithm to perform encrypted sampling on high curvature areas to obtain a refined background model; Using the fine background model to geometrically compensate the sparse foreground tensor, calculating the distance field from each point to the fine background model, and using the distance threshold to perform foreground segmentation to obtain compensated foreground data; Inputting the compensated foreground data into the surface adaptive defect detection model, and generating a defect probability heat map through multi-scale feature extraction and attention mechanism enhancement; Based on the defect probability heat map, local adaptive threshold segmentation is performed, and the dynamic segmentation threshold is calculated by combining the local curvature, material reflectivity and defect probability distribution, and the defect area is finely segmented to obtain the defect boundary; Performing morphological post-processing on the defect boundary, the morphological post-processing including weighted opening and closing operations and regional connectivity analysis, removing small area noise and merging adjacent defects to obtain an optimized defect area; Based on the optimized defect area, a multidimensional feature vector is extracted, including geometric features, shape features and texture features, and the defect type identification and severity evaluation are performed on the multidimensional feature vector to obtain the target defect detection result of the precision structural part to be tested.

8. A precision structural parts machine vision dynamic defect detection device, characterized in that: For implementing the steps of the method according to any one of claims 1 to 7, the device comprises: A scanning module is used to perform multi-angle synchronous scanning on the sample precision structural parts to obtain the original image sequence; A processing module, used for inputting the original image sequence into the GPU for parallel processing and extracting feature points to obtain a pre-processed image sequence and a target feature point set; A matching module is used to perform image matching based on the target feature point set, and to eliminate erroneous matching points using the RANSAC algorithm, and to obtain a camera pose transformation matrix through an inlier neighborhood space consistency check; A correction module is used to dynamically correct the pre-processed image sequence according to the camera posture transformation matrix, using a structured light stereo vision algorithm combined with projection coded grating pattern analysis to obtain high-precision three-dimensional point cloud data; A training module, used for inputting the high-precision three-dimensional point cloud data into an improved U-Net deep learning network training and transfer learning including a geometric perception module to obtain a surface adaptive defect detection model; The detection module is used to perform full matrix sampling analysis and geometric model compensation on the target three-dimensional point cloud data scanned in real time on the precision structural parts to be tested based on the surface adaptive defect detection model to obtain the target defect detection result.

9. A computer device comprising a memory and a processor, wherein a computer program is stored in the memory, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images