A ball screw thread defect detection method and system integrating computer vision

Abstract

The present application relates to the field of computer vision technology, and discloses a method and system for detecting defects in ball screw threads that integrate computer vision. The method includes: performing binocular stereo imaging on the ball screw to be tested, obtaining a stereo image of the thread and constructing a three-dimensional point cloud model of the thread; performing pyramid decomposition, morphological filtering and fusion to obtain a comprehensive feature map of the thread; constructing an ideal thread generatrix equation and calculating the thread pitch deviation value; constructing a thread valley cone surface equation between two adjacent threads and calculating the thread tooth profile deviation value; inputting the thread stereo image into a fully convolutional network model for semantic segmentation of defect distribution, and combining the pitch constraint of the thread pitch deviation value and the tooth profile constraint of the thread tooth profile deviation value to obtain a pixel-level defect distribution map; back-projecting the pixel-level defect distribution map to the three-dimensional point cloud model of the thread, and calculating the three-dimensional morphological parameters and defect severity level of the defect area point cloud. The present application improves the accuracy of ball screw thread defect detection.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a ball screw thread defect detection method and system integrating computer vision. Background Art

[0002] As a key component in mechanical transmission devices, the quality and precision of ball screws directly affect the performance and life of mechanical systems. However, various defects such as thread pitch deviation and tooth profile deviation are inevitable in the process of manufacturing and using ball screws.

[0003] Existing technologies mainly rely on manual inspection and traditional contact inspection methods, which have the problems of low efficiency, insufficient precision, and test results that are easily affected by subjective factors. Manual inspection requires highly skilled operators, and long hours of work can easily cause fatigue, resulting in unstable test results; contact inspection methods can easily damage the surface of the ball screw, affecting its subsequent use. In addition, traditional methods often have difficulty in providing comprehensive and accurate test data when faced with complex thread structures and minor defects. Summary of the invention

[0004] The present application provides a ball screw thread defect detection method and system integrating computer vision, and the present application improves the accuracy of ball screw thread defect detection.

[0005] In a first aspect, the present application provides a ball screw thread defect detection method integrated with computer vision, the ball screw thread defect detection method integrated with computer vision comprising:

[0006] Perform binocular stereo imaging on the ball screw to be tested to obtain a thread stereo image, and construct a thread three-dimensional point cloud model based on the thread stereo image;

[0007] Performing pyramid decomposition on the thread stereo image to obtain a plurality of initial sub-images, and performing morphological filtering and fusion on the plurality of initial sub-images to obtain a thread comprehensive feature map;

[0008] Constructing an ideal thread generatrix equation of the thread profile according to the thread comprehensive characteristic diagram, and calculating the corresponding thread pitch deviation value;

[0009] Constructing a thread valley cone surface equation between two adjacent threads according to the thread comprehensive characteristic diagram, and calculating a thread profile deviation value;

[0010] Input the thread stereo image into a preset fully convolutional network model to perform semantic segmentation of defect distribution, and combine the pitch constraint corresponding to the thread pitch deviation value and the tooth profile constraint corresponding to the thread tooth profile deviation value to obtain a pixel-level defect distribution map;

[0011] The pixel-level defect distribution map is back-projected onto the thread three-dimensional point cloud model to obtain a defect area point cloud, and the three-dimensional morphological parameters and defect severity level of the defect area point cloud are calculated.

[0012] In a second aspect, the present application provides a ball screw thread defect detection system integrated with computer vision, the ball screw thread defect detection system integrated with computer vision comprising:

[0013] An imaging module is used to perform binocular stereo imaging on the ball screw to be tested, obtain a thread stereo image, and construct a thread three-dimensional point cloud model based on the thread stereo image;

[0014] A decomposition module, used for performing pyramid decomposition on the thread stereo image to obtain a plurality of initial sub-images, and performing morphological filtering and fusion on the plurality of initial sub-images to obtain a thread comprehensive feature map;

[0015] A first calculation module is used to construct an ideal thread generatrix equation of the thread profile according to the thread comprehensive characteristic diagram, and calculate the corresponding thread pitch deviation value;

[0016] A second calculation module is used to construct a thread valley cone surface equation between two adjacent threads according to the thread comprehensive characteristic diagram, and calculate the thread profile deviation value;

[0017] A semantic segmentation module, used for inputting the thread stereo image into a preset fully convolutional network model to perform semantic segmentation of defect distribution, and combining the pitch constraint corresponding to the thread pitch deviation value and the tooth profile constraint corresponding to the thread tooth profile deviation value to obtain a pixel-level defect distribution map;

[0018] The projection module is used to back-project the pixel-level defect distribution map onto the thread three-dimensional point cloud model to obtain the defect area point cloud, and calculate the three-dimensional morphological parameters and defect severity level of the defect area point cloud.

[0019] The third aspect of the present application provides a computer device, comprising: a memory and at least one processor, wherein the memory stores instructions; the at least one processor calls the instructions in the memory so that the computer device executes the above-mentioned ball screw thread defect detection method with integrated computer vision.

[0020] A fourth aspect of the present application provides a computer-readable storage medium, in which instructions are stored, and when the computer-readable storage medium is run on a computer, the computer executes the above-mentioned ball screw thread defect detection method integrated with computer vision.

[0021] In the technical solution provided by the present application, the three-dimensional point cloud model of the ball screw thread is accurately obtained through binocular stereo imaging technology. Stereo matching and triangulation methods are used to ensure the accuracy and integrity of three-dimensional reconstruction. Pyramid decomposition and morphological filtering techniques are used to extract multi-scale thread comprehensive feature maps, enhance local features of the image, provide more detailed information, and improve the sensitivity of defect detection. The ideal thread generatrix equation and thread valley cone surface equation are constructed to calculate the thread pitch deviation value and thread tooth profile deviation value respectively to ensure the accurate measurement of thread geometric parameters. The geometric features of the thread profile and thread valley area are extracted to accurately locate defects. Combined with the fully convolutional network model, pixel-level semantic segmentation of defect distribution is achieved to improve the accuracy and automation of detection. The thread pitch deviation value and thread tooth profile deviation value are embedded in the network model as prior knowledge parameters to optimize the semantic segmentation results and improve the robustness and generalization ability of the model. Through the back-projection technology, the pixel-level defect distribution map is mapped back to the three-dimensional point cloud model to obtain the three-dimensional morphological parameters of the defect area. The volume, surface area, three-dimensional size and other parameters of the defect area are calculated, and the defect severity level is predicted through the support vector machine regression model. Multiple defect point cloud clusters and their three-dimensional morphological parameters are visualized to generate intuitive three-dimensional visualization results of thread defects, thereby improving the accuracy of ball screw thread defect detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0023] Figure 1 A schematic diagram of an embodiment of a ball screw thread defect detection method integrating computer vision in an embodiment of the present application;

[0024] Figure 2 Schematic diagram of an embodiment of a ball screw thread defect detection system integrating computer vision in an embodiment of the present application. DETAILED DESCRIPTION

[0025] The embodiments of the present application provide a method and system for detecting thread defects of a ball screw with integrated computer vision. The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" or "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0026] For ease of understanding, the specific process of the embodiment of the present application is described below. Figure 1 In the embodiment of the present application, an embodiment of the ball screw thread defect detection method integrating computer vision includes:

[0027] Step 101, performing binocular stereo imaging on the ball screw to be tested to obtain a thread stereo image, and constructing a thread three-dimensional point cloud model according to the thread stereo image;

[0028] It is understandable that the execution subject of the present application can be a ball screw thread defect detection system integrated with computer vision, or a terminal or a server, which is not limited here. The present application embodiment is described by taking the server as the execution subject as an example.

[0029] Specifically, a binocular stereo imaging is performed on the ball screw to be tested by a binocular camera to obtain the thread axial section and thread generatrix projection. The camera calibration is performed on these images to determine the internal and external parameter matrix of the binocular camera. The parameters include the focal length, principal point coordinates, rotation matrix and translation vector of the camera, which can accurately describe the imaging geometric characteristics of the camera. According to the internal and external parameter matrix, the thread axial section and thread generatrix projection are polar corrected to eliminate the geometric distortion that may occur in the stereo imaging process, and the corrected stereo image pair is obtained. On this basis, the corrected stereo image pair is stereo matched to generate a disparity map. The disparity map reflects the depth information of the object surface by calculating the displacement of the corresponding points in the image pair. Using the disparity map and the internal and external parameter matrix of the camera, the first depth map of the thread surface is calculated by triangulation method to obtain the three-dimensional structural information of the thread. The first depth map is denoised and smoothed to obtain a more accurate second depth map, reduce noise interference, smooth the change of depth value, and make the depth information of the thread surface more reliable. The second depth map and the corrected stereo image pair are used to perform depth color fusion to generate a thread stereo image. Generate point cloud from the thread stereo image to obtain the initial thread 3D point cloud. Point cloud data is a collection of a large number of points in 3D space. Each point has a clear spatial coordinate and can accurately describe the 3D shape of the object. Perform statistical outlier filtering and voxel downsampling on the initial thread 3D point cloud. Statistical outlier filtering can remove noise points and outliers in the point cloud. Voxel downsampling divides the point cloud into multiple voxels (volume pixels) and replaces all points in each voxel with the representative point in the voxel, thereby reducing the amount of point cloud data and improving processing efficiency to obtain the target thread 3D point cloud. Perform normal vector estimation and surface reconstruction on the target thread 3D point cloud. Normal vector estimation is used to calculate the normal vector of each point, that is, the local surface direction of the point. Surface reconstruction generates a complete thread 3D point cloud model by converting point cloud data into a continuous surface model.

[0030] Step 102: performing pyramid decomposition on the thread stereo image to obtain a plurality of initial sub-images, and performing morphological filtering and fusion on the plurality of initial sub-images to obtain a thread comprehensive feature map;

[0031] Specifically, Gaussian filtering is performed on the thread stereo image to obtain a smoothed thread image. Laplacian pyramid decomposition is performed on the smoothed thread image to obtain multiple initial sub-images of different scales. Laplacian pyramid decomposition is a multi-scale image processing method that obtains detailed information of the image at different scales by decomposition. According to the scale of each initial sub-image, morphological structural elements of different sizes are selected to obtain multiple scale-adaptive structural elements. By adaptively selecting the size of the structural element, the thread features at different scales are better retained and highlighted. Based on the scale-adaptive structural element, each initial sub-image is opened and closed respectively to obtain multiple morphologically filtered sub-images. Opening and closing operations are two basic morphological operations that can effectively remove noise and small holes in the image and smooth the boundary of the target area. Top-hat transformation and bottom-hat transformation are performed on the morphologically filtered sub-image to obtain multiple enhanced sub-images that highlight local features. Top-hat transformation and bottom-hat transformation can enhance the local brightness and darkness changes in the image, making the detailed features in the image more obvious. Wavelet transformation is performed on the enhanced sub-image to obtain low-frequency coefficients and high-frequency coefficients of each enhanced sub-image. Wavelet transform is a time-frequency analysis tool that decomposes an image into components of different frequencies. The low-frequency coefficients contain the overall information of the image, and the high-frequency coefficients contain the detailed information of the image. The local energy and directional consistency of each enhanced sub-image are calculated based on the low-frequency coefficients and high-frequency coefficients to obtain the feature weight of each enhanced sub-image. The local energy and directional consistency can reflect the feature intensity and directional distribution of the image in different regions. According to the feature weights, multiple enhanced sub-images are weighted fused to obtain the initial fused image. The initial fused image is processed by non-local mean denoising and adaptive histogram equalization to obtain a thread comprehensive feature map. Non-local mean denoising can effectively remove noise in the image while retaining the detailed information of the image; adaptive histogram equalization can enhance the contrast of the image, making the thread features more obvious and clear.

[0032] Step 103: construct an ideal thread generatrix equation of the thread profile according to the thread comprehensive characteristic diagram, and calculate the corresponding thread pitch deviation value;

[0033] Specifically, edge detection is performed on the thread comprehensive feature map to extract the thread contour line. The extracted thread contour line is tracked to obtain the coordinate sequence of the thread contour line. The coordinate sequence of the thread contour line is fitted by the least square method to obtain the corresponding first thread contour line analytical equation. In order to make the thread contour line parallel to the coordinate axis of the image, the analytical equation is transformed by angle and translation to obtain the second thread contour line analytical equation. Based on the second thread contour line analytical equation, the spiral interpolation algorithm is used to construct the ideal thread generatrix equation in three-dimensional space. Multiple thread contour line feature points are sampled at equal intervals on the thread comprehensive feature map to obtain the coordinates of these feature points. The feature point coordinates of the thread contour line are back-projected into the thread three-dimensional point cloud model, and the three-dimensional feature point coordinates of each thread contour line feature point coordinate on the ideal thread generatrix equation are calculated. According to the three-dimensional feature point coordinates, the three-dimensional Euclidean distance between two adjacent feature points is calculated to obtain the distance between adjacent feature points of multiple thread contour lines. The distance between adjacent feature points of multiple thread contour lines is compared with the standard thread pitch, and the absolute value of the difference between the distance between each adjacent feature point and the standard thread pitch is calculated to obtain multiple thread pitch deviation values. The deviation value is used to evaluate whether the thread pitch meets the standard, thereby realizing the detection of ball screw thread defects.

[0034] Step 104, constructing a thread valley cone surface equation between two adjacent threads according to the thread comprehensive characteristic diagram, and calculating the thread profile deviation value;

[0035] Specifically, the thread comprehensive feature map is threshold segmented to obtain a binary image of the thread valley area, and the thread valley area is distinguished from other parts. According to the binary image of the thread valley area, two adjacent thread valley areas are marked to obtain paired thread valley areas. Morphological refinement is performed on the paired thread valley areas to obtain the skeleton line image of the thread valley. Morphological refinement can effectively extract the center line of the thread valley area and retain its structural features. The thread valley skeleton line image is processed by Hough transform, the straight line segments of the thread valley skeleton line are detected, and the parameters of these straight line segments are obtained. According to the parameters of the straight line segments of the thread valley skeleton line, the intersection coordinates of the two adjacent straight line segments of the thread valley skeleton line are calculated to obtain the intersection coordinates of the thread valley skeleton line. Based on the intersection coordinates, a three-dimensional mesh model of the thread valley area is constructed, and the thread valley cone surface equation of the thread valley area is fitted by the least squares method. The least squares method is a commonly used mathematical fitting method that can effectively fit the three-dimensional surface of the thread valley area. The feature points of multiple thread valley areas are sampled at equal intervals on the thread comprehensive feature map, and the pixel coordinates of these feature points are extracted to obtain the coordinates of the feature points of multiple thread valley areas. These feature points represent the specific location and morphological features of the thread valley area. The coordinates of the feature points in the thread valley area are back-projected into the three-dimensional point cloud model of the thread, and the distance from the coordinates of each feature point in the thread valley area to the thread valley cone surface equation is calculated to obtain the distance values ​​from multiple feature points in the thread valley area to the cone surface. According to the distance value from the feature point in the thread valley area to the cone surface, the corresponding thread profile deviation value is generated. The thread profile deviation value can reflect the geometric error of the thread profile.

[0036] Step 105: input the thread stereo image into a preset full convolutional network model to perform semantic segmentation of defect distribution, and combine the pitch constraint corresponding to the thread pitch deviation value and the tooth profile constraint corresponding to the thread tooth profile deviation value to obtain a pixel-level defect distribution map;

[0037] Specifically, an end-to-end fully convolutional network model is constructed with a thread stereo image as input and a pixel-level defect distribution map as output. The thread pitch deviation value and thread profile deviation value are used as the prior knowledge parameters of the fully convolutional network model and embedded into the loss function of the model, thereby improving the accuracy and robustness of the model. In the multi-scale feature extraction module of the fully convolutional network model, the thread stereo image is processed through multiple convolutional layers and pooling layers to extract its multi-scale texture features and generate a multi-scale texture feature map. In the attention mechanism module of the fully convolutional network model, the feature weight at each pixel position in the multi-scale texture feature map is controlled by a gating unit to obtain a thread attention feature map. The attention mechanism can effectively enhance important features, suppress irrelevant information, and make the model pay more attention to the key areas in the thread image. In the feature fusion module of the fully convolutional network model, the shallow multi-scale feature map in the thread attention feature map is fused with the deep attention feature map through jump connections to generate a thread fusion feature map. Jump connections can combine feature information at different levels, improve the expression ability of feature maps, and enhance the detection effect of the model on thread defects. According to the thread pitch deviation value, a pitch constraint module is set in the full convolutional network model. Through the convolution layer whose convolution kernel size matches the pitch, the thread pitch features are extracted, and the pitch constraint is embedded in the semantic segmentation result. At the same time, according to the thread tooth profile deviation value, a tooth profile constraint module is set in the full convolutional network model. Through the convolution layer whose convolution kernel shape matches the thread valley cone surface, the thread tooth profile features are extracted, and the tooth profile constraint is embedded in the semantic segmentation result. In this way, the model can accurately capture the detailed features of the thread tooth profile and ensure the accuracy and reliability of the segmentation results. In the multi-scale semantic segmentation module of the full convolutional network model, the thread fusion feature map is semantically segmented at multiple scales through the transposed convolution layer and the upsampling layer, and the pitch constraint and tooth profile constraint are combined to obtain the pixel-level defect distribution map. Multi-scale semantic segmentation can ensure that the details and overall structure of the segmentation results at different scales can be fully reflected, thereby improving the accuracy and integrity of the segmentation results.

[0038] Step 106: back-project the pixel-level defect distribution map onto the thread three-dimensional point cloud model to obtain the defect area point cloud, and calculate the three-dimensional morphological parameters and defect severity level of the defect area point cloud.

[0039] Specifically, according to the internal and external parameter matrices, the defect pixel coordinates in the pixel-level defect distribution map are back-projected into the thread three-dimensional point cloud model to obtain the three-dimensional coordinate set of the defect area. According to the obtained three-dimensional coordinate set of the defect area, the point cloud of the defect area is extracted by the point cloud segmentation algorithm, and the point clouds of the defect area are spatially clustered to form multiple defect point cloud clusters. Point cloud segmentation and clustering can divide complex point cloud data into multiple independent defect areas for further analysis. The three-dimensional morphological parameters of multiple defect point cloud clusters are calculated, and these parameters include the volume, surface area and three-dimensional size of the defect. By calculating these morphological parameters, the spatial characteristics of the defect can be fully described. According to the three-dimensional morphological parameters, the three-dimensional morphological complexity is calculated, and the morphological complexity index is used to quantify the complexity index set of the defect area. The morphological complexity index can reflect the complexity of the defect shape and provide a reference for the evaluation of the defect severity level. According to the three-dimensional coordinate set of the defect area and the thread three-dimensional point cloud model, the distance deviation is calculated, and the distance deviation index is used to quantify the distance deviation index set of the defect area. The distance deviation index can reflect the degree of deviation between the defect area and the normal thread surface. Combining the above steps, a three-dimensional morphological feature vector is constructed according to the defect area complexity index set and the defect area distance deviation index set. The corresponding defect severity level is predicted by inputting the feature vector into the support vector machine regression model. The support vector machine regression model is a machine learning method that can accurately predict the output value based on the input features and has high accuracy and robustness in defect detection. Multiple defect point cloud clusters, three-dimensional morphological parameters and defect severity levels are visualized to generate a three-dimensional visualization result of the thread defect. The spatial distribution and morphological characteristics of the defects are displayed through three-dimensional visualization.

[0040] In the embodiment of the present application, the three-dimensional point cloud model of the ball screw thread is accurately obtained by binocular stereo imaging technology. Stereo matching and triangulation methods are used to ensure the accuracy and integrity of three-dimensional reconstruction. Pyramid decomposition and morphological filtering techniques are used to extract multi-scale thread comprehensive feature maps, enhance local features of the image, provide more detailed information, and improve the sensitivity of defect detection. The ideal thread generatrix equation and thread valley cone surface equation are constructed to calculate the thread pitch deviation value and thread tooth profile deviation value respectively to ensure accurate measurement of thread geometric parameters. The geometric features of the thread profile and thread valley area are extracted to accurately locate defects. Combined with the fully convolutional network model, pixel-level semantic segmentation of defect distribution is achieved to improve the accuracy and automation of detection. The thread pitch deviation value and thread tooth profile deviation value are embedded in the network model as prior knowledge parameters to optimize the semantic segmentation results and improve the robustness and generalization ability of the model. Through the back-projection technology, the pixel-level defect distribution map is mapped back to the three-dimensional point cloud model to obtain the three-dimensional morphological parameters of the defect area. The volume, surface area, three-dimensional size and other parameters of the defect area are calculated, and the defect severity level is predicted through the support vector machine regression model. Multiple defect point cloud clusters and their three-dimensional morphological parameters are visualized to generate intuitive three-dimensional visualization results of thread defects, thereby improving the accuracy of ball screw thread defect detection.

[0041] In a specific embodiment, the process of executing step 101 may specifically include the following steps:

[0042] (1) Use a binocular camera to perform binocular stereo imaging on the ball screw to obtain the thread axial section view and thread generatrix projection view;

[0043] (2) Perform camera calibration on the thread axial profile and thread generatrix projection to obtain the internal and external parameter matrix of the binocular camera;

[0044] (3) Perform polar line correction on the thread axial section and thread generatrix projection according to the internal and external parameter matrices to obtain a corrected stereo image pair;

[0045] (4) Performing stereo matching on the corrected stereo image pair to obtain a disparity map, and performing triangulation based on the disparity map and the internal and external parameter matrix to obtain a first depth map of the thread surface;

[0046] (5) Denoising and smoothing the first depth map to obtain a second depth map, and performing depth and color fusion on the second depth map and the corrected stereo image pair to obtain a thread stereo image;

[0047] (6) Generate a point cloud from the thread stereo image to obtain an initial thread 3D point cloud, and perform statistical outlier filtering and voxel downsampling on the initial thread 3D point cloud to obtain a target thread 3D point cloud;

[0048] (7) Perform normal vector estimation and surface reconstruction on the target thread 3D point cloud to obtain a thread 3D point cloud model.

[0049] Specifically, the axial cross-section and projection of the thread generatrix are obtained. Binocular stereo imaging uses two cameras to shoot the same target from different angles to obtain an image pair containing depth information. The acquired images are calibrated to determine the internal and external parameter matrix of the binocular camera, where the internal parameter matrix includes the focal length , principal point coordinates Etc., are used to describe the internal characteristics of the camera; the external parameter matrix includes the rotation matrix (R) and the translation vector (t), which are used to describe the relationship between the camera and the world coordinate system. According to the internal and external parameter matrix of the binocular camera, the thread axial profile and the thread generatrix projection are polar corrected, and the points in the two image pairs are aligned to the same horizontal line, thereby simplifying the stereo matching process. In the corrected stereo image pair, the pixels on the same horizontal line correspond to the same three-dimensional point, which is convenient for subsequent processing. Stereo matching is performed on the corrected stereo image pair to obtain a disparity map. The disparity map reflects the depth information of the object surface by calculating the displacement of the corresponding points in the image pair. The larger the disparity value, the closer the object is to the camera. According to the disparity map and the internal and external parameter matrix of the camera, the first depth map of the thread surface is calculated by the triangulation formula. The triangulation formula is as follows:

[0050] ;

[0051] in, Indicates the depth value, is the focal length of the camera, is the baseline distance between the binocular cameras, is the disparity value. The first depth map is denoised and smoothed. Denoising can reduce the noise in the depth map, and smoothing can eliminate small fluctuations, making the depth map smoother and more accurate. The processed second depth map is fused with the corrected stereo image pair for depth and color to generate a thread stereo image. Combining depth information with color information, the three-dimensional image of the thread not only has an accurate geometric structure, but also retains rich visual features. The thread stereo image is converted into an initial thread three-dimensional point cloud. A point cloud is a collection of a large number of points in a three-dimensional space. Each point has a clear spatial coordinate and can accurately describe the three-dimensional morphology of an object. However, there may be noise points and outliers in the initial point cloud data. In order to improve the quality of the point cloud data, statistical outlier filtering and voxel downsampling are performed on it. Statistical outlier filtering can remove noise points in the point cloud, and voxel downsampling divides the point cloud into multiple voxels (volume pixels) and replaces all points in each voxel with representative points in the voxel, thereby reducing the amount of point cloud data and improving processing efficiency, and obtaining a target thread three-dimensional point cloud. The target thread three-dimensional point cloud is subjected to normal vector estimation and surface reconstruction. Normal vector estimation is used to calculate the normal vector of each point, that is, the local surface direction of the point. Surface reconstruction generates a complete thread 3D point cloud model by converting point cloud data into a continuous surface model.

[0052] In a specific embodiment, the process of executing step 102 may specifically include the following steps:

[0053] (1) Gaussian filtering is performed on the thread stereo image to obtain a smoothed thread image, and Laplacian pyramid decomposition is performed on the smoothed thread image to obtain multiple initial sub-images of different scales;

[0054] (2) According to the scale of each initial sub-image, morphological structural elements of different sizes are selected to obtain multiple scale-adaptive structural elements;

[0055] (3) Based on the scale-adaptive structural element, each initial sub-image is opened and closed to obtain multiple morphologically filtered sub-images;

[0056] (4) performing top-hat transformation and bottom-hat transformation on multiple morphologically filtered sub-images to obtain multiple enhanced sub-images that highlight local features;

[0057] (5) Performing wavelet transform on the enhanced sub-image to obtain the low-frequency coefficients and high-frequency coefficients of each enhanced sub-image;

[0058] (6) Calculate the local energy and directional consistency of each enhanced sub-image based on the low-frequency coefficients and high-frequency coefficients to obtain the feature weight of each enhanced sub-image;

[0059] (7) Perform weighted fusion on multiple enhanced sub-images according to the feature weights to obtain an initial fused image, and perform non-local mean denoising and adaptive histogram equalization on the initial fused image to obtain a comprehensive feature map of the thread.

[0060] Specifically, the original image is processed using Gaussian filtering. Gaussian filtering is a technique for smoothing an image using a Gaussian function, which can effectively remove noise from the image while retaining the main features of the image. The mathematical expression of Gaussian filtering is:

[0061] ;

[0062] in, represents the value of the Gaussian function, is the standard deviation of the Gaussian function, and is the image coordinate. Through Gaussian filtering, a smoothed spiral image is obtained. The smoothed spiral image is decomposed into multiple scales using the Laplacian pyramid decomposition method. Laplacian pyramid is a commonly used image pyramid decomposition method. By performing multi-level decomposition on the image, the detailed information of the image at different scales is obtained. The decomposition process includes Gaussian pyramid downsampling and Laplacian pyramid reconstruction. The downsampled images represent initial sub-images of different scales. According to the scale of each initial sub-image, morphological structure elements of different sizes are selected to obtain multiple scale-adaptive structure elements. Morphological structure elements are used to define the shape and size of the operation in image processing. Images of different scales need to match structure elements of different scales so as to extract effective features at different scales. Based on the scale-adaptive structure element, each initial sub-image is opened and closed respectively. The opening operation is an operation of first erosion and then expansion, which is mainly used to remove small objects and noise; the closing operation is an operation of first expansion and then erosion, which is used to fill small holes and smooth the edges of objects. After the opening and closing operations, multiple morphologically filtered sub-images are obtained. Top-hat transformation and bottom-hat transformation are performed on multiple morphologically filtered sub-images. The top hat transform can highlight the bright details in the image, while the bottom hat transform can highlight the dark details in the image. Through the transformation, multiple enhanced sub-images that highlight local features are obtained. Wavelet transform is performed on the enhanced sub-image. Wavelet transform is a technique that decomposes a signal into different frequency components, and the low-frequency coefficients and high-frequency coefficients of each enhanced sub-image can be obtained. The low-frequency coefficients represent the overall information of the image, while the high-frequency coefficients contain the detailed information of the image. The formula of wavelet transform is:

[0063] ;

[0064] in, represents the coefficients of wavelet transform, It's a signal. is the wavelet function, and are scale and displacement parameters, respectively, and * represents complex conjugate. Based on the low-frequency coefficients and high-frequency coefficients, the local energy and directional consistency of each enhanced sub-image are calculated to obtain the feature weight of each enhanced sub-image. The local energy can be represented by the sum of squares, while the directional consistency can be measured by the variance of the gradient direction. The feature weight can reflect the importance of the features of the image at different positions. According to the feature weights, multiple enhanced sub-images are weighted fused to obtain the initial fused image. The features of different images are weighted averaged according to the feature weights to generate a fused image with more details and features. The initial fused image is processed by non-local mean denoising and adaptive histogram equalization to obtain a comprehensive feature map of the thread. Non-local mean denoising can remove noise by using similar blocks in the image, while adaptive histogram equalization can enhance the contrast of the image, making the thread features more obvious and clear.

[0065] In a specific embodiment, the process of executing step 103 may specifically include the following steps:

[0066] (1) Perform edge detection on the thread comprehensive feature map, extract the thread contour line, and track the thread contour line to obtain the thread contour line coordinate sequence;

[0067] (2) According to the thread profile coordinate sequence, the corresponding first thread profile analytical equation is fitted by the least square method, and the first thread profile analytical equation is transformed by angle and translation so that the thread profile is parallel to the image coordinate axis, thereby obtaining the second thread profile analytical equation;

[0068] (3) According to the analytical equation of the second thread profile, an ideal thread generatrix equation in three-dimensional space is constructed by a spiral interpolation algorithm, and multiple thread profile feature points are sampled at equal intervals on the thread comprehensive feature graph to obtain the coordinates of multiple thread profile feature points;

[0069] (4) Back-projecting the coordinates of the thread contour feature points onto the thread three-dimensional point cloud model, and calculating the coordinates of each thread contour feature point on the ideal thread generatrix equation;

[0070] (5) According to the coordinates of the three-dimensional feature points of the thread contour line, the three-dimensional Euclidean distance between two adjacent feature points is calculated to obtain the distance between multiple adjacent feature points of the thread contour line;

[0071] (6) Compare the distances between adjacent feature points of multiple thread contour lines with the standard thread pitch, calculate the absolute value of the difference between the distance between each adjacent feature point and the standard thread pitch, and obtain multiple thread pitch deviation values.

[0072] Specifically, the thread comprehensive feature map is processed using an edge detection algorithm, such as the Canny edge detection algorithm. Canny edge detection is a classic image processing technique that extracts edge information from an image by calculating the gradient of the image and performing non-maximum suppression and double threshold detection. The image after edge detection will show the contour of the thread, which reflects the geometric shape of the thread. The extracted thread contour is tracked to obtain a coordinate sequence of the thread contour. Contour tracking can use a method called chain code to record the position of the contour by following the path of edge pixels. The obtained coordinate sequence can be expressed as ,in is the index of the point on the contour line, and It is The coordinates of the points. According to the coordinate sequence of the thread profile, the corresponding first thread profile analytical equation is fitted by the least squares method. The least squares method is a mathematical optimization technique that finds the best fitting curve by minimizing the sum of squares of the errors between the data points and the fitting curve. Assuming that the thread profile can be represented by a quadratic polynomial, the fitting equation can be written as:

[0073] ;

[0074] in, , and is the fitting parameter, which is solved by the least squares method. The goal of the least squares method is to minimize the following sum of squared errors:

[0075] ;

[0076] Perform angle transformation and translation transformation on the analytical equation of the first thread profile so that the thread profile is parallel to the image coordinate axis. Assume that the angle to be rotated is , then the rotation matrix is:

[0077] ;

[0078] The coordinates of the point after rotation transformation It can be expressed as:

[0079] ;

[0080] Through the above transformation, the thread profile analytical equation is transformed into the second thread profile analytical equation, so that the thread profile is parallel to the image coordinate axis. According to the second thread profile analytical equation, the ideal thread generatrix equation in three-dimensional space is constructed by the spiral interpolation algorithm. The spiral interpolation algorithm is a method for generating three-dimensional spiral curves. By interpolating the quadratic polynomial equation, the ideal thread generatrix equation in three-dimensional space is obtained. Multiple thread profile feature points are sampled at equal intervals on the thread comprehensive feature map to obtain the coordinates of the feature points. The coordinates of the thread contour feature points are back-projected to the thread 3D point cloud model, and the 3D feature point coordinates of each thread contour feature point on the ideal thread generatrix equation are calculated. Back projection is the process of converting two-dimensional image coordinates into three-dimensional space coordinates. This process uses the internal and external parameter matrices of the camera for projection transformation. According to the coordinates of the three-dimensional feature points of the thread contour, the three-dimensional Euclidean distance between two adjacent feature points is calculated to obtain the distance between adjacent feature points of multiple thread contours. The three-dimensional Euclidean distance formula is:

[0081] ;

[0082] in, Indicates and The distance between feature points, and are the three-dimensional coordinates of the two feature points respectively. Compare the distance between adjacent feature points of multiple thread contours with the standard thread pitch, calculate the absolute value of the difference between the distance between each adjacent feature point and the standard thread pitch, and obtain multiple thread pitch deviation values. The calculation formula for the pitch deviation value is:

[0083] ;

[0084] in, Indicates The deviation value of the distance between feature points, is the standard pitch. By calculating these deviations, the accuracy of the thread can be evaluated and any defects can be identified.

[0085] In a specific embodiment, the process of executing step 104 may specifically include the following steps:

[0086] (1) Threshold segmentation is performed on the thread comprehensive feature map to obtain a binary image of the thread valley region. Based on the binary image of the thread valley region, two adjacent thread valley regions are marked to obtain a pair of thread valley regions.

[0087] (2) Perform morphological refinement on paired thread valley regions to obtain thread valley skeleton line images, and detect the straight line segments of the thread valley skeleton lines through Hough transform based on the thread valley skeleton line images to obtain the straight line segment parameters of the thread valley skeleton lines;

[0088] (3) According to the straight line segment parameters of the thread valley skeleton line, the coordinates of the intersection of two adjacent straight line segments of the thread valley skeleton line are calculated to obtain the coordinates of the intersection of the thread valley skeleton line;

[0089] (4) Construct a three-dimensional mesh model of the thread valley area based on the coordinates of the intersection points of the thread valley skeleton lines, and fit the thread valley cone surface equation of the thread valley area by the least squares method based on the three-dimensional mesh model of the thread valley area;

[0090] (5) sampling multiple thread valley region feature points at equal intervals on the thread comprehensive feature map, extracting the pixel coordinates of the feature points, and obtaining the coordinates of multiple thread valley region feature points;

[0091] (6) Back-projecting the coordinates of multiple thread valley region feature points onto the thread three-dimensional point cloud model, calculating the distance between the coordinates of the thread valley region feature points and the thread valley cone surface equation, and obtaining the distance values ​​from the multiple thread valley region feature points to the cone surface;

[0092] (7) Generate corresponding thread profile deviation values ​​based on the distance values ​​from multiple thread valley area feature points to the cone surface.

[0093] Specifically, by setting a threshold, the image is converted into a binary image. Threshold segmentation is a basic image processing technique that segments an image into foreground and background regions by comparing pixel values ​​with a predetermined threshold. For the thread valley region, select an appropriate threshold The binary image generated to distinguish the thread valley from other parts can be expressed as:

[0094] ;

[0095] in, is the pixel value of the binary image, is the pixel value of the original image. After obtaining the binary image of the thread valley area, the two adjacent thread valley areas are marked. The connected pixels in the binary image are divided into different regions by the connected component labeling algorithm, so that each thread valley area is marked. In the marked image, each thread valley area has a unique identifier. Morphological thinning is performed on the paired thread valley areas. Morphological thinning is an image processing operation that removes redundant pixels in the area and retains the skeleton structure of the area. The thinned image can more clearly show the skeleton line of the thread valley. The thread valley area is processed using the thinning algorithm in the morphological operation, such as the Hit-or-Miss transform, to obtain the thread valley skeleton line image. After obtaining the thread valley skeleton line image, the Hough transform is used to detect the straight line segment of the thread valley skeleton line. The Hough transform is a commonly used image processing technique for detecting straight lines from an image. Through the Hough transform, the points in the image are converted into the parameter space, and the parameters of the straight line segment are determined by finding the local maximum in the parameter space. Assume that the equation of the detected straight line is: .in, is the slope, is the intercept. These parameters describe the straight line segment of the thread valley skeleton line. According to the straight line segment parameters of the thread valley skeleton line, calculate the coordinates of the intersection of two adjacent straight line segments of the thread valley skeleton line. The coordinates of the intersection can be obtained by solving the system of equations of the two lines. According to the coordinates of the intersection, construct a three-dimensional mesh model of the thread valley area. The three-dimensional mesh model is a mathematical model used to describe the geometric shape of the surface of an object. By connecting adjacent intersections, a three-dimensional mesh structure can be formed. The thread valley cone surface equation of the thread valley area is fitted by the least squares method. The least squares method is a mathematical optimization method used to minimize the sum of squared errors between data points and the model to find the best fitting surface. Assume that the thread valley cone surface equation can be expressed as:

[0096] ;

[0097] in, , , , , and is the fitting parameter. Multiple thread valley area feature points are sampled at equal intervals on the thread comprehensive feature map, and the pixel coordinates of these feature points are extracted to obtain the coordinates of multiple thread valley area feature points. . Back-project the coordinates of the feature points in the thread valley area to the thread three-dimensional point cloud model. Back-projection is a method of converting two-dimensional image coordinates into three-dimensional space coordinates, using the internal and external parameter matrices of the camera for projection transformation. Calculate the distance from the coordinates of each feature point in the thread valley area to the thread valley cone surface equation, and obtain the distance values ​​from multiple feature points in the thread valley area to the cone surface. The distance formula is:

[0098] ;

[0099] in, It is The distance value from the feature point to the cone surface, is the three-dimensional coordinate of the feature point. The corresponding thread profile deviation value is generated based on these distance values. The thread profile deviation value can be used to describe the degree of deviation of the thread geometry, thereby evaluating the quality of the thread.

[0100] In a specific embodiment, the process of executing step 105 may specifically include the following steps:

[0101] (1) An end-to-end fully convolutional network model is constructed, which takes a thread stereo image as input and outputs a pixel-level defect distribution map. The thread pitch deviation value and thread profile deviation value are used as prior knowledge parameters of the fully convolutional network model and embedded into the loss function of the fully convolutional network model.

[0102] (2) In the multi-scale feature extraction module of the fully convolutional network model, the multi-scale texture features of the thread stereo image are extracted through multiple convolutional layers and pooling layers to obtain a multi-scale texture feature map;

[0103] (3) In the attention mechanism module of the fully convolutional network model, the feature weight at each pixel position in the multi-scale texture feature map is controlled by the gating unit to obtain the thread attention feature map;

[0104] (4) In the feature fusion module of the fully convolutional network model, the shallow multi-scale feature map in the thread attention feature map is fused with the deep attention feature map through skip connections to obtain the thread fusion feature map;

[0105] (5) According to the thread pitch deviation value, a pitch constraint module is set in the full convolutional network model. The thread pitch features are extracted through a convolutional layer whose convolution kernel size matches the pitch, and the pitch constraint is embedded in the semantic segmentation result.

[0106] (6) According to the thread profile deviation value, a thread profile constraint module is set in the full convolutional network model. The thread profile features are extracted through the convolution layer whose convolution kernel shape matches the thread valley cone surface, and the thread profile constraint is embedded in the semantic segmentation result.

[0107] (7) In the multi-scale semantic segmentation module of the fully convolutional network model, multi-scale semantic segmentation of the thread fusion feature map is performed through the transposed convolution layer and the upsampling layer. At the same time, the pitch constraint and the tooth shape constraint are combined to obtain the pixel-level defect distribution map.

[0108] Specifically, a suitable fully convolutional neural network is constructed. The input of the model is a thread stereo image, and the output is a pixel-level defect distribution map. In order to improve the accuracy of the model, the thread pitch deviation value and the thread profile deviation value are embedded into the loss function of the fully convolutional network model as prior knowledge parameters. In the multi-scale feature extraction module of the fully convolutional network model, the multi-scale texture features of the thread stereo image are extracted through multiple convolutional layers and pooling layers to obtain a multi-scale texture feature map. The convolution layer is used to extract local features in the image, and the pooling layer is used to reduce the dimension of the feature map while retaining important information. Assume that the input image is , the feature map after the convolution layer and the pooling layer is , it can be expressed as:

[0109] ;

[0110] in, is the weight of the convolution kernel, It is a convolution and pooling operation. The attention mechanism module controls the feature weight at each pixel position in the multi-scale texture feature map through the gate unit to obtain the thread attention feature map. The attention mechanism enhances important features and suppresses unimportant features by giving different weights to feature maps at different positions. Assuming the feature map The feature map after the attention mechanism is , then:

[0111] ;

[0112] in, is the attention weight, which represents the importance of each pixel in the feature map. In the feature fusion module, the shallow multi-scale feature map in the thread attention feature map is fused with the deep attention feature map through jump connection to obtain the thread fusion feature map. Jump connection can combine shallow features with deep features, so that the model can use multi-level information in semantic segmentation tasks. Assume that the fused feature map is , then:

[0113] ;

[0114] According to the thread pitch deviation value, a pitch constraint module is set in the full convolutional network model. The pitch constraint module extracts the thread pitch feature through a convolution layer whose convolution kernel size matches the pitch, and embeds the pitch constraint into the semantic segmentation result. This ensures that the model takes into account the pitch change when segmenting the thread, making the segmentation result more accurate. Assume that the feature map after the pitch constraint is , then:

[0115] ;

[0116] in, is the weight of the convolution kernel that matches the pitch, is a convolution operation. Similarly, according to the thread profile deviation value, a thread profile constraint module is set in the full convolutional network model. The thread profile constraint module extracts thread profile features through a convolution layer whose convolution kernel shape matches the thread valley cone surface, and embeds the thread profile constraint into the semantic segmentation result. This ensures that the model takes into account the changes in thread profile when segmenting the thread, making the segmentation result more accurate. Assume that the feature map after the thread profile constraint is , then:

[0117] ;

[0118] in, is the weight of the convolution kernel that matches the spiral valley cone, is a convolution operation. In the multi-scale semantic segmentation module, the thread fusion feature map is segmented at multiple scales through the transposed convolution layer and the upsampling layer. At the same time, the pitch constraint and the tooth shape constraint are combined to finally obtain the pixel-level defect distribution map. The transposed convolution layer is used for upsampling to restore the feature map to the size of the original image. After combining the pitch constraint and the tooth shape constraint, the segmentation result is more accurate. Assume that the final pixel-level defect distribution map is , then:

[0119] ;

[0120] in, is the transposed convolution and upsampling operation.

[0121] In a specific embodiment, the process of executing step 106 may specifically include the following steps:

[0122] (1) According to the internal and external parameter matrix, the defect pixel coordinates in the pixel-level defect distribution map are back-projected onto the thread 3D point cloud model to obtain the 3D coordinate set of the defect area;

[0123] (2) According to the three-dimensional coordinate set of the defect area, the point cloud of the defect area is extracted by the point cloud segmentation algorithm, and the point cloud of the defect area is spatially clustered to obtain multiple defect point cloud clusters;

[0124] (3) Calculate the three-dimensional morphological parameters of multiple defect point cloud clusters, including defect volume, surface area and three-dimensional size;

[0125] (4) According to the three-dimensional morphological parameters, the three-dimensional morphological complexity is calculated and the morphological complexity index is used to quantify the defect area complexity index set; according to the three-dimensional coordinate set of the defect area and the three-dimensional point cloud model of the thread, the distance deviation is calculated and the distance deviation index is used to quantify the defect area distance deviation index set;

[0126] (5) Construct a three-dimensional morphological feature vector based on the defect area complexity index set and the defect area distance deviation index set, and predict the corresponding defect severity level through the support vector machine regression model based on the three-dimensional morphological feature vector;

[0127] (6) Visualize multiple defect point cloud clusters, three-dimensional morphological parameters, and defect severity levels to generate three-dimensional visualization results of thread defects.

[0128] Specifically, obtain the intrinsic and extrinsic parameter matrices. These parameter matrices include the camera's intrinsic parameter matrix (intrinsic properties, such as focal length, optical axis, etc.) and extrinsic parameter matrix (extrinsic properties, such as rotation and translation). Using these parameters, convert the two-dimensional image coordinates into three-dimensional world coordinates. Assume that the intrinsic parameter matrix is , the external parameter matrix is ,in is the rotation matrix, is the translation vector, and the defect pixel coordinates in the pixel-level defect distribution map are , then the back-projection process can be expressed as:

[0129] ;

[0130] in, is a three-dimensional world coordinate. Through the above formula, the defect pixel coordinates are back-projected onto the threaded three-dimensional point cloud model to obtain the three-dimensional coordinate set of the defect area. According to the obtained three-dimensional coordinate set of the defect area, the point cloud segmentation algorithm is used to extract the point cloud of the defect area. The point cloud segmentation algorithm can use, for example, Euclidean clustering segmentation, which sets a distance threshold and aggregates points with a distance less than the threshold into a cluster. The entire point cloud is divided into multiple small areas, and the defect area point cloud is extracted. The defect area point cloud is spatially clustered to obtain multiple defect point cloud clusters. Spatial clustering can use, for example, the DBSCAN algorithm, which clusters points with higher density into a cluster by setting the minimum number of points and distance parameters. Assume that the defect point cloud cluster obtained is ,in is the index of the cluster. Calculate the three-dimensional morphological parameters of multiple defect point cloud clusters. The three-dimensional morphological parameters include defect volume, surface area and three-dimensional size. The defect volume can be obtained by calculating the convex hull volume of the point cloud. Assume that the convex hull is , the volume is The surface area can be obtained by calculating the surface area of ​​the convex hull, assuming that the surface area is The three-dimensional size can be obtained by calculating the size of the bounding box of the point cloud, assuming the size is , respectively represent length, width and height. According to the three-dimensional morphological parameters, the three-dimensional morphological complexity is calculated and quantified using the morphological complexity index to obtain the defect area complexity index set. The three-dimensional morphological complexity can be obtained by calculating the ratio of the defect volume to the surface area. Assuming that the morphological complexity is

[0131] ;

[0132] According to the three-dimensional coordinate set of the defect area and the three-dimensional point cloud model of the thread, the distance deviation is calculated and quantified using the distance deviation index to obtain the distance deviation index set of the defect area. The distance deviation can be obtained by calculating the distance between the three-dimensional coordinates of the defect area and the ideal position of the thread model. Assume that the distance deviation is ,in Indicates Points:

[0133] ;

[0134] According to the defect area complexity index set and the defect area distance deviation index set, a three-dimensional morphological feature vector is constructed. The feature vector can contain the morphological complexity and distance deviation The corresponding defect severity level is predicted by the support vector machine regression model. Support vector machine regression is a supervised learning method that predicts new defect data by learning the relationship between feature vectors and defect severity levels. Assume that the three-dimensional morphological feature vector is , the predicted defect severity level is

[0135] ;

[0136] Visualize multiple defect point cloud clusters, three-dimensional morphological parameters, and defect severity levels to generate a three-dimensional visualization result of thread defects. Visualization can be done by drawing a three-dimensional point cloud diagram and using color or size to indicate the severity of the defect. For example, suppose there is a thread stereo image and a corresponding internal and external parameter matrix. The defect pixel coordinates in the pixel-level defect distribution map are converted into three-dimensional coordinates by back-projection to obtain a three-dimensional coordinate set of the defect area. The defect area point cloud is extracted using the Euclidean clustering segmentation algorithm, and the DBSCAN algorithm is used for spatial clustering to obtain multiple defect point cloud clusters. Calculate the three-dimensional morphological parameters of each defect point cloud cluster. Assume that the convex hull volume of a defect point cloud cluster is 10 , with a surface area of ​​20 , the bounding box size is (2mm, 3mm, 4mm). Based on these parameters, calculate the morphological complexity:

[0137] ;

[0138] Calculate the distance deviation between the defect area and the ideal position of the thread model. Assume that the mean distance deviation is 1 mm and the variance is Based on these indicators, a three-dimensional morphological feature vector is constructed Through the support vector machine regression model, the defect severity level is predicted to be These results can be visualized to generate a 3D visualization of the thread defects. For example, red is used to represent severe defects, yellow is used to represent medium defects, and green is used to represent minor defects.

[0139] The above describes the ball screw thread defect detection method integrating computer vision in the embodiment of the present application. The following describes the ball screw thread defect detection system integrating computer vision in the embodiment of the present application. Figure 2 In the embodiment of the present application, an embodiment of a ball screw thread defect detection system integrating computer vision includes:

[0140] An imaging module 201 is used to perform binocular stereo imaging on the ball screw to be tested, obtain a thread stereo image, and construct a thread three-dimensional point cloud model based on the thread stereo image;

[0141] The decomposition module 202 is used to perform pyramid decomposition on the thread stereo image to obtain multiple initial sub-images, and perform morphological filtering and fusion on the multiple initial sub-images to obtain a thread comprehensive feature map;

[0142] The first calculation module 203 is used to construct an ideal thread generatrix equation of the thread profile according to the thread comprehensive characteristic diagram, and calculate the corresponding thread pitch deviation value;

[0143] The second calculation module 204 is used to construct a thread valley cone surface equation between two adjacent threads according to the thread comprehensive characteristic diagram, and calculate the thread profile deviation value;

[0144] The semantic segmentation module 205 is used to input the thread stereo image into a preset full convolutional network model to perform semantic segmentation of defect distribution, and combine the pitch constraint corresponding to the thread pitch deviation value and the tooth shape constraint corresponding to the thread tooth shape deviation value to obtain a pixel-level defect distribution map;

[0145] The projection module 206 is used to back-project the pixel-level defect distribution map onto the thread three-dimensional point cloud model to obtain the defect area point cloud, and calculate the three-dimensional morphological parameters and defect severity level of the defect area point cloud.

[0146] Through the cooperation of the above components, the three-dimensional point cloud model of the ball screw thread is accurately obtained through binocular stereo imaging technology. Stereo matching and triangulation methods are used to ensure the accuracy and integrity of three-dimensional reconstruction. Pyramid decomposition and morphological filtering techniques are used to extract multi-scale thread comprehensive feature maps, enhance the local features of the image, provide more detailed information, and improve the sensitivity of defect detection. The ideal thread generatrix equation and thread valley cone surface equation are constructed to calculate the thread pitch deviation value and thread tooth profile deviation value respectively to ensure the accurate measurement of thread geometric parameters. The geometric features of the thread profile and thread valley area are extracted to accurately locate defects. Combined with the fully convolutional network model, pixel-level semantic segmentation of defect distribution is realized to improve the accuracy and automation of detection. The thread pitch deviation value and thread tooth profile deviation value are embedded in the network model as prior knowledge parameters to optimize the semantic segmentation results and improve the robustness and generalization ability of the model. Through the back-projection technology, the pixel-level defect distribution map is mapped back to the three-dimensional point cloud model to obtain the three-dimensional morphological parameters of the defect area. The volume, surface area, three-dimensional size and other parameters of the defect area are calculated, and the defect severity level is predicted through the support vector machine regression model. Multiple defect point cloud clusters and their three-dimensional morphological parameters are visualized to generate intuitive three-dimensional visualization results of thread defects, thereby improving the accuracy of ball screw thread defect detection.

[0147] The present application also provides a computer device, which includes a memory and a processor, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the processor executes the steps of the ball screw thread defect detection method integrated with computer vision in the above-mentioned embodiments.

[0148] The present application also provides a computer-readable storage medium, which may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium, wherein instructions are stored in the computer-readable storage medium, and when the instructions are executed on a computer, the computer executes the steps of the ball screw thread defect detection method with integrated computer vision.

[0149] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described systems, systems and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0150] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store program codes.

[0151] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A ball screw thread defect detection method integrated with computer vision, characterized in that: The method comprises: Perform binocular stereo imaging on the ball screw to be tested to obtain a thread stereo image, and construct a thread three-dimensional point cloud model based on the thread stereo image; Performing pyramid decomposition on the thread stereo image to obtain a plurality of initial sub-images, and performing morphological filtering and fusion on the plurality of initial sub-images to obtain a thread comprehensive feature map; Constructing an ideal thread generatrix equation of the thread profile according to the thread comprehensive characteristic diagram, and calculating the corresponding thread pitch deviation value; Constructing a thread valley cone surface equation between two adjacent threads according to the thread comprehensive characteristic diagram, and calculating a thread profile deviation value; The thread stereo image is input into a preset full convolutional network model for defect distribution semantic segmentation, and the pitch constraint corresponding to the thread pitch deviation value and the tooth profile constraint corresponding to the thread tooth profile deviation value are combined to obtain a pixel-level defect distribution map; specifically comprising: constructing an end-to-end full convolutional network model with the thread stereo image as input and the pixel-level defect distribution map as output, and using the thread pitch deviation value and the thread tooth profile deviation value as prior knowledge parameters of the full convolutional network model, and embedding them into the loss function of the full convolutional network model; in the multi-scale feature extraction module of the full convolutional network model, extracting the multi-scale texture features of the thread stereo image through multiple convolutional layers and pooling layers to obtain a multi-scale texture feature map; in the attention mechanism module of the full convolutional network model, controlling the feature weight at each pixel position in the multi-scale texture feature map through a gating unit to obtain a thread attention map. Attention feature map; in the feature fusion module of the full convolutional network model, the shallow multi-scale feature map in the thread attention feature map is fused with the deep attention feature map through jump connection to obtain a thread fusion feature map; according to the thread pitch deviation value, a pitch constraint module is set in the full convolutional network model, and the thread pitch feature is extracted through a convolution layer whose convolution kernel size matches the pitch, and the pitch constraint is embedded in the semantic segmentation result; according to the thread tooth shape deviation value, a tooth shape constraint module is set in the full convolutional network model, and the thread tooth shape feature is extracted through a convolution layer whose convolution kernel shape matches the thread valley cone surface, and the tooth shape constraint is embedded in the semantic segmentation result; in the multi-scale semantic segmentation module of the full convolutional network model, the thread fusion feature map is subjected to multi-scale semantic segmentation through a transposed convolution layer and an upsampling layer, and the pitch constraint and the tooth shape constraint are combined to obtain a pixel-level defect distribution map; The pixel-level defect distribution map is back-projected onto the thread three-dimensional point cloud model to obtain a defect area point cloud, and the three-dimensional morphological parameters and defect severity level of the defect area point cloud are calculated.

2. The ball screw thread defect detection method integrated with computer vision according to claim 1 is characterized in that: The method performs binocular stereo imaging on the ball screw to be tested to obtain a thread stereo image, and constructs a thread three-dimensional point cloud model according to the thread stereo image, including: The binocular camera is used to perform binocular stereo imaging on the ball screw to be tested, and the thread axial section view and thread generatrix projection view are obtained; Performing camera calibration on the thread axial section view and the thread generatrix projection view to obtain an internal and external parameter matrix of the binocular camera; Performing polar line correction on the thread axial section image and the thread generatrix projection image according to the internal and external parameter matrix to obtain a corrected stereo image pair; Performing stereo matching on the corrected stereo image pair to obtain a disparity map, and performing triangulation based on the disparity map and the internal and external parameter matrix to obtain a first depth map of the thread surface; Denoising and smoothing the first depth map to obtain a second depth map, and performing depth and color fusion on the second depth map and the corrected stereo image pair to obtain a thread stereo image; Generating a point cloud from the thread stereo image to obtain an initial thread three-dimensional point cloud, and performing statistical outlier filtering and voxel downsampling on the initial thread three-dimensional point cloud to obtain a target thread three-dimensional point cloud; Normal vector estimation and surface reconstruction are performed on the target thread three-dimensional point cloud to obtain a thread three-dimensional point cloud model.

3. The ball screw thread defect detection method integrated with computer vision according to claim 2 is characterized in that: The step of performing pyramid decomposition on the thread stereo image to obtain a plurality of initial sub-images, and performing morphological filtering and fusion on the plurality of initial sub-images to obtain a thread comprehensive feature map comprises: Performing Gaussian filtering on the thread stereo image to obtain a smoothed thread image, and performing Laplace pyramid decomposition on the smoothed thread image to obtain a plurality of initial sub-images of different scales; According to the scale of each initial sub-image, morphological structural elements of different sizes are selected to obtain multiple scale-adaptive structural elements; Based on the scale-adaptive structuring element, each initial sub-image is respectively opened and closed to obtain a plurality of morphologically filtered sub-images; Performing top-hat transformation and bottom-hat transformation on the multiple morphologically filtered sub-images to obtain multiple enhanced sub-images that highlight local features; Performing wavelet transform on the enhanced sub-image to obtain low-frequency coefficients and high-frequency coefficients of each enhanced sub-image; Calculating the local energy and directional consistency of each enhanced sub-image according to the low-frequency coefficient and the high-frequency coefficient to obtain a feature weight of each enhanced sub-image; The multiple enhanced sub-images are weightedly fused according to the feature weights to obtain an initial fused image, and the initial fused image is subjected to non-local mean denoising and adaptive histogram equalization processing to obtain a thread comprehensive feature map.

4. The ball screw thread defect detection method integrated with computer vision according to claim 3 is characterized in that: The ideal thread generatrix equation of the thread profile is constructed according to the thread comprehensive characteristic diagram, and the corresponding thread pitch deviation value is calculated, including: Performing edge detection on the thread comprehensive feature graph, extracting thread contour lines, and tracking the thread contour lines to obtain a thread contour line coordinate sequence of the thread contour lines; According to the thread profile coordinate sequence, a corresponding first thread profile analytical equation is fitted by the least square method, and the first thread profile analytical equation is transformed in angle and translation so that the thread profile is parallel to the image coordinate axis, thereby obtaining a second thread profile analytical equation; According to the second thread profile analytical equation, an ideal thread generatrix equation in three-dimensional space is constructed by a spiral interpolation algorithm, and multiple thread profile feature points are sampled at equal intervals on the thread comprehensive feature graph to obtain multiple thread profile feature point coordinates; Back-projecting the thread contour feature point coordinates onto the thread three-dimensional point cloud model, and calculating the thread contour three-dimensional feature point coordinates of each thread contour feature point on the ideal thread generatrix equation; Calculating the three-dimensional Euclidean distance between two adjacent feature points according to the coordinates of the three-dimensional feature points of the thread contour line to obtain the distance between the adjacent feature points of the plurality of thread contour lines; The distances between adjacent characteristic points of multiple thread contour lines are compared with the standard thread pitch, and the absolute value of the difference between the distance between each adjacent characteristic point and the standard thread pitch is calculated to obtain multiple thread pitch deviation values.

5. The ball screw thread defect detection method integrated with computer vision according to claim 4 is characterized in that: The method of constructing a thread valley cone surface equation between two adjacent threads according to the thread comprehensive feature diagram and calculating the thread profile deviation value includes: Performing threshold segmentation on the thread comprehensive feature map to obtain a binary image of the thread valley region, and marking two adjacent thread valley regions based on the binary image of the thread valley region to obtain a pair of thread valley regions; Performing morphological refinement on the paired thread valley regions to obtain a thread valley skeleton line image, and detecting straight line segments of the thread valley skeleton line through Hough transform based on the thread valley skeleton line image to obtain thread valley skeleton line straight line segment parameters; According to the thread valley skeleton line straight line segment parameters, the coordinates of the intersection of two adjacent thread valley skeleton line straight line segments are calculated to obtain the coordinates of the thread valley skeleton line intersection; According to the coordinates of the intersection points of the thread valley skeleton lines, a three-dimensional mesh model of the thread valley area is constructed, and according to the three-dimensional mesh model of the thread valley area, a thread valley cone surface equation of the thread valley area is fitted by the least squares method; Sampling multiple thread valley region feature points at equal intervals on the thread comprehensive feature map, extracting pixel coordinates of the feature points, and obtaining coordinates of multiple thread valley region feature points; Back-projecting the coordinates of the plurality of thread valley region feature points onto the thread three-dimensional point cloud model, calculating the distances from the coordinates of the thread valley region feature points to the thread valley cone surface equation, and obtaining the distance values ​​from the plurality of thread valley region feature points to the cone surface; The corresponding thread profile deviation values ​​are generated according to the distance values ​​from the characteristic points of the thread valley area to the cone surface.

6. The ball screw thread defect detection method integrated with computer vision according to claim 1, characterized in that: The back-projecting of the pixel-level defect distribution map onto the thread three-dimensional point cloud model to obtain a defect area point cloud, and calculating the three-dimensional morphological parameters and defect severity level of the defect area point cloud includes: According to the internal and external parameter matrix, the defect pixel coordinates in the pixel-level defect distribution map are back-projected onto the thread three-dimensional point cloud model to obtain a three-dimensional coordinate set of the defect area; According to the three-dimensional coordinate set of the defect area, a point cloud of the defect area is extracted by a point cloud segmentation algorithm, and the point cloud of the defect area is spatially clustered to obtain a plurality of defect point cloud clusters; Calculating three-dimensional morphological parameters of the plurality of defect point cloud clusters, the three-dimensional morphological parameters including defect volume, surface area and three-dimensional size; According to the three-dimensional morphological parameters, the three-dimensional morphological complexity is calculated and the morphological complexity index is used to quantify the defect area complexity index set; according to the three-dimensional coordinate set of the defect area and the three-dimensional point cloud model of the thread, the distance deviation is calculated and the distance deviation index is used to quantify the defect area distance deviation index set; Constructing a three-dimensional morphological feature vector according to the defect area complexity index set and the defect area distance deviation index set, and predicting a corresponding defect severity level through a support vector machine regression model according to the three-dimensional morphological feature vector; The multiple defect point cloud clusters, the three-dimensional morphological parameters and the defect severity levels are visualized to generate a three-dimensional visualization result of the thread defect.

7. A ball screw thread defect detection system integrated with computer vision, used to execute the ball screw thread defect detection method integrated with computer vision as claimed in claim 1, characterized in that: The system comprises: An imaging module is used to perform binocular stereo imaging on the ball screw to be tested, obtain a thread stereo image, and construct a thread three-dimensional point cloud model based on the thread stereo image; A decomposition module, used for performing pyramid decomposition on the thread stereo image to obtain a plurality of initial sub-images, and performing morphological filtering and fusion on the plurality of initial sub-images to obtain a thread comprehensive feature map; A first calculation module is used to construct an ideal thread generatrix equation of the thread profile according to the thread comprehensive characteristic diagram, and calculate the corresponding thread pitch deviation value; A second calculation module is used to construct a thread valley cone surface equation between two adjacent threads according to the thread comprehensive characteristic diagram, and calculate the thread profile deviation value; A semantic segmentation module is used to input the thread stereo image into a preset full convolutional network model for defect distribution semantic segmentation, and obtain a pixel-level defect distribution map by combining the pitch constraint corresponding to the thread pitch deviation value and the tooth profile constraint corresponding to the thread tooth profile deviation value; specifically comprising: constructing an end-to-end full convolutional network model with the thread stereo image as input and the pixel-level defect distribution map as output, and using the thread pitch deviation value and the thread tooth profile deviation value as prior knowledge parameters of the full convolutional network model, and embedding them into the loss function of the full convolutional network model; in the multi-scale feature extraction module of the full convolutional network model, extracting multi-scale texture features of the thread stereo image through multiple convolutional layers and pooling layers to obtain a multi-scale texture feature map; in the attention mechanism module of the full convolutional network model, controlling the feature weight at each pixel position in the multi-scale texture feature map through a gating unit, and obtaining to the thread attention feature map; in the feature fusion module of the full convolutional network model, the shallow multi-scale feature map in the thread attention feature map is fused with the deep attention feature map through jump connections to obtain a thread fusion feature map; according to the thread pitch deviation value, a pitch constraint module is set in the full convolutional network model, and the thread pitch feature is extracted through a convolution layer whose convolution kernel size matches the pitch, and the pitch constraint is embedded in the semantic segmentation result; according to the thread tooth profile deviation value, a tooth profile constraint module is set in the full convolutional network model, and the thread tooth profile feature is extracted through a convolution layer whose convolution kernel shape matches the thread valley cone surface, and the tooth profile constraint is embedded in the semantic segmentation result; in the multi-scale semantic segmentation module of the full convolutional network model, the thread fusion feature map is subjected to multi-scale semantic segmentation through a transposed convolution layer and an upsampling layer, and the pitch constraint and the tooth profile constraint are combined to obtain a pixel-level defect distribution map; The projection module is used to back-project the pixel-level defect distribution map onto the thread three-dimensional point cloud model to obtain the defect area point cloud, and calculate the three-dimensional morphological parameters and defect severity level of the defect area point cloud.

8. A computer device, characterized in that: The computer device comprises: a memory and at least one processor, wherein instructions are stored in the memory; The at least one processor calls the instructions in the memory so that the computer device executes the ball screw thread defect detection method integrated with computer vision as described in any one of claims 1-6.

9. A computer-readable storage medium having instructions stored thereon, characterized in that: When the instructions are executed by the processor, the ball screw thread defect detection method integrated with computer vision as described in any one of claims 1-6 is implemented.

Images