Medicine bottle detection method and system based on curved surface fitting

By combining multimodal data acquisition and processing of industrial cameras and line laser scanners, adaptive surface fitting and U-Net networks are used to detect bottle defects, which solves the problems of low efficiency and insufficient accuracy of traditional detection methods, and achieves efficient and accurate bottle detection.

CN120411102AActive Publication Date: 2025-08-01SICHUAN XINSHENG PACKAGING TECH CO LTD

Patent Information

Application Number
CN202510912243.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2025-07-03
Publication Date
2025-08-01
Estimated Expiration
2045-07-03

AI Technical Summary

Technical Problem

Traditional bottle detection methods are inefficient and the detection results are greatly affected by human factors, making it difficult to accurately detect surface defects of bottles with complex curved structures. The existing single sensor technology has insufficient detection accuracy and cannot adapt to the diversified production process of bottles.

Method used

The combination of industrial cameras and line laser scanners is adopted to achieve efficient and accurate detection of the bottle through multimodal data acquisition, preprocessing, feature extraction and fusion, adaptive surface fitting and intelligent defect detection.

Benefits of technology

It realizes comprehensive inspection of the surface of the medicine bottle, can adapt to different types of medicine bottles, improves detection accuracy and efficiency, reduces the false detection rate, and meets the needs of large-scale production.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN120411102A_ABST
    Figure CN120411102A_ABST
Patent Text Reader

Abstract

The invention provides a medicine bottle detection method and system based on curved surface fitting, and belongs to the technical field of medicine bottle detection and the technical field of curved surface fitting. S2, preprocessing the multi-modal data; s3, feature extraction and fusion; s4, performing three-dimensional reconstruction and optimization; s5, fitting a self-adaptive curved surface; and S6, defect intelligent detection and classification. The device can efficiently and accurately detect different medicine bottle types.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical fields of medicine bottle detection and surface fitting, and particularly relates to a medicine bottle detection method and system based on surface fitting. Background Art

[0002] As a key carrier for drug storage and transportation, the quality of medicine bottles is directly related to the safety and effectiveness of drugs. Therefore, accurate detection of surface defects on medicine bottles is an important link in ensuring drug quality. Traditional medicine bottle detection methods mainly rely on manual visual inspection. This method is not only inefficient and difficult to meet the needs of large-scale production, but also the detection results are greatly affected by the subjective factors of the inspectors, and it is easy to miss and misdetect. With the development of machine vision technology, detection methods based on two-dimensional images have gradually been applied. However, this method can only obtain the planar information on the surface of the medicine bottle, and for medicine bottles with complex curved surface structures, internal defects and three-dimensional shape defects cannot be accurately detected. In addition, some detection technologies using a single sensor, such as only using a laser scanner or an industrial camera, have problems such as incomplete information acquisition and insufficient detection accuracy, and it is difficult to adapt to the diverse production processes and complex surface features of medicine bottles. Therefore, there is an urgent need for an efficient, accurate and adaptable detection method for different types of medicine bottles to meet the growing quality control requirements of the pharmaceutical industry. Summary of the Invention

[0003] The present invention provides a medicine bottle detection method and system based on surface fitting, which can detect efficiently, accurately and adapt to different types of medicine bottles.

[0004] One aspect of this specification discloses a medicine bottle detection method based on surface fitting, including:

[0005] S1. Multi-modal data acquisition:

[0006] Using an industrial camera and a line laser scanner, data is synchronously acquired with the central axis of the medicine bottle as the rotation axis. Among them, the industrial camera obtains a set of two-dimensional images at multiple angles, and the line laser scanner generates original point cloud data;

[0007] S2. Multi-modal data preprocessing:

[0008] Based on the set of two-dimensional images at multiple angles, for each image, graying, bilateral filtering for denoising, and Canny edge detection are sequentially performed to obtain an edge image:

[0009] Statistical filtering and outlier removal are performed on the original point cloud data to obtain denoised point cloud;

[0010] S3. Feature extraction and fusion:

[0011] Using the SIFT algorithm to extract feature points from each edge image to obtain a set of image features;

[0012] Calculate the PFH features for the denoised point cloud to obtain a point cloud feature set;

[0013] Through epipolar constraint matching, associate the image feature points in the image feature set with the points in the point cloud features in the point cloud feature set that are projected onto the corresponding image plane, establish image-point cloud feature pairs, and obtain fused features;

[0014] S4. 3D Reconstruction and Optimization:

[0015] Utilize the image-point cloud correspondence in the fused features, calculate the 3D coordinates through triangulation to obtain an initial point cloud;

[0016] Based on the initial point cloud and camera parameters, construct an objective function to minimize the reprojection error and obtain an optimized 3D point cloud model;

[0017] S5. Adaptive Surface Fitting:

[0018] Based on the consistency of the point cloud normal vectors, use K-means clustering to divide the optimized 3D point cloud model into multiple sub-regions. When clustering, use the normal vector and coordinates as features, and adopt a distance metric to divide points with similar geometric features into the same sub-region;

[0019] For each sub-region, select a surface basis function according to the curvature, calculate the surface parameters of each sub-region through a weighted least squares fitting objective function, and construct an adaptive theoretical surface model;

[0020] S6. Intelligent Defect Detection and Classification:

[0021] For each point in the 3D point cloud model, calculate the signed distance to the sub-surface in the nearest adaptive theoretical surface model, arrange the signed distances of all points according to their corresponding positions in the point cloud or image to generate a distance map;

[0022] Calculate the difference value between the normal vector of each point in the 3D point cloud model and the normal vector at the projected point on the adaptive theoretical surface model, arrange the difference values of all points according to their corresponding positions in the point cloud or image to generate a normal vector difference map;

[0023] Taking each point in the 3D point cloud model as the center, set a fixed-size neighborhood, count the number of points in this neighborhood, and use this as the point density value of this point. Arrange the point density values of all points according to their corresponding positions in the point cloud or image to generate a point density map;

[0024] Use the distance map, normal vector difference map, and point density map as the input of the U-Net network, and the U-Net network outputs a defect probability map;

[0025] Extract neighborhood features for regions where the absolute value of the signed distance is greater than a preset threshold;

[0026] After finding the maximum probability value corresponding to each suspicious region according to the defect probability map, the neighborhood features are used to assist in verifying the accuracy of this judgment; when the maximum probability ≥ 0.9 and the neighborhood features match the feature pattern of the corresponding defect type, it is determined as the corresponding defect type, otherwise it is marked as a suspicious region.

[0027] In this specification, in S2:

[0028] Grayscale conversion: Use the weighted average method , convert the color image into a grayscale image; Let be the grayscale value of the grayscale image after conversion at the coordinate , , , Let the red, green, and blue channel values of the original color image at the coordinate ;

[0029] Bilateral filtering for denoising: For each pixel , calculate the weighted mean within the neighborhood : ;

[0030] Among them, is the grayscale value of the image i after bilateral filtering at the coordinate , is the original grayscale image at the coordinate The spatial distance weight represents the influence of the spatial distance between the neighborhood pixel and the current pixel on the weight, the closer the distance, the greater the weight; the grayscale difference weight reflects the effect of the grayscale difference between the neighborhood pixel and the current pixel on the weight, the closer the grayscale values, the greater the weight; is the exponential function operator, is the standard deviation of the spatial distance in bilateral filtering, is the standard deviation of the grayscale difference in bilateral filtering;

[0031] Canny edge detection: Use double thresholds to extract edges and obtain an edge image.

[0032] In this specification, in S2, perform statistical filtering on the original point cloud data: First, calculate the k-nearest neighbor average distance of each point to measure the distance relationship between the point and its surrounding points, is the j-th point in the k-nearest neighbor point set of the point , is the point The k-nearest neighbor point set; then calculate the global mean of the k-nearest neighbor average distances of all points and the standard deviation of the k-nearest neighbor average distances of all points , which are used to describe the distance distribution characteristics of the entire point cloud data, where M is the number of points in the original point cloud data; finally, retain the points with distances, remove the outliers, and obtain the denoised point cloud.

[0033] In this specification, in S3, for each edge image, use the SIFT algorithm to extract feature points, and each feature point contains the position , scale , orientation and a 128-dimensional descriptor to obtain the image feature set , where , is the number of feature points in the i-th image.

[0034] In this specification, in S3, for the denoised point cloud, calculate the PFH feature of each point : First, calculate the normal vector through the eigenvector corresponding to the minimum eigenvalue of the k-nearest neighbor covariance matrix to obtain the local geometric direction information of the point cloud; then for each point pair , calculate the angular difference: ; The angular difference describes the relationship between the directions and positions of the point pair , is the normal vector of point and the vector , is the normal vector of point and the normal vector of point ; is the angle between the vector and the normal vector of point ; Finally, construct a 3D histogram, and quantize into 10 bins respectively to obtain a 1000-dimensional PFH descriptor , that is, obtain the point cloud feature set , where N is the number of points in the denoised point cloud,[[ID=7l]]

[0035] In this specification, in S3, through epipolar constraint matching, associate the image feature points with the points projected onto the corresponding image plane in the point cloud features , and for each point cloud point , through the camera intrinsic matrix K and extrinsic , Projected onto the coordinates of image i , is the rotation matrix of the i-th camera, is the translation vector of the i-th camera, is the point Projected onto the horizontal and vertical coordinates of the i-th image plane; Find image feature points in Points with Euclidean distance less than 5 pixels are found, and image-point cloud feature pairs , is the k-th feature point of the point cloud data; The SIFT descriptor and the PFH descriptor are concatenated into a 228-dimensional vector to obtain the fused feature , is the vector concatenation operator.

[0036] In this specification, in S4, using the fused feature in the image-point cloud correspondence, the three-dimensional coordinates are calculated by triangulation. For the feature pair , in the projection equations of camera i and camera j: ;

[0037] where is the scale factor, X is the three-dimensional point coordinate to be solved, , are the horizontal and vertical coordinates of the feature point on the i-th image plane, , are the horizontal and vertical coordinates of the feature point on the j-th image plane, , are the rotation matrices of the i-th and j-th cameras, , are the translation vectors of the i-th and j-th cameras, K is the camera intrinsic matrix, and the initial coordinates of point X are solved by linear least squares method to obtain the initial point cloud.

[0038] In this specification, in S4, construct an objective function to minimize the reprojection error: ;

[0039] where, is the three-dimensional point coordinate of the j-th point, Q is the total number of feature points; is the projection function, , project the three-dimensional point onto the z-axis, i.e., the two-dimensional image plane; is the observed coordinate of the feature point j in image i.

[0040] In this specification, in S5, for each sub-region , select the surface basis function according to the curvature:

[0041] For the low-curvature region of the curvature , use a quadratic polynomial , where \(u\) and \(v\) are surface parameters, and \(x\) and \(y\) are the coordinate components of the points in the three-dimensional point cloud model on the two-dimensional plane;

[0042] For the high-curvature region of the curvature , use a NURBS surface;

[0043] The fitting objective function is weighted least squares: ;

[0044] where, are the surface parameters of the polynomial coefficients or NURBS control vertices, represents the surface function value corresponding to the point at the given parameter , that is, the theoretical \(z\)-coordinate value of the point on the fitting surface. \(x_j\), \(y_j\), \(z_j\) are the \(x\), \(y\), and \(z\) coordinates of the \(j\)-th point in the three-dimensional point cloud model respectively, where is substituted as the input into the surface function to calculate the theoretical value, while is the actual \(z\)-coordinate of the point; the weight , is the local point density of point \(j\), is the average density of the sub-region, is the curvature of point \(j\), to avoid a zero denominator.

[0045] Another aspect of the embodiments of this specification discloses a medicine bottle detection system based on surface fitting, which is used to implement the medicine bottle detection method based on surface fitting described in any one of the above, and the medicine bottle detection system based on surface fitting includes:

[0046] A multi-modal data acquisition module, which is used for:

[0047] Using an industrial camera and a line laser scanner, synchronously acquire data with the central axis of the medicine bottle as the rotation axis. Among them, the industrial camera obtains a set of two-dimensional images at multiple angles, and the line laser scanner generates raw point cloud data;

[0048] A multi-modal data preprocessing module, which is used for:

[0049] Based on the set of two-dimensional images at multiple angles, for each image, successively perform grayscale conversion, bilateral filtering for denoising, and Canny edge detection to obtain an edge image:

[0050] Perform statistical filtering and outlier removal on the original point cloud data to obtain the denoised point cloud;

[0051] Feature extraction and fusion module, for:

[0052] Use the SIFT algorithm to extract feature points from each edge image to obtain an image feature set;

[0053] Calculate the PFH features for the denoised point cloud to obtain a point cloud feature set;

[0054] Through epipolar constraint matching, associate the image feature points in the image feature set with the points in the point cloud feature set that are projected onto the corresponding image plane, establish image-point cloud feature pairs, and obtain the fused features;

[0055] 3D reconstruction and optimization module, for:

[0056] Utilize the image-point cloud correspondence relationship in the fused features, calculate the 3D coordinates through triangulation to obtain the initial point cloud;

[0057] Based on the initial point cloud and camera parameters, construct an objective function to minimize the reprojection error and obtain the optimized 3D point cloud model;

[0058] Adaptive surface fitting module, for:

[0059] Based on the consistency of the point cloud normal vectors, use K-means clustering to divide the optimized 3D point cloud model into multiple sub-regions. When clustering, use the normal vector and coordinates as features, and adopt a distance metric to divide points with similar geometric features into the same sub-region;

[0060] For each sub-region, select a surface basis function according to the curvature, calculate the surface parameters of each sub-region through a weighted least squares fitting objective function, and construct an adaptive theoretical surface model;

[0061] Defect intelligent detection and classification module, for:

[0062] For each point in the 3D point cloud model, calculate the signed distance to the sub-surface in the nearest adaptive theoretical surface model, arrange the signed distances of all points according to their corresponding positions in the point cloud or image to generate a distance map;

[0063] Calculate the difference value between the normal vector of each point in the 3D point cloud model and the normal vector at the projected point on the adaptive theoretical surface model, arrange the difference values of all points according to their corresponding positions in the point cloud or image to generate a normal vector difference map;

[0064] Taking each point in the 3D point cloud model as the center, set a neighborhood with a fixed size, and count the number of points in this neighborhood, which is used as the point density value of this point. Arrange the point density values of all points according to their corresponding positions in the point cloud or image to generate a point density map;

[0065] Use the distance map, normal vector difference map, and point density map as the input of the U-Net network, and the U-Net network outputs a defect probability map;

[0066] Extract neighborhood features for regions where the absolute value of the signed distance is greater than a preset threshold;

[0067] After finding the maximum probability value corresponding to each suspicious region according to the defect probability map, the neighborhood features are used to assist in verifying the accuracy of this judgment; when the maximum probability ≥ 0.9 and the neighborhood features match the feature pattern of the corresponding defect type, it is determined as the corresponding defect type, otherwise it is marked as a suspicious region.

[0068] The embodiments of this specification can at least achieve the following beneficial effects:

[0069] Through multi-modal data acquisition (combining industrial cameras and laser scanners), obtain the 2D image and 3D point cloud data of the medicine bottle, and comprehensively describe the surface characteristics of the medicine bottle from multiple dimensions. Combining feature extraction algorithms such as SIFT (Scale-Invariant Feature Transform) and PFH (Point Feature Histogram), as well as Bundle Adjustment global optimization technology, achieve high-precision 3D reconstruction, enabling accurate capture of even the slightest defects on the medicine bottle surface. Use the adaptive surface fitting method to automatically select appropriate surface basis functions (quadratic polynomials or NURBS surfaces) for fitting according to the curvature characteristics of different regions on the medicine bottle surface, which can adapt to the complex and diverse surface shapes of the medicine bottle, and can achieve accurate modeling for both regular and irregular regions. The improved U-Net network uses the distance map, normal vector difference map, and point density map as inputs, and can effectively identify various types of defects such as scratches, depressions, and protrusions, greatly improving the detection accuracy compared to traditional methods. The entire detection process is automated, and the system can automatically complete the process from data acquisition to result output, avoiding the low efficiency problem of manual detection. Use parallel computing and optimization algorithms, such as using epipolar constraint matching to quickly establish the image-point cloud correspondence relationship in the feature extraction and fusion link, greatly shortening the detection time and meeting the real-time detection requirements on large-scale production lines. Description of the Drawings

[0070] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0071] Figure 1 It is a schematic diagram of the steps of the vial detection method based on surface fitting involved in some embodiments of the present invention.

[0072] Figure 2 It is a schematic diagram of the process of feature extraction and fusion involved in some embodiments of the present invention.

[0073] Figure 3 It is a schematic diagram of the process of intelligent defect detection and classification involved in some embodiments of the present invention.

[0074] Figure 4 It is a schematic diagram of the modules of the vial detection system based on surface fitting involved in some embodiments of the present invention. Detailed implementation manners

[0075] In the following, only some exemplary embodiments are briefly described. As those skilled in the art can recognize, the described embodiments can be modified in various different ways without departing from the spirit or scope of the present invention. Therefore, the drawings and the description are considered to be exemplary in nature rather than restrictive.

[0076] The following will describe the embodiments of the present invention in detail with reference to the drawings.

[0077] As Figure 1 shown, this embodiment provides a vial detection method based on surface fitting, including:

[0078] S1. Multimodal data acquisition; S2. Multimodal data preprocessing; S3. Feature extraction and fusion; S4. 3D reconstruction and optimization; S5. Adaptive surface fitting; S6. Intelligent defect detection and classification.

[0079] In some embodiments, S1. Multimodal data acquisition:

[0080] Three sets of industrial cameras (each set contains two cameras with different focal lengths) and one line laser scanner are used to synchronously acquire data every 15° with the central axis of the vial as the rotation axis. The industrial cameras are used to obtain a set of two-dimensional images from multiple angles , and the resolution of each image is 2048X2048 pixels. Through a camera calibration board (checkerboard pattern, with a single grid size of 10mm×10mm), the internal parameter matrix is calibrated according to the Zhang Zhengyou calibration method , where , is the focal length pixel value, reflecting the light converging ability of the camera lens; , is the principal point coordinate, representing the position of the center of the image plane in the pixel coordinate system.

[0081] The line laser scanner emits a laser beam with a wavelength of 650 nm and scans the surface of the medicine bottle at a sampling interval of 0.1 mm to generate the original point cloud data , where M is the initial number of point clouds. Calibrate the coordinate transformation matrix of the scanner through the target ball array , ensuring the accurate mapping of the point cloud data to the world coordinate system. To ensure data synchronization, a hardware trigger module is used to generate a synchronization pulse, making the image acquisition timestamp and the point cloud acquisition timestamp have an error less than 10 μs.

[0082] In some embodiments, S2. Multi-modal data preprocessing:

[0083] 2.1 Image preprocessing (for each image )

[0084] Grayscale conversion: The weighted average method is used to convert the color image into a grayscale image . Based on the sensitivity differences of the human eye to different colors of light, this formula converts the color information into a single grayscale value by assigning different weights to the red (R), green (G), and blue (B) channels, facilitating subsequent processing.

[0085] Bilateral filtering for denoising: For each pixel , calculate the weighted mean within the neighborhood : ;

[0086] Among them, the spatial distance weight represents the influence of the spatial distance between the neighborhood pixel and the current pixel on the weight, and the closer the distance, the greater the weight; the gray level difference weight reflects the effect of the gray level difference between the neighborhood pixel and the current pixel on the weight, and the closer the gray level values, the greater the weight. , are preset control parameters, and by adjusting them, the smoothing and edge preservation effects of the filtering can be balanced. While removing noise, bilateral filtering can effectively preserve the edge information of the image, avoiding edge blurring, and is suitable for protecting the edge contour of the medicine bottle.

[0087] Canny edge detection: Use a double threshold (low threshold 30, high threshold 90) to extract edges and obtain the edge image This method accurately extracts the edge of the medicine bottle by smoothing the image through Gaussian filtering, calculating the gradient magnitude and direction, thinning the edge through non-maximum suppression, and double-threshold detection and edge connection.

[0088] 2.2 Point cloud preprocessing

[0089] Perform statistical filtering on the original point cloud data D: First, calculate the average distance of the k-nearest neighbors (k = 50) of each point to measure the distance relationship between the point and its surrounding points; then calculate the global mean and the standard deviation to describe the distance distribution characteristics of the entire point cloud data; finally, retain the points with a distance and remove the outliers to obtain the denoised point cloud D'. This method is based on statistical principles and regards the points with abnormal distance distribution as noise and eliminates them to ensure the accuracy of the point cloud data. Output data: Edge image set

[0090] , denoised point cloud D'.

[0091] In some embodiments, as Figure 2 shown, S3. Feature extraction and fusion:

[0092] 3.1 Image feature extraction

[0093] For each edge image , use the SIFT algorithm to extract feature points. Each feature point includes the position (image coordinates), scale , direction and a 128-dimensional descriptor to obtain the image feature set , where , is the number of feature points of the i-th image. The SIFT algorithm can extract feature points with scale invariance and rotation invariance by constructing a scale space, detecting extreme points, accurately positioning feature points, assigning directions, and generating descriptors, adapting to the changes of the medicine bottle under different perspectives.

[0094] 3.2 Point cloud feature extraction

[0095] For the denoised point cloud D', calculate the PFH feature of each point : First, calculate the normal vector through the eigenvector corresponding to the minimum eigenvalue of the covariance matrix of the k-nearest neighbors (k = 30) to obtain the local geometric direction information of the point cloud; then for each point pair , calculate the angular difference:

[0096] ;

[0097] Angle difference description point pairs The relationship between directions and positions among them; finally, a 3D histogram is constructed, and are respectively quantized into 10 bins to obtain a 1000-dimensional PFH descriptor . The PFH feature can comprehensively describe the local geometric features of the point cloud, providing rich information for subsequent processing. Obtain the point cloud feature set , where N is the number of point clouds after denoising.

[0098] 3.3 Feature fusion

[0099] Through epipolar constraint matching, the image feature points are associated with the points projected onto the corresponding image plane in the point cloud features . For each point cloud point , through the camera intrinsic parameter K and extrinsic parameter , Projected onto the coordinates of image i ; Search for the image feature points in which the Euclidean distance from is less than 5 pixels, and establish the image-point cloud feature pair ; Concatenate the SIFT descriptor and the PFH descriptor into a 228-dimensional vector to obtain the fused feature . Epipolar constraint matching utilizes the camera imaging geometric relationship to narrow the feature matching range and improve the matching efficiency and accuracy.

[0100] The present invention does not improve the calculation methods of the SIFT algorithm and the PFH feature, so it will not be elaborated here. Refer to the prior art, and the same applies to other existing algorithms that are not explained.

[0101] In some embodiments, S4. 3D reconstruction and optimization:

[0102] 4.1 Initial 3D reconstruction

[0103] Utilize the image-point cloud correspondence in the fused feature to calculate the 3D coordinates through triangulation. For the feature pair , the projection equations in camera i and camera j: ;

[0104] where is the scale factor, and X is the 3D point coordinate to be solved. Solve the initial coordinates of point X through the linear least squares method to obtain the initial point cloud . Triangulation is based on the principle of multi-view geometry. By using the projection relationship of feature points from different perspectives, the three-dimensional coordinates of spatial points are calculated to achieve the preliminary three-dimensional reconstruction of the medicine bottle.

[0105] 4.2 Bundle Adjustment Global Optimization

[0106] Construct the objective function to minimize the reprojection error: ;

[0107] Among them, is the rotation matrix of camera i, used to describe the rotation attitude of the camera; is the translation vector, determining the spatial position of the camera; is the coordinate of the jth three-dimensional point, and Q is the total number of feature points; is the projection function, , which projects the three-dimensional point onto the two-dimensional image plane; is the observed coordinate of feature point j in image i. The Levenberg-Marquardt algorithm is used for iterative optimization, and the convergence condition is that the change in error is less than pixels. Bundle Adjustment global optimization simultaneously adjusts all camera parameters and three-dimensional point coordinates, minimizes the reprojection error, and improves the accuracy of three-dimensional reconstruction.

[0108] Output: Optimized three-dimensional point cloud model .

[0109] In some embodiments, S5. Adaptive Surface Fitting:

[0110] 5.1 Point Cloud Region Division

[0111] Based on the consistency of the point cloud normal vector, use K-means clustering to divide P into K sub-regions , and the clustering feature is the normal vector and the coordinate , and the distance metric is:

[0112] ;

[0113] Among them, is the three-dimensional point cloud model in, are two points, are the normal vectors of points respectively, is the three-dimensional coordinate of point is the three-dimensional coordinate of point The three-dimensional coordinates; this distance metric comprehensively considers the differences in the normal vectors and spatial positions of the points, where the weight of the normal vector difference is 0.6 and the weight of the spatial position difference is 0.4. Through K-means clustering, points with similar geometric features are divided into the same sub-region to prepare for subsequent targeted surface fitting.

[0114] 5.2 Weighted Surface Fitting

[0115] For each sub-region , select the surface basis function according to the curvature:

[0116] For low-curvature regions (curvature ), use a quadratic polynomial , whose simple form is suitable for regions with relatively regular shapes and gentle changes;

[0117] For high-curvature regions (curvature ), use a NURBS surface, and the control vertices are initialized by a greedy algorithm. The NURBS surface can flexibly represent complex shapes and adapt to the irregular regions on the surface of the medicine bottle.

[0118] The fitting objective function is weighted least squares: ;

[0119] Where are surface parameters (polynomial coefficients or NURBS control vertices); the weight , is the local point density of point j (calculated by k-nearest neighbor counting), is the average density of the sub-region, is the curvature of point j (calculated by principal curvature), to avoid a zero denominator. This weight formula gives points with high point density and low curvature higher weights in the fitting, thereby improving the accuracy and adaptability of the surface fitting.

[0120] Output: Adaptive theoretical surface model , each sub-surface contains parameters and the region division label.

[0121] In some embodiments, as Figure 3 shown, S6. Intelligent Defect Detection and Classification:

[0122] 6.1 Geometric Difference Calculation

[0123] For each point , calculate the signed distance to the nearest sub-surface : ;

[0124] Where is the projection point on the curved surface , and is the normal vector at the projection point. Through vector operations, this formula obtains the perpendicular distance from a point to the curved surface and determines the sign of the distance based on the positional relationship between the point and the curved surface, which is used to measure the difference between the actual surface of the medicine bottle and the theoretical curved surface.

[0125] Distance map: For each point in the optimized three-dimensional point cloud model , calculate its signed distance to the nearest sub-curved surface in the adaptive theoretical curved surface model S. Arrange the distances of all points in accordance with their corresponding positions in the point cloud or image, and a distance map can be formed. The distance map reflects the difference between the actual surface of the medicine bottle and the theoretical curved surface, and intuitively shows the areas where defects may exist.

[0126] Normal vector difference map: In S3, the normal vectors of the point cloud have been calculated. For each point in the three-dimensional point cloud model P, its normal vector represents the local geometric direction at that point. After constructing the adaptive theoretical curved surface model S in S5, calculate the difference between the normal vector of each point in the point cloud model P and the normal vector at the projection point on the corresponding theoretical curved surface S. For example, the difference degree can be measured by calculating the cosine value of the angle between the two normal vectors (where is the normal vector of a certain point in the point cloud model P, and is the normal vector of the projection point of that point on the theoretical curved surface S). Arrange the normal vector difference values of all points in accordance with the corresponding positions to generate a normal vector difference map. This map can reflect the deviation of the local geometric direction of the medicine bottle surface from the theoretical model, which helps to identify defects with abnormal shapes.

[0127] Point density map: In S3 and S5, the calculation of point density is involved. For the three-dimensional point cloud model P, with each point as the center, set a fixed-size neighborhood (such as a spherical neighborhood with a radius of r), and count the number of points in this neighborhood, which is used as the local point density of that point. Arrange the point density values of all points in accordance with the corresponding positions to form a point density map. The point density map reflects the density of the point cloud distribution on the medicine bottle surface. In some defect areas, there may be abnormal changes in point density. For example, the point density may be relatively low in concave areas and relatively high in convex areas, thus providing an important basis for defect detection.

[0128] 6.2 Defect classification model

[0129] Use an improved U-Net network (input channels: 3, namely distance map, normal vector difference map, and point density map). The network structure includes an encoder, a decoder, and an output layer. The encoder consists of 4 convolutional layers (with a 3×3 kernel and a stride of 2), each followed by BatchNorm and ReLU. The convolutional layers are used to extract image features. BatchNorm can accelerate network convergence and reduce overfitting, and the ReLU activation function introduces non-linearity, enabling the network to learn complex feature representations. The decoder uses 4 transposed convolutional layers (with a 2×2 kernel and a stride of 2), with skip connections to the corresponding layers in the encoder. The transposed convolutional layers are used for upsampling to restore the image resolution, and the skip connections fuse features from different levels in the encoder, retaining more detailed information. The output layer generates a defect probability map through a 1×1 convolution, with a resolution consistent with the grid after downsampling the point cloud.

[0130] Training data: The positive samples are artificially manufactured defective point clouds (scratches, depressions, protrusions, a total of 1000 samples), and the defect types are labeled (0 represents no defect, 1 - 3 represent defect types); the negative samples are qualified medicine bottle point clouds (500 samples).

[0131] Loss function: ;

[0132] where is the square of the L2 norm, represents the true label of the i-th sample, = 0: indicates that the i-th sample is defect-free (negative sample), = 1, 2, 3: correspond to different defect types (such as scratches, depressions, protrusions, etc., specifically defined according to the training data annotation); log is the natural logarithm function, N1 is the total number of samples, is the predicted probability, W is the network weight, is the regularization coefficient. This loss function combines cross-entropy loss and L2 regularization. The cross-entropy loss measures the difference between the predicted probability and the true label, and L2 regularization prevents network overfitting and improves the generalization ability of the model.

[0133] Detection process: For the distance (initial threshold = 0.3mm) region, extract 3×3 neighborhood features (including distance, normal vector difference, and point density); after finding the maximum probability value corresponding to each suspicious region according to the defect probability map, the neighborhood features are used to assist in verifying the accuracy of this judgment; when the maximum probability ≥ 0.9 and the neighborhood features match the feature pattern of the corresponding defect type, it is determined as the corresponding defect type, otherwise it is marked as a suspicious region.

[0134] When judging whether the normal vector difference and point density change shown by the neighborhood features match the typical features of defects, it is necessary to combine the prior knowledge of different defect types and achieve it by setting reasonable judgment criteria and calculation methods. The specific methods are as follows: Establish a defect feature database: Based on a large amount of experimental and actual detection data accumulation, for different defect types such as scratches, dents, and protrusions, respectively count their typical features in normal vector difference and point density change. For example, in terms of normal vector difference, scratch defects usually show obvious mutations in the normal vector along the scratch direction, and the normal vector angle changes greatly between adjacent points; in terms of point density, the point density in the scratch area may show local sparsity due to surface damage. The normal vector difference of dent defects shows that the direction of the normal vector in the dent area is different from that of the surrounding normal area, forming a trend of converging towards the center of the dent; the point density will relatively increase in the dent area because more points will be collected by laser scanning at the dent. The normal vector of protrusion defects has significant changes at the top and edge of the protrusion, and the normal vector angle changes irregularly; the point density will also increase in the protrusion area, and there may be a point density gradient change at the edge of the protrusion. Record these typical features in the defect feature database in the form of data as the basis for subsequent judgment. Calculate the neighborhood feature statistics: For the extracted 3×3 neighborhood features, calculate the relevant statistics of normal vector difference and point density change. Calculate the average angle difference between the normal vectors of each point in the neighborhood as a measurement index of normal vector difference; calculate the ratio of the neighborhood point density to the average point density of the surrounding normal area to measure the point density change. Suppose there are n points in the neighborhood, and the normal vectors are , then the average angle difference can be calculated by the following formula: ;

[0135] arccos is the inverse cosine function, and the point density ratio r is the ratio of the neighborhood point density to the average point density of the surrounding normal area , that is . Set the matching threshold: According to the data distribution in the defect feature database, set reasonable matching thresholds for the normal vector difference and point density change statistics of different defect types. For scratch defects, if the average angle difference is greater than the threshold , and the point density ratio r is less than the threshold , it is considered that the normal vector difference and point density change of this neighborhood match the typical features of scratch defects; for dent defects, when the average angle difference is within a certain range (for example, greater than and less than ), and the point density ratio r is greater than the threshold , it is determined to match the dent defect features; for protrusion defects, if the average angle difference is greater than the threshold and the dot density ratio r is greater than the threshold it is considered to match the convex defect feature. These thresholds in this specification can all be optimized and adjusted on the training data through methods such as cross-validation to improve the accuracy of judgment. Comprehensive judgment: Compare the calculated neighborhood feature statistic with the set threshold, and combine the result of the defect probability map for comprehensive judgment. Only when the defect probability map shows that the probability of a certain area being a certain type of defect ≥ 0.9, and the neighborhood feature statistic of this area meets the matching conditions of the corresponding defect type, is it finally determined that this area is the corresponding defect type; otherwise, it is marked as a suspicious area and awaits further manual inspection and confirmation.

[0136] In a specific embodiment, the detection process of the medicine bottle based on surface fitting is as follows:

[0137] I. Experimental environment

[0138] 1. Hardware equipment: Three groups of industrial cameras (Basler acA2500-14gm, each group is equipped with lenses with focal lengths of 12mm and 25mm respectively) and one Keyence LJ-V7000 line laser scanner are adopted; the data processing device is a workstation equipped with an Intel Core i9-12900K processor, 64GB of memory, and an NVIDIA RTX 3090 graphics card.

[0139] 2. Software platform: Python 3.8 is used to write data processing code, OpenCV 4.5.5 is used for image preprocessing and feature extraction, PCL 1.12.1 is used to process point cloud data, and an improved U-Net network model is built based on PyTorch 1.12.0.

[0140] II. Experimental objects

[0141] Cylindrical plastic medicine bottles produced by a certain pharmaceutical enterprise are selected, with a bottle body height of 80mm, a bottom diameter of 30mm, and a wall thickness of 2mm. Three types of defects are artificially created: 10 scratches with a length of 15mm and a depth of 0.3mm are made on the bottle body; 10 depressions with a diameter of 8mm and a depth of 0.5mm; 10 protrusions with a diameter of 6mm and a height of 0.4mm; at the same time, 10 defect-free qualified medicine bottles are prepared as control samples.

[0142] III. Experimental steps and data

[0143] 1. Multi-modal data acquisition

[0144] Taking the central axis of the medicine bottle as the rotation axis, images are collected every 15° using industrial cameras, and a total of 24 two-dimensional images with a resolution of 2048×2048 pixels are obtained; the line laser scanner scans synchronously to generate the original point cloud data D containing 23,568 points.

[0145] 2. Multi-modal Data Preprocessing

[0146] Image Preprocessing: After grayscale conversion of each image, bilateral filtering is performed for denoising ( =3, =20). The proportion of noise points in the image before denoising is about 8%, which is reduced to 1.2% after processing; Canny edge detection is used (low threshold 30, high threshold 90), and the clear edge contour of the medicine bottle is successfully extracted to obtain the edge image set .

[0147] Point Cloud Preprocessing: Statistical filtering (k = 50) is performed on the original point cloud data D, and 1287 outlier points are removed to obtain the denoised point cloud D' containing 22,281 points.

[0148] 3. Feature Extraction and Fusion

[0149] Image Feature Extraction: The SIFT algorithm is used to extract feature points from 24 edge images. On average, 327 feature points are extracted from each image, and each feature point generates a 128-dimensional descriptor to construct the image feature set .

[0150] Point Cloud Feature Extraction: The PFH feature is calculated for the denoised point cloud D', and each point generates a 1000-dimensional PFH descriptor to form the point cloud feature set , which contains the features of 22,281 points.

[0151] Feature Fusion: Through epipolar constraint matching, 1865 groups of image-point cloud feature pairs are established. The SIFT descriptor and the PFH descriptor are concatenated into a 228-dimensional vector to obtain the fused feature .

[0152] 4. 3D Reconstruction and Optimization

[0153] Initial 3D Reconstruction: Based on the fused feature , the 3D coordinates are calculated through triangulation to obtain the initial point cloud containing 1865 points .

[0154] Global Optimization (Bundle Adjustment): Based on the initial point cloud and camera parameters, an objective function is constructed to minimize the reprojection error. After iterative optimization by the Levenberg-Marquardt algorithm, the reprojection error is reduced from 0.87 pixels to 0.08 pixels, and the optimized 3D point cloud model P containing 1865 accurate 3D point coordinates is obtained.

[0155] 5. Adaptive Surface Fitting

[0156] Point cloud region division: Based on the consistency of the point cloud normal vector, the three-dimensional point cloud model P is divided into 5 sub-regions (3 low-curvature regions and 2 high-curvature regions) using K-means clustering. In the clustering distance metric, the weight of the normal vector difference is 0.6, and the weight of the coordinate difference is 0.4.

[0157] Weighted surface fitting: Quadratic polynomial fitting is used for the low-curvature regions, and NURBS surface fitting is used for the high-curvature regions. After fitting, the average distance from the points in the low-curvature regions to the fitted surface is 0.03 mm, and that in the high-curvature regions is 0.05 mm, constructing an adaptive theoretical surface model S.

[0158] 6. Intelligent defect detection and classification

[0159] Geometric difference calculation: Calculate the signed distance from the points in the three-dimensional point cloud model P to the theoretical surface model S, and set the initial threshold = 0.3 mm, detecting 38 suspected defect regions, and obtaining the distance map, normal vector difference map, and point density map.

[0160] Defect classification model: Input the above three types of maps into the improved U-Net network. The network training data includes 1000 positive samples (defect point clouds) and 500 negative samples (qualified medicine bottle point clouds). For the suspected defect regions, extract the 3×3 neighborhood features and combine them with the defect probability map for judgment. If a certain region shows a relatively high probability of scratch defect in the defect probability map, but the average angle difference of the normal vectors of the neighborhood features in this region is 15° (set the threshold of the average angle difference of the normal vectors for scratch defects = 20°), and the point density ratio is 0.9 (set the threshold of the point density ratio for scratch defects = 0.8), which does not meet the typical features of scratch defects. Even if the probability value reaches the threshold, it is still marked as a suspicious region. After detection, the detection accuracies of the network for scratches, depressions, and protrusions are 92%, 88%, and 90% respectively, and finally 29 real defects are accurately identified.

[0161] IV. Experimental result analysis

[0162] This experiment verifies the effectiveness of the medicine bottle detection method based on surface fitting. Through multi-modal data fusion and three-dimensional reconstruction, the detection accuracy for artificially set defects reaches 90.6% (29 / 32), which is about 25% higher than that of the single-modal detection method. The combination of adaptive surface fitting and the improved U-Net network with neighborhood feature judgment effectively reduces the false detection rate, providing reliable technical support for the quality detection of medicine bottles.

[0163] This embodiment provides a medicine bottle detection system based on surface fitting for implementing the medicine bottle detection method based on surface fitting described in any one of the above, as Figure 4 shown, the medicine bottle detection system based on surface fitting includes:

[0164] A multi-modal data acquisition module, for:

[0165] Using an industrial camera and a line laser scanner, synchronously acquire data with the central axis of the medicine bottle as the rotation axis, where the industrial camera obtains a multi-angle two-dimensional image set, and the line laser scanner generates raw point cloud data;

[0166] A multi-modal data preprocessing module, for:

[0167] Based on the multi-angle two-dimensional image set, for each image, sequentially perform grayscale conversion, bilateral filtering for denoising, and Canny edge detection to obtain an edge image:

[0168] Perform statistical filtering and outlier removal on the raw point cloud data to obtain denoised point cloud;

[0169] A feature extraction and fusion module, for:

[0170] Use the SIFT algorithm to extract feature points from each edge image to obtain an image feature set;

[0171] Calculate the PFH feature for the denoised point cloud to obtain a point cloud feature set;

[0172] Through epipolar constraint matching, associate the image feature points in the image feature set with the points in the point cloud feature set that are projected onto the corresponding image plane in the point cloud feature, establish an image-point cloud feature pair, and obtain a fused feature;

[0173] A 3D reconstruction and optimization module, for:

[0174] Utilize the image-point cloud correspondence relationship in the fused feature, calculate the 3D coordinates through triangulation to obtain an initial point cloud;

[0175] Based on the initial point cloud and camera parameters, construct an objective function to minimize the reprojection error to obtain an optimized 3D point cloud model;

[0176] An adaptive surface fitting module, for:

[0177] Based on the consistency of point cloud normal vectors, use K-means clustering to divide the optimized 3D point cloud model into multiple sub-regions. When clustering, use the normal vector and coordinates as features, and adopt a distance metric to divide points with similar geometric features into the same sub-region;

[0178] For each sub-region, select a surface basis function according to the curvature, calculate the surface parameters of each sub-region through a weighted least squares fitting objective function, and construct an adaptive theoretical surface model;

[0179] A defect intelligent detection and classification module, for:

[0180] For each point in the three-dimensional point cloud model, calculate the signed distance to the sub-surface in the nearest adaptive theoretical surface model, arrange the signed distances of all points according to their corresponding positions in the point cloud or image, and generate a distance map;

[0181] Calculate the difference value between the normal vector of each point in the three-dimensional point cloud model and the normal vector at the projected point on the adaptive theoretical surface model, arrange the difference values of all points according to their corresponding positions in the point cloud or image, and generate a normal vector difference map;

[0182] Taking each point in the three-dimensional point cloud model as the center, set a neighborhood with a fixed size, count the number of points in this neighborhood, and use this as the point density value of this point. Arrange the point density values of all points according to their corresponding positions in the point cloud or image, and generate a point density map;

[0183] Use the distance map, normal vector difference map, and point density map as the input of the U-Net network, and the U-Net network outputs a defect probability map;

[0184] Extract the neighborhood features for the regions where the absolute value of the signed distance is greater than the preset threshold;

[0185] After finding the maximum probability value corresponding to each suspicious region according to the defect probability map, the neighborhood features are used to assist in verifying the accuracy of this judgment; when the maximum probability ≥ 0.9 and the neighborhood features match the feature pattern of the corresponding defect type, it is determined as the corresponding defect type, otherwise it is marked as a suspicious region.

[0186] In summary, multiple specific embodiments of the present invention are disclosed. Without self-contradiction, each embodiment can be freely combined to form a new embodiment, that is, the embodiments belonging to the replacement scheme can be freely replaced with each other, but cannot be combined with each other; the embodiments that do not belong to the replacement scheme can be combined with each other, and these new embodiments also belong to the substantial content of the present invention.

[0187] The above embodiments describe multiple specific implementation manners of the present invention. However, those skilled in the art should understand that without departing from the principles and essence of the present invention, various changes or modifications can be made to these implementation manners, but these changes and modifications all fall within the protection scope of the present invention.

Claims

1. A medicine bottle detection method based on surface fitting, characterized in that Including: S1. Multimodal data acquisition: Using an industrial camera and a line laser scanner, data is synchronously acquired with the central axis of the medicine bottle as the rotation axis. Among them, the industrial camera obtains a set of two-dimensional images from multiple angles, and the line laser scanner generates raw point cloud data; S2. Multimodal data preprocessing: Based on the set of two-dimensional images from multiple angles, for each image, grayscale conversion, bilateral filtering for denoising, and Canny edge detection are sequentially performed to obtain an edge image: Statistical filtering and outlier removal are performed on the raw point cloud data to obtain denoised point cloud; S3. Feature extraction and fusion: Using the SIFT algorithm to extract feature points from each edge image to obtain an image feature set; Calculating the PFH feature for the denoised point cloud to obtain a point cloud feature set; Through epipolar constraint matching, the image feature points in the image feature set are associated with the points in the point cloud feature set that are projected onto the corresponding image plane in the point cloud feature, establishing an image-point cloud feature pair to obtain fused features; S4. 3D reconstruction and optimization: Using the image-point cloud correspondence relationship in the fused features, calculating the 3D coordinates through triangulation to obtain an initial point cloud; Based on the initial point cloud and camera parameters, constructing an objective function to minimize the reprojection error to obtain an optimized 3D point cloud model; S5. Adaptive surface fitting: Based on the consistency of the point cloud normal vectors, using K-means clustering to divide the optimized 3D point cloud model into multiple sub-regions. When clustering, the normal vector and coordinates are used as features, and the distance metric is used to divide points with similar geometric features into the same sub-region; For each sub-region, select the surface basis function according to the curvature, calculate the surface parameters of each sub-region through the weighted least squares fitting objective function, and construct an adaptive theoretical surface model; S6. Intelligent defect detection and classification: For each point in the 3D point cloud model, calculate the signed distance to the sub-surface in the nearest adaptive theoretical surface model, arrange the signed distances of all points according to their corresponding positions in the point cloud or image to generate a distance map; Calculate the difference value between the normal vector of each point in the 3D point cloud model and the normal vector at the projected point on the adaptive theoretical surface model, arrange the difference values of all points according to their corresponding positions in the point cloud or image to generate a normal vector difference map; Taking each point in the 3D point cloud model as the center, set a fixed-size neighborhood, and count the number of points in this neighborhood as the point density value of this point. Arrange the point density values of all points according to their corresponding positions in the point cloud or image to generate a point density map; Taking the distance map, normal vector difference map, and point density map as the input of the U-Net network, and the U-Net network outputs a defect probability map; Extract the neighborhood features for the region where the absolute value of the signed distance is greater than the preset threshold; After finding the maximum probability value corresponding to each suspicious region according to the defect probability map, the neighborhood features are used to assist in verifying the accuracy of this judgment; when the maximum probability ≥ 0.9 and the neighborhood features match the feature pattern of the corresponding defect type, it is determined as the corresponding defect type, otherwise it is marked as a suspicious region.

2. The method for detecting a medicine bottle based on surface fitting according to claim 1, wherein In S2: Grayscale conversion: Using the weighted average method , convert the color image into a grayscale image; Let be the grayscale value of the image after grayscale conversion at the coordinate , , , Let the red, green, and blue channel values of the original color image at the coordinate be; Bilateral filtering for denoising: For each pixel , calculate the weighted mean within the neighborhood : ; Among them, is the gray value of the image i after bilateral filtering at the coordinate . is the original gray image at the coordinate . The spatial distance weight represents the influence of the spatial distance between neighboring pixels and the current pixel on the weight. The closer the distance, the greater the weight. The gray difference weight reflects the effect of the gray difference between neighboring pixels and the current pixel on the weight. The closer the gray values, the greater the weight. is the exponential function operator, is the standard deviation of the spatial distance in bilateral filtering, is the standard deviation of the gray difference in bilateral filtering. Canny edge detection: Using double thresholds to extract edges to obtain an edge image.

3. The method for detecting a medicine bottle based on surface fitting according to claim 1, wherein Statistical filtering is performed on the original point cloud data in S2: First, calculate each point 's k-nearest neighbor average distance to measure the distance relationship between the point and its surrounding points. is the j-th point in the k-nearest neighbor point set of the point , and is the k-nearest neighbor point set of the point ; then calculate the global mean of the k-nearest neighbor average distances of all points and the standard deviation of the k-nearest neighbor average distances of all points, which are used to describe the distance distribution characteristics of the entire point cloud data, and M is the number of points in the original point cloud data; finally, retain the points with a distance and remove the outliers to obtain the denoised point cloud.

4. The method for detecting a medicine bottle based on surface fitting according to claim 1, wherein In S3, for each edge image, the SIFT algorithm is used to extract feature points, and each feature point contains a position , scale , orientation and a 128-dimensional descriptor , obtaining an image feature set , where , is the number of feature points in the i-th image.

5. The method for detecting a medicine bottle based on surface fitting according to claim 4, wherein, In S3, for the denoised point cloud, calculate each point 's PFH feature: First, calculate the normal vector through the eigenvector corresponding to the minimum eigenvalue of the k-nearest neighbor covariance matrix to obtain the local geometric direction information of the point cloud; then for each point pair , calculate the angular difference: ; The angular difference describes the relationship between the directions and positions of the point pair , is the angle between the normal vector of point and the vector , is the angle between the normal vector of point and the normal vector of point , ; is the angle between the vector and the normal vector of point ; Finally, construct a 3D histogram, and are respectively quantized into 10 bins to obtain a 1000-dimensional PFH descriptor , that is, obtain the point cloud feature set , where N is the number of points in the denoised point cloud, is the three-dimensional coordinate value of the k-th point in the point cloud data.

6. The method for detecting a medicine bottle based on surface fitting according to claim 5, wherein In S3, through epipolar constraint matching, the image feature points are associated with the points projected onto the corresponding image plane in the point cloud feature . For each point cloud point , through the camera intrinsic matrix K and extrinsic , it is projected onto the coordinates of image i , is the rotation matrix of the i-th camera, is the translation vector of the i-th camera, are the horizontal and vertical coordinates of the point projected onto the i-th image plane; Search for the points in the image feature points whose Euclidean distance from is less than 5 pixels, and establish the image-point cloud feature pair , is the k-th feature point of the point cloud data; Concatenate the SIFT descriptor and the PFH descriptor into a 228-dimensional vector to obtain the fused feature , is the vector concatenation operator.

7. The method for detecting medicine bottles based on surface fitting according to claim 1, wherein In S4, using the fused features in the image-point cloud correspondence, calculate the three-dimensional coordinates through triangulation. For the feature pair , the projection equations of camera i and camera j are: ; where is the scale factor, X is the three-dimensional point coordinate to be solved, and are the feature points in the horizontal and vertical coordinates of the i-th image plane, and are the feature points in the horizontal and vertical coordinates of the j-th image plane, and are the rotation matrices of the i-th and j-th cameras, and are the translation vectors of the i-th and j-th cameras, K is the camera internal parameter matrix, and the initial coordinates of point X are solved by linear least squares to obtain the initial point cloud.

8. The method for detecting a medicine bottle based on surface fitting according to claim 7, wherein, In S4, a target function is constructed to minimize the reprojection error: ; where, is the coordinate of the j-th three-dimensional point, and Q is the total number of feature points; is the projection function, projects the three-dimensional point onto the z-axis, i.e., the two-dimensional image plane; is the observed coordinate of feature point j in image i.

9. The method for detecting a medicine bottle based on surface fitting according to claim 1, wherein In S5, for each sub-region , select surface basis functions according to the curvature: For curvature In the low-curvature region , where \(u\) and \(v\) are surface parameters, and \(x\) and \(y\) are the coordinate components of points in the 3D point cloud model on the 2D plane; For curvature For high-curvature regions, NURBS surfaces are used; The fitting objective function is weighted least squares: ; Among them, is the surface parameter of the polynomial coefficient or NURBS control vertex, indicating that at the given parameter the surface function value corresponding to the point i.e., the z - coordinate value of this point on the fitting surface theoretically, are the x, y, and z coordinates of the j - th point in the 3D point cloud model respectively, where is substituted as the input into the surface function to calculate the theoretical value, while is the actual z - coordinate of this point; the weight , is the local point density of point j, is the average density of the sub - region, is the curvature of point j, to avoid a zero denominator.

10. The medicine bottle detection system based on surface fitting is characterized in that For implementing the method for detecting medicine bottles based on surface fitting described in any one of claims 1 to 9, the system for detecting medicine bottles based on surface fitting includes: A multimodal data acquisition module, configured to: Use an industrial camera and a line laser scanner to synchronously acquire data with the central axis of the medicine bottle as the rotation axis. Among them, the industrial camera obtains a set of two-dimensional images from multiple angles, and the line laser scanner generates raw point cloud data; A multimodal data preprocessing module, configured to: Based on the set of two-dimensional images from multiple angles, for each image, perform grayscale conversion, bilateral filtering for denoising, and Canny edge detection in sequence to obtain an edge image; Perform statistical filtering and outlier removal on the raw point cloud data to obtain the denoised point cloud; A feature extraction and fusion module, configured to: Use the SIFT algorithm to extract feature points from each edge image to obtain a set of image features; Calculate the PFH features for the denoised point cloud to obtain a set of point cloud features; Through epipolar constraint matching, associate the image feature points in the set of image features with the points in the point cloud features in the set of point cloud features that are projected onto the corresponding image plane, establish image-point cloud feature pairs, and obtain fused features; A 3D reconstruction and optimization module, configured to: Utilize the image-point cloud correspondence relationship in the fused features, calculate the 3D coordinates through triangulation to obtain the initial point cloud; Based on the initial point cloud and camera parameters, construct an objective function to minimize the reprojection error to obtain an optimized 3D point cloud model; An adaptive surface fitting module, configured to: Based on the consistency of the point cloud normal vectors, use K-means clustering to divide the optimized 3D point cloud model into multiple sub-regions. When clustering, use the normal vector and coordinates as features, and adopt a distance metric to divide points with similar geometric features into the same sub-region; For each sub-region, select a surface basis function according to the curvature, calculate the surface parameters of each sub-region through a weighted least squares fitting objective function, and construct an adaptive theoretical surface model; A defect intelligent detection and classification module, configured to: For each point in the 3D point cloud model, calculate the signed distance to the sub-surface in the nearest adaptive theoretical surface model, arrange the signed distances of all points according to their corresponding positions in the point cloud or image to generate a distance map; Calculate the difference value between the normal vector of each point in the 3D point cloud model and the normal vector at the projected point on the adaptive theoretical surface model, arrange the difference values of all points according to their corresponding positions in the point cloud or image to generate a normal vector difference map; Set a fixed-size neighborhood centered on each point in the 3D point cloud model, count the number of points in this neighborhood as the point density value of this point, and arrange the point density values of all points according to their corresponding positions in the point cloud or image to generate a point density map; Use the distance map, normal vector difference map, and point density map as the input of the U-Net network, and the U-Net network outputs a defect probability map; Extract neighborhood features for regions where the absolute value of the signed distance is greater than a preset threshold; After finding the maximum probability value corresponding to each suspicious area according to the defect probability map, the neighborhood features are used to assist in verifying the accuracy of this judgment; when the maximum probability ≥ 0.9 and the neighborhood features match the feature pattern of the corresponding defect type, it is determined as the corresponding defect type, otherwise it is marked as a suspicious area.

Citation Information

Patent Citations

  • Multimode visual fusion ballastless track structure hidden defect detection method and system

    CN115841477A

  • Special-shaped cavity component model reconstruction method based on laser and ultrasonic information fusion

    CN118411462A

  • Visual inspection method for appearance of product printed by 3D printer

    CN120014177A

  • Vision system and analytical method for planar surface segmentation

    US20160203387A1

  • Surface defect detection model training method, and surface defect detection method and system

    US20240169510A1

Cited By

  • Intelligent mold quality detection method and system, equipment and medium

    CN120800259A

  • An intelligent mold quality detection method, system, device and medium

    CN120800259B

  • Shell type identification method and system based on deep learning

    CN121074524A

  • Defect detection method and system, computer equipment and storage medium

    CN121213556A

  • Defect detection method, system, computer device and storage medium

    CN121213556B