Online detection system for medicine packaging based on three-dimensional reconstruction

Abstract

The application discloses a medicine packaging online detection system based on three-dimensional reconstruction and relates to the technical field of three-dimensional reconstruction.The system collects a double-view group covering the whole surface of medicine packaging and containing overlapping surfaces through a numerical control turntable cooperating with a binocular industrial camera;feature point registration is performed through a polarization fusion SIFT algorithm; a packaging region is positioned through a model fusing packaging topological priori and edge gradient constraint; a disparity map is generated based on a semi-global block matching algorithm; and a complete packaging point cloud is spliced by combining camera calibration parameters and turntable rotation angles; a three-dimensional grid model is generated through Poisson surface reconstruction; surface concave-convex defects are accurately detected through curvature mutation analysis and a tolerance threshold comparison method; and a defect information set is finally output to complete qualification determination through linkage of a sorting mechanism.The system can detect the whole surface of medicine packaging with high precision, is suitable for multiple specifications of packaging, and meets the quality control requirements of industrial production.

Description

Technical Field

[0001] This invention relates to the field of three-dimensional reconstruction technology, and more specifically to an online inspection system for pharmaceutical packaging based on three-dimensional reconstruction. Background Technology

[0002] In the field of pharmaceutical packaging quality inspection, online inspection technology needs to balance inspection accuracy, efficiency, and adaptability to multiple specifications to meet the control requirements of industrialized mass production. Current technologies, such as traditional pharmaceutical packaging inspection, rely heavily on two-dimensional image analysis, which can only capture surface planar information and cannot cover the entire surface of the packaging's three-dimensional structure. This leads to missed detections of three-dimensional defects such as depressions and protrusions due to blind spots. Furthermore, two-dimensional inspection is significantly affected by changes in lighting and blurred surface textures. The lack of effective noise suppression and polarization information fusion mechanisms during feature registration results in large registration errors. In the region segmentation stage, existing models often fail to consider the topological characteristics of pharmaceutical packaging, making them difficult to adapt to packaging products of different specifications and shapes. Relying solely on single edge features can lead to boundary positioning deviations, affecting subsequent inspection accuracy. In addition, existing three-dimensional inspection solutions often suffer from discontinuous point cloud stitching, reliance on manual experience or simple threshold comparisons for defect judgment, and an inability to accurately classify and quantify concave and convex defects. These issues result in low inspection efficiency and high false positive rates, failing to meet the demands of online, high-precision quality control for pharmaceutical packaging. Summary of the Invention

[0003] The purpose of this invention is to provide an online inspection system for pharmaceutical packaging based on three-dimensional reconstruction. By using polarization fusion SIFT for precise registration, topological prior combined with edge gradient constraints to locate the packaging area, and combining three-dimensional reconstruction with curvature mutation quantitative analysis, the system achieves high-precision, blind-angle-free defect detection of pharmaceutical packaging.

[0004] The technical solution to achieve the objective of this invention is as follows:

[0005] The online inspection system for pharmaceutical packaging based on 3D reconstruction includes an acquisition module, an image registration module, a region segmentation module, a point cloud generation module, and a calculation and judgment module.

[0006] The acquisition module controls the turntable to rotate so that the binocular industrial camera can capture images of the drug packaging, obtaining a set of dual views covering the entire surface of the drug packaging and including overlapping surfaces;

[0007] The image registration module filters and denoises the dual-view group and obtains polarization parameters. It uses the polarization fusion SIFT algorithm to extract texture features and corrects them in combination with polarization parameters to obtain corrected feature points. The feature point registration relationship is determined based on the random sampling consensus algorithm optimized by particle swarm optimization.

[0008] The region segmentation module locates the left packaging region based on a segmentation model that integrates packaging topological priors and edge gradient constraints, and maps the right packaging region according to the registration relationship.

[0009] The point cloud generation module obtains parallax through a semi-global block matching algorithm, and combines it with camera calibration parameters and turntable rotation angle to obtain a complete packaged point cloud;

[0010] The calculation and judgment module obtains a three-dimensional mesh model through Poisson surface reconstruction, uses curvature mutation analysis and tolerance threshold comparison to detect defects, generates a defect information set, and links with the sorting mechanism to determine the pass / fail status.

[0011] Furthermore, the drug packaging is transferred to the CNC turntable on the production line. When the drug packaging is on the CNC turntable, this is the initial pose of the drug packaging. At this time, the binocular industrial camera fixed on the side of the production line simultaneously captures the left and right views to form a primary view group. After the turntable rotates a certain angle, the drug packaging presents a new observation angle, and the binocular industrial camera triggers synchronous acquisition again to obtain a secondary view group. The two view groups can completely cover all the outer surfaces of the drug packaging, and there is an overlapping surface under adjacent shooting angles. This overlapping surface is used for subsequent point cloud stitching and positioning.

[0012] Furthermore, the image registration module establishes the registration relationship between the two views using the polarization fusion SIFT algorithm, including the following steps:

[0013] The polarization parameters are obtained by using the built-in polarization sensor of the binocular industrial camera. The polarization parameters include the transmission axis angle of the polarizer of the left camera and the transmission axis angle of the polarizer of the right camera. Based on the first left view and the first right view of the first view group, and the second left view and the second right view of the second view group, the polarization angle and polarization degree corresponding to each pixel coordinate position in each view are obtained. Gaussian filtering is performed on the first left view, the first right view, the second left view, and the second right view to suppress high-frequency noise in the image and retain the texture edge features of the pharmaceutical packaging surface. Finally, the first standard left view, the first standard right view, the second standard left view, and the second standard right view are obtained.

[0014] Based on the standard images of the first-view group and the second-view group, the SIFT algorithm is used to extract initial feature points. A multi-scale image pyramid is constructed for each standard image, and the difference Gaussian response of the adjacent scale layer images is calculated. The difference Gaussian response is the difference between the Gaussian blurred image of the next scale layer and the Gaussian blurred image of the previous scale layer. A difference Gaussian scale space is generated to locate potential feature points. For local extrema points in the difference Gaussian scale space, low-contrast points and edge response points are removed by Taylor expansion fitting. After screening, the first-left initial feature point set, the first-right initial feature point set, the second-left initial feature point set, and the second-right initial feature point set are obtained respectively. Each initial feature point set contains several feature points, and each feature point contains coordinates and scale information.

[0015] Each feature point in the initial feature point set is assigned an orientation value to make the feature points rotation invariant. Taking the feature point as the center, the pixel neighborhood within a preset multiple of its scale is taken. Based on the gray-level difference in the horizontal and vertical directions of the pixel, the gradient magnitude and gradient direction of each pixel in the pixel neighborhood are calculated. The gradient directions in the entire range are weighted and statistically analyzed to generate a multi-interval orientation histogram. The direction corresponding to the peak of the histogram is taken as the main direction of the feature point. If there are other peaks that exceed the fixed proportion of the main peak, they are added as auxiliary directions.

[0016] Based on the direction assignment results, initial descriptors are generated for each initial feature point set. Taking the main direction of the feature point as the reference, its pixel neighborhood is divided into several sub-regions. For each sub-region, the gradient histogram of all directions is calculated at a preset angle interval to obtain the vector of the corresponding dimension. The vectors of all sub-regions are concatenated to form the initial descriptor. For each initial feature point set, a first-order left initial descriptor set, a first-order right initial descriptor set, a second-order left initial descriptor set, and a second-order right initial descriptor set are obtained respectively.

[0017] The initial feature point sets and corresponding initial descriptor sets are fused using polarization features to generate corrected feature points and corrected descriptors. For each feature point in each initial feature point set, its corresponding pixel neighborhood when the initial descriptor set was generated is taken. Based on the polarization angle and polarization degree within this pixel neighborhood, a polarization feature sub-vector is calculated. Starting from the horizontal direction, the polarization angle range corresponding to a half-turn clockwise is divided into several intervals. The proportion of pixels in each interval within the neighborhood is counted to obtain the interval proportion vector of the corresponding dimension. At the same time, the arithmetic mean of the polarization degree within the neighborhood is calculated to obtain a 1-dimensional feature. The interval proportion vector is then compared with the arithmetic mean... The initial descriptor is concatenated to form a polarization feature sub-vector. The initial descriptor and the polarization feature sub-vector of the corresponding feature point are weighted and fused to obtain the corrected descriptor. The fusion weight includes a first weight and a second weight, and the first weight plus the second weight equals 1. The second weight is dynamically adjusted based on the arithmetic mean of the polarization degree in the polarization feature sub-vector. The adjustment parameter is a preset value calibrated by experiments to avoid weight imbalance when the arithmetic mean of the polarization degree is too small. Each initial feature point is associated with its corresponding corrected descriptor to form a first left corrected feature point set, a first right corrected feature point set, a second left corrected feature point set, and a second right corrected feature point set, respectively.

[0018] For each feature point in the first-order left-corrected feature point set, traverse all feature points in the first-order right-corrected feature point set, calculate the Euclidean distance between the corrected descriptors of the two feature points, select the two first-order right-corrected feature points with the smallest Euclidean distance for each first-order left-corrected feature point, calculate the ratio of the two smallest distances, and if the ratio is less than a threshold, then the first-order left-corrected feature point and the corresponding first-order right-corrected feature point are determined to be an initial matching pair. After traversing all first-order left-corrected feature points, a first-order initial matching pair set is obtained. Perform the same operation on the second-order left-corrected feature point set and the second-order right-corrected feature point set to obtain a second-order initial matching pair set.

[0019] A random sampling consensus algorithm with fusion particle swarm optimization is adopted to optimize the first and second initial matching pairs, eliminate mismatches, and determine the final feature point registration relationship. It is assumed that the left and right views of the same view group satisfy the perspective transformation relationship, and the perspective transformation is described by the homography matrix that characterizes the coordinate mapping relationship of the feature points of the two views.

[0020] Build size For a particle swarm, each particle is assigned a set of randomly selected minimum non-collinear matching pairs. There are 4 matching pairs in total, which meet the minimum sample size for solving the homography matrix. This set of matching pairs is defined as the initial sample set of the particles. The initial homography matrix of the corresponding particles is solved based on the initial sample set.

[0021] For each particle's initial homography matrix, the reprojection error of all initial matching pairs after transformation by this matrix is ​​calculated. The reprojection error is the Euclidean distance between the actual coordinates of the feature points in the right view and the predicted coordinates of the feature points in the left view obtained after homography matrix transformation. Matching pairs with reprojection errors less than a preset error threshold are determined as inliers, and the number of inliers is used to determine the inlier count. (1 − average reprojection error of interior points) is used as the fitness value of the particle;

[0022] Record the local optimal position of each particle, which is the sample set corresponding to the highest historical fitness value of that particle. Record the global optimal position of the entire particle swarm, which is the sample set corresponding to the highest historical fitness value of all particles. Adjust the particle speed according to the particle's current speed, inertia weight, local optimal guiding force, and global optimal guiding force. The inertia weight maintains the particle's current search direction. The local optimal guiding force prompts the particle to move closer to its own high-quality sample set. The global optimal guiding force prompts the particle to move closer to the high-quality sample set of the swarm. Update the particle's sample set according to the adjusted speed. Prioritize retaining high-frequency matching pairs in the global optimal position and local optimal position sample sets. High-frequency matching pairs are regarded as potential high-quality interior points. Replace matching pairs in the sample set that have low frequency of occurrence and large reprojection error. Resolve the homography matrix corresponding to the particle based on the updated sample set.

[0023] Repeat the process until the number of iterations reaches the preset maximum, or the particle fitness value corresponding to the global optimal position remains unchanged for multiple consecutive rounds. At this point, convergence is determined, and the homography matrix corresponding to the global optimal position is selected as the final transformation model. All interior points in the initial matching pairs with reprojection errors less than the preset error threshold are retained to form the final matching pair set. Based on the one-to-one correspondence between the left view corrected feature points and the right view corrected feature points in the final matching pair set, the feature point registration relationship between the left view and the right view in the first view group and the second view group is constructed respectively.

[0024] Furthermore, the region segmentation module identifies the left wrapper region of the left view of each standard view group by fusing the wrapper topological prior and edge gradient constraints into an image segmentation model, and obtains the right wrapper region based on the feature point registration relationship, including the following steps:

[0025] Based on the standard design parameters of pharmaceutical packaging, a packaging topology prior is constructed. The packaging topology prior includes the overall spatial topology of the packaging, the geometric feature constraints of each face, and the projection morphology features under typical views. The packaging topology prior is transformed into a quantifiable set of parameters, which includes the minimum area threshold of the face region, the maximum area threshold of the face region, the angle constraints of adjacent faces in the image, and the smoothness threshold of the face boundary, all calculated based on the actual size of the packaging and the image resolution.

[0026] Edge gradient features are extracted from the first and second standard left views and an initial edge map is generated. The gradient operator is used to perform multi-scale gradient calculation on the two standard left views to obtain the segmentation gradient magnitude and segmentation gradient direction of each pixel. The segmentation gradient magnitude represents the edge intensity to distinguish the packaging boundary from the background area, and the segmentation gradient direction represents the edge direction to determine the boundary continuity. The scale parameter is dynamically adjusted according to the texture of the packaging surface according to the experimentally calibrated rules.

[0027] Non-maximum suppression is applied to the segmentation gradient magnitude to refine the edges by retaining strong gradient pixels at the edge center and removing weak gradient pixels on both sides of the edge. A dual threshold method is used to filter continuous edge contours, retaining strong edges through a high threshold and connecting weak edges through a low threshold, thus forming the initial edge map.

[0028] Based on the geometric constraints and edge gradient features of the packaging topology prior, the left view is initially segmented and candidate packaging regions are selected. A region growing algorithm is adopted, using strong edge pixels in the initial edge map as seed points, and multiple connected regions are generated by region growing according to the consistency of the segmentation gradient direction and gray-level similarity.

[0029] The generated connected regions are matched and verified with the packaging topology prior. The verification includes area verification, shape verification, and connectivity verification. Area verification removes regions with areas smaller than the minimum area threshold or larger than the maximum area threshold. Shape verification extracts the region shape using the minimum bounding rectangle or convex hull algorithm and removes regions whose projection shape deviates from the corresponding surface shape in the topology prior by more than a preset range. Connectivity verification checks whether the adjacent regions of the retained regions satisfy the connectivity relationship in the topology prior and removes isolated regions that do not conform to the connectivity relationship. After screening, a set of candidate packaging regions for the left view is obtained.

[0030] The candidate packaging region set is refined. Based on the gradient direction in the edge gradient features, the boundary of the candidate region is located and the boundary offset is corrected at the sub-pixel level to ensure that the boundary is aligned with the actual packaging edge. Morphological operations are used to remove small hole noise in the candidate region. Small noise regions with an area ratio less than the preset ratio are removed by connected component analysis. Combined with the constraint of covering all faces in the packaging topology prior, it is checked whether the candidate region set covers all packaging faces that should be included in the left view. If there are any missing faces, the region growth and screening are re-executed until the integrity requirements are met, and finally the left packaging region is formed.

[0031] Based on the standard view group feature point registration relationship obtained by the image registration module, the left packaging region is mapped to the corresponding right view. For each pixel coordinate in the left packaging region, its corresponding coordinate in the right view is calculated through the feature point registration relationship to form a mapped region. The boundary of the mapped region is corrected based on the edge gradient features of the right view. The boundary that coincides with the strong edge of the right view is retained, and the boundary that deviates from the strong edge is adjusted to the nearest strong edge position to ensure that the boundary of the right packaging region is consistent with the actual packaging edge of the right view. The topological relationship between the right packaging region and the left packaging region is checked. If they are inconsistent, the registration relationship is re-optimized and the mapping is performed until the consistency requirement is met. Finally, the right packaging region corresponding to the two view groups is obtained.

[0032] Furthermore, the point cloud generation module obtains the parallax of the packaging region in the dual-view group through a semi-global block matching algorithm, generates a single-view packaging point cloud by combining it with calibration parameters, and stitches the dual-view group point clouds according to the turntable rotation angle to obtain a complete packaging point cloud, including the following steps:

[0033] For the left and right packaging areas of the first standard view group and the left and right packaging areas of the second standard view group, parallax calculations are performed respectively. Based on the surface texture density of the drug packaging, the size of the rectangular matching window is set according to the experimentally calibrated dynamic adjustment rules. The window covers continuous pixel blocks within the packaging area.

[0034] For each pixel in the left packaging region, within the preset disparity range of the right packaging region, the matching cost is calculated by combining the sum of the absolute differences in gray levels with the gradient consistency constraint. The sum of the absolute differences in gray levels is the sum of the absolute differences in gray levels of the current pixel matching window in the left packaging region and the candidate pixel window of the same size in the right packaging region. The gradient consistency constraint is the deviation of the segmentation gradient direction of the corresponding pixels in the two windows. A preset penalty value is assigned according to the magnitude of the deviation. The final matching cost is the weighted sum of the two.

[0035] A semi-global block matching algorithm is employed to optimize the matching cost. Based on a multi-path cost aggregation strategy, the matching cost is aggregated along the horizontal, vertical, and diagonal directions of the image. The matching cost of the current pixel is equal to its own matching cost and the matching costs of its neighboring pixels multiplied by a distance weight. A winner-takes-all strategy is used on the aggregated global cost matrix, selecting the disparity value with the lowest cost as the initial disparity. Invalid disparities are eliminated through a consistency check between the left and right packaging regions. If the disparity of a pixel in the left packaging region deviates from the disparity of the corresponding pixel in the right packaging region by more than a preset threshold, it is marked as an invalid disparity. Interpolation is used to fill invalid disparity regions to ensure the integrity of the disparity map, ultimately yielding the disparity maps of the first-order view group and the second-order view group.

[0036] Combining the pre-calibrated binocular industrial camera parameters, the disparity map is converted into a 3D point cloud using the triangulation principle. The camera intrinsic and extrinsic parameters are obtained after offline calibration. The intrinsic parameters include the focal lengths of the left and right cameras and the coordinates of the principal point of the left camera. The extrinsic parameters include the baseline distance of the binocular industrial camera. For each effective disparity pixel in the left packaging area, its 3D spatial coordinates are calculated based on the triangulation principle. Based on the maximum / minimum coordinate range of the packaging in the preset coordinate system of the packaging standard design parameters, if the coordinates of a certain 3D point exceed the range, it is judged as a noise point and removed. The effective points are retained to form the packaging point cloud of the first view group and the packaging point cloud of the second view group.

[0037] Statistical filtering is performed on the packaging point clouds of the two view groups respectively. The neighborhood search radius and the threshold of the number of neighborhood points are set. For each point, the Euclidean distance of all other points in its neighborhood is calculated. The mean and standard deviation of these distances are solved. If the deviation of a point from the mean exceeds the preset standard deviation multiple, it is judged as a noise point and removed. The main structure of the point cloud is preserved. The voxel grid size is set to divide the point cloud space into several voxel grids of equal size. The centroid coordinates of all points in each voxel grid are calculated. The centroid coordinates are used to replace all the original points. The amount of point cloud data is reduced while preserving the geometric features of the packaging surface.

[0038] Based on the turntable rotation angle corresponding to the secondary standard view group, the point clouds of the two view groups are unified to the same coordinate system. The turntable rotation angle in the acquisition module is retrieved. Taking the coordinate system of the point cloud of the primary view group as the reference, the point cloud of the secondary view group is rotated through a rotation matrix. The rotation matrix is ​​constructed by rotating the turntable around the Z-axis of the coordinate system.

[0039] Registration is performed using the overlapping surface of the two view groups. Three-dimensional feature points of the overlapping surface are extracted from the point cloud of the transformed secondary view group and the point cloud of the primary view group respectively. The curvature of each point is calculated by the rate of change of the normal vector of the fitted plane between the point and its neighboring points. The points with the top half of the curvature values ​​are selected as salient feature points, forming the overlapping surface feature point set from the point cloud of the primary view group and the overlapping surface feature point set from the point cloud of the secondary view group.

[0040] Calculate the Euclidean distance between the two sets of feature points. For each point in the overlapping surface feature point set of the primary view group point cloud, find the point with the smallest distance in the overlapping surface feature point set of the secondary view group point cloud as a matching pair. Minimize the distance error of the matching point pair by iterative nearest point algorithm. Calculate the average distance error of the current matching point pair. Solve for the optimal rotation matrix and translation vector by singular value decomposition. Update the coordinates of the secondary view group point cloud. Repeat the iteration until the average distance error is less than a preset threshold. Merge the fine-tuned secondary view group point cloud with the primary view group point cloud, retaining all valid 3D points to form a preliminary complete packaged point cloud.

[0041] The preliminary complete packaging point cloud is optimized by calculating the Euclidean distance between adjacent 3D points in the overlapping area. When the distance between two points is less than a preset threshold, one of the points is retained and redundant and duplicate points are removed. The preliminary complete packaging point cloud is checked to see if it covers all the outer surfaces of the packaging, combined with the packaging topology prior. If there are missing points in a certain area, they are supplemented by interpolation of the point cloud features of the adjacent surfaces. The effective points around the missing area are used to fit the plane equation. The coordinates of the points in the missing area are calculated according to the plane equation, and finally the complete packaging point cloud is obtained.

[0042] Furthermore, the calculation and judgment module uses the Poisson surface reconstruction algorithm to perform 3D mesh reconstruction on the complete packaging point cloud, obtains the packaging model, detects surface irregularities and defects, determines its conformity, and initiates sorting, including the following steps:

[0043] Based on the point cloud of the complete packaging, a continuous and smooth 3D mesh model is constructed using the Poisson surface reconstruction algorithm. According to the point cloud density of the complete packaging and the detail accuracy requirements of the pharmaceutical packaging, the core parameters of Poisson reconstruction are set, including sampling density factor, reconstruction depth, and boundary weight.

[0044] For each 3D point in the complete packaging point cloud, a local normal vector is calculated based on its neighborhood point set, and the direction of the normal vector is unified through a global consistency optimization algorithm. Taking the spatial range of the complete packaging point cloud as the boundary, a 3D voxel mesh covering the point cloud space is constructed. Points with normal vectors are projected onto the 3D voxel mesh and the gradient contribution is calculated. The Poisson equation system is constructed and solved to obtain the 3D implicit function field. The zero isosurface of the implicit function is extracted to generate an initial triangular mesh. A continuous and smooth 3D mesh model is generated by the Laplace smoothing algorithm. This model completely preserves the outer surface geometry of the drug packaging.

[0045] The surface curvature is calculated based on a 3D mesh model, and a reference curvature is set. For each triangular mesh vertex in the 3D mesh model, a predetermined number of neighboring vertices are selected to construct a local neighborhood. The reference plane of this neighborhood is fitted using the least squares method. By calculating the distance deviation from the triangular mesh vertex to the reference plane and the rate of change of the normal vector of the vertices in the neighborhood, the average curvature value of the vertex is solved. Based on the preset shape parameters of the pharmaceutical packaging, a reference curvature is set for each surface of the 3D mesh model.

[0046] Based on the material characteristics and production process requirements of the pharmaceutical packaging, a curvature mutation threshold is set. All vertices of the 3D mesh model are traversed, and the absolute value of the difference between the actual average curvature of each vertex and the reference curvature at the corresponding position is calculated. If the absolute value of the difference exceeds the curvature mutation threshold, the vertex is determined to be a curvature mutation point and marked as a suspected defect point. The region growth algorithm is executed with the suspected defect point as the initial seed point. The adjacent vertices of the seed point are judged to meet the curvature mutation consistency and mesh topology continuity. If they meet the requirements, the vertex is included in the suspected defect region and the seed point set is updated. When the number of newly added vertices in a suspected defect region is less than the preset number threshold for three consecutive times, the growth stops and an independent set of suspected defect regions is formed.

[0047] By combining preset tolerance thresholds, suspected defect areas are verified to confirm whether they are genuine defects and to distinguish the defect type. Based on the quality standards for pharmaceutical packaging, preset tolerance thresholds are used for defect determination. These thresholds include the maximum permissible protrusion height, the maximum permissible indentation depth, and the maximum permissible area of ​​a single defect.

[0048] For each region in the suspected defect area set, the vertical distance from the vertex to the corresponding reference plane is calculated. When the distance is positive, the maximum value is taken as the convex height, and when the distance is negative, the maximum absolute value is taken as the concave depth. The actual area of ​​the region is obtained by calculating the total area of ​​the triangular mesh facets covered by the suspected defect area. The parameters of each region are compared with the tolerance threshold to screen out the real defects that exceed the standard and remove the false defects that do not exceed the standard. The real defects are divided into convex defects and concave defects according to the positive and negative distances. The location, type, height / depth and actual area of ​​each defect are recorded to form a defect information set. Based on the defect information set and the preset qualification standards, the qualification of drug packaging is judged and sorted in conjunction with the process.

[0049] Compared with the prior art, the significant advantages of this invention are:

[0050] 1. The polarization fusion SIFT algorithm is adopted. By extracting the proportion of polarization angle intervals in the neighborhood of pixels and the arithmetic mean of polarization degree, a polarization feature sub-vector is constructed. It is then dynamically weighted and fused with the SIFT initial descriptor for optimization, overcoming the limitations of traditional feature registration in resisting illumination and noise interference.

[0051] 2. A segmentation model combining topological priors and edge gradient constraints is proposed. The topological characteristics of packaging are quantified into parameters such as area, angle, and smoothness. The model is coupled with multi-scale edge gradient features of the entire image pixels to efficiently solve the problem of adapting packaging of multiple specifications. Attached Figure Description

[0052] Figure 1 This is a block diagram of an online drug packaging inspection system based on 3D reconstruction.

[0053] Figure 2 This is a flowchart of the polarization fusion SIFT feature registration algorithm in this invention;

[0054] Figure 3 This is a flowchart illustrating the segmentation process of topological priors and edge gradient constraints in this invention.

[0055] Figure 4 This is a flowchart of the three-dimensional reconstruction and defect detection process in this invention. Detailed Implementation

[0056] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0057] like Figure 1 As shown, this invention discloses an online inspection system for pharmaceutical packaging based on three-dimensional reconstruction, including an acquisition module, an image registration module, a region segmentation module, a point cloud generation module, and a calculation and judgment module.

[0058] The acquisition module controls the rotation of the turntable so that the binocular industrial camera in a fixed direction takes two pictures of the drug packaging, obtaining a primary view group and a secondary view group. Each view group includes a left view and a right view. The two view groups cover all surfaces of the drug packaging and there is at least one overlapping surface.

[0059] The image registration module filters and denoises the dual-view group and obtains the polarization parameters of the binocular industrial camera. Using the polarization fusion SIFT algorithm, it extracts the texture features of the two views in each standard view group after denoising and corrects them in combination with the polarization parameters to obtain the left and right corrected feature points of each standard view group. The random sampling consensus algorithm based on particle swarm optimization determines the feature point registration relationship of the two views in each standard view group.

[0060] The region segmentation module utilizes an image segmentation model that integrates packaging topology priors and edge gradient constraints. By embedding packaging topology priors as geometric constraints for region selection and coupling edge gradient features to enhance packaging boundary localization, it identifies the left packaging region of the left view in each standard view group and maps the corresponding right packaging region of the right view based on the feature point registration relationship of the standard view group.

[0061] The point cloud generation module obtains the parallax between the left and right packaging areas within the same standard view group through a semi-global block matching algorithm. It then generates the packaging point cloud for each view by combining the pre-calibrated baseline distance and camera focal length. Based on the turntable rotation angle corresponding to the secondary standard view group, the packaging point clouds of the two standard view groups are stitched together to obtain the complete packaging point cloud.

[0062] The calculation and judgment module performs three-dimensional mesh reconstruction of the complete packaging point cloud based on the Poisson surface reconstruction algorithm to obtain the packaging model. It then performs surface unevenness defect detection on the packaging model by comparing the curvature mutation analysis with the preset tolerance threshold, generates a defect information set, and makes a decision on whether the packaging is qualified for linkage sorting.

[0063] Furthermore, the pharmaceutical packaging is transported to the CNC turntable at a fixed rate on the production line. When the pharmaceutical packaging is located on the CNC turntable, it is in its initial pose. At this time, a binocular industrial camera fixed on the side of the production line simultaneously captures the left and right views to form a primary view group. When the turntable rotates a certain angle, the pharmaceutical packaging presents a new observation angle, and the binocular industrial camera triggers synchronous acquisition again to obtain a secondary view group. The two view groups can completely cover all the outer surfaces of the pharmaceutical packaging, and there is at least one overlapping surface under adjacent shooting angles, which is used for subsequent point cloud stitching and positioning.

[0064] like Figure 2 As shown, further, the polarization fusion SIFT algorithm is used to extract and correct the texture features of the left and right views respectively, and feature point matching is performed in combination with the polarization parameters of the binocular industrial camera to establish the registration relationship between the two views, including the following steps:

[0065] The polarization parameters, including the transmission axis angle of the polarizer of the left camera and the transmission axis angle of the polarizer of the right camera, are obtained through the built-in polarization sensor of the binocular industrial camera. Based on the first left view and the first right view of the first view group, and the second left view and the second right view of the second view group, the polarization angle and polarization degree corresponding to the coordinate position of each pixel in each view are obtained. Gaussian filtering is performed on the first left view, the first right view, the second left view, and the second right view to suppress high-frequency noise in the image and preserve the texture edge features of the pharmaceutical packaging surface. Finally, the first standard left view, the first standard right view, the second standard left view, and the second standard right view are obtained.

[0066] Based on the standard images of the first-view group and the second-view group, the SIFT algorithm is used to extract initial feature points, including: constructing a multi-scale image pyramid for each standard image and calculating the difference Gaussian response of adjacent scale layers, i.e., the difference between the Gaussian blurred image of the later scale layer and the Gaussian blurred image of the previous scale layer, generating a difference Gaussian scale space for locating potential feature points; for local extrema in the difference Gaussian scale space, low-contrast points and edge response points are eliminated by Taylor expansion fitting, where low-contrast points refer to points whose absolute response value is less than the low-contrast threshold, and edge response points are determined by calculating the ratio of the trace of the Hessian matrix at the extrema point to the determinant. When this ratio is greater than the ratio of the square of the sum of the edge threshold plus 1 to the edge threshold, it is determined to be an edge response point. The low-contrast threshold and the edge threshold are preset values ​​calibrated through experiments. After screening, the first-left initial feature point set, the first-right initial feature point set, the second-left initial feature point set, and the second-right initial feature point set are obtained respectively. Each initial feature point set contains several feature points, and each feature point contains coordinates and scale information.

[0067] Each feature point in the initial feature point set is assigned an orientation value to make the feature points rotation invariant. Taking the feature point as the center, the pixel neighborhood within a preset multiple of its scale is taken. Based on the gray-level difference in the horizontal and vertical directions of the pixel, the gradient magnitude and gradient direction of each pixel in the pixel neighborhood are calculated. The gradient directions in the entire range are weighted and statistically analyzed to generate a multi-interval orientation histogram. The direction corresponding to the peak of the histogram is taken as the main direction of the feature point. If there are other peaks that exceed the fixed proportion of the main peak, they are added as auxiliary directions.

[0068] Based on the direction assignment results, initial descriptors are generated for each initial feature point set. Taking the main direction of the feature point as the reference, its pixel neighborhood is divided into several sub-regions. For each sub-region, the gradient histogram of all directions is calculated at a preset angle interval to obtain the vector of the corresponding dimension. The vectors of all sub-regions are concatenated to form the initial descriptor. For each initial feature point set, a first-order left initial descriptor set, a first-order right initial descriptor set, a second-order left initial descriptor set, and a second-order right initial descriptor set are obtained respectively.

[0069] The initial feature point set and the corresponding initial descriptor set are fused with polarization features to generate corrected feature points and corrected descriptors. For each feature point in each initial feature point set, the pixel neighborhood corresponding to the initial descriptor set is taken. The polarization feature sub-vector is calculated based on the polarization angle and polarization degree in the pixel neighborhood. Starting from the horizontal direction, the polarization angle range corresponding to half a circle in the clockwise direction is divided into several intervals. The proportion of pixels in each interval in the neighborhood is counted and arranged in the interval division order to form the interval proportion vector of the corresponding dimension. At the same time, the arithmetic mean of the polarization degree in the neighborhood is calculated to obtain a 1-dimensional feature. The above vector and the arithmetic mean are concatenated to form the polarization feature sub-vector.

[0070] The initial descriptor is weighted and fused with the polarization feature sub-vectors of the corresponding feature points to obtain the modified descriptor. The fusion weights include a first weight and a second weight, and the first weight plus the second weight equals 1. The second weight is dynamically adjusted based on the arithmetic mean of the polarization degree in the polarization feature sub-vectors. Specifically, it is the ratio of the arithmetic mean of the polarization degree to the arithmetic mean of the polarization degree plus an adjustment parameter. The adjustment parameter is a preset value calibrated by experiments to avoid weight imbalance when the arithmetic mean of the polarization degree is too small. Each initial feature point is associated with its corresponding modified descriptor to form a first left modified feature point set, a first right modified feature point set, a second left modified feature point set, and a second right modified feature point set, respectively.

[0071] Calculate the similarity of each obtained modified feature point set, and select initial matching pairs. For each feature point in the first left modified feature point set, traverse all feature points in the first right modified feature point set, and calculate the Euclidean distance between the modified descriptors of the two feature points, which is the square root of the sum of the squares of the differences of the corresponding dimensional components. For each first left modified feature point, select the two first right modified feature points with the smallest Euclidean distance, and calculate the ratio of these two minimum distances, i.e., the ratio of the first minimum distance to the second minimum distance. If this ratio is less than a threshold... If the left-corrected feature point and the corresponding first right-corrected feature point are determined to be the initial matching pair, after traversing all the left-corrected feature points, the first initial matching pair set is obtained. The same operation is performed on the second left-corrected feature point set and the second right-corrected feature point set to obtain the second initial matching pair set.

[0072] A random sampling consensus algorithm based on particle swarm optimization is used to optimize the first and second initial matching pairs, eliminate mismatches, and determine the final feature point registration relationship. It is assumed that the left and right views of the same view group satisfy the perspective transformation relationship, and the perspective transformation is described by the homography matrix that characterizes the coordinate mapping relationship of the feature points of the two views.

[0073] Build size For a particle swarm, each particle is assigned a set of randomly selected minimum non-collinear matching pairs. There are 4 matching pairs in total, which meet the minimum sample size for solving the homography matrix. This set of matching pairs is defined as the initial sample set of the particles. The initial homography matrix of the corresponding particles is solved based on the initial sample set.

[0074] For each particle's initial homography matrix, the reprojection error of all initial matching pairs after transformation by this matrix is ​​calculated. The reprojection error is the Euclidean distance between the actual coordinates of the feature points in the right view and the predicted coordinates of the feature points in the left view obtained after homography matrix transformation. Matching pairs with reprojection errors less than a preset error threshold are determined as inliers, and the number of inliers is used to determine the inlier count. (1 − average reprojection error of interior points) is used as the fitness value of a particle. The more interior points there are and the smaller the average reprojection error of interior points, the higher the fitness value of the particle.

[0075] Record the local optimal position of each particle, which is the sample set corresponding to the highest historical fitness value of that particle. Record the global optimal position of the entire particle swarm, which is the sample set corresponding to the highest historical fitness value of all particles. Adjust the particle speed according to the particle's current speed, inertia weight, local optimal guiding force, and global optimal guiding force. The inertia weight maintains the particle's current search direction. The local optimal guiding force prompts the particle to move closer to its own high-quality sample set. The global optimal guiding force prompts the particle to move closer to the high-quality sample set of the swarm. Update the particle's sample set according to the adjusted speed. Prioritize retaining high-frequency matching pairs in the global optimal position and local optimal position sample sets. High-frequency matching pairs are regarded as potential high-quality interior points. Replace matching pairs in the sample set that have low frequency of occurrence and large reprojection error. Resolve the homography matrix corresponding to the particle based on the updated sample set.

[0076] Repeat the process until the number of iterations reaches the preset maximum, or the particle fitness value corresponding to the global optimal position remains unchanged for multiple consecutive rounds. At this point, convergence is determined, and the homography matrix corresponding to the global optimal position is selected as the final transformation model. Interior points with reprojection errors less than a preset error threshold in all initial matching pairs are retained to form the final matching pair set. Based on the one-to-one correspondence between the left-view corrected feature points and the right-view corrected feature points in the final matching pair set, the feature point registration relationship between the left and right views in the first-view group and the second-view group is constructed respectively. Through the swarm optimization mechanism of particle swarm optimization, the blindness of traditional random consistency sampling is reduced, the optimal interior point set and homography matrix are located more efficiently, and the accuracy and convergence efficiency of feature point registration are improved.

[0077] like Figure 3 As shown, further, by fusing packaging topological priors and edge gradient constraints into an image segmentation model, the left packaging region of the left view of each standard view group is identified, and the right packaging region is obtained based on the feature point registration relationship, including the following steps:

[0078] Based on the standard design parameters of pharmaceutical packaging, a packaging topological prior is constructed, including the overall spatial topological structure of the packaging, the geometric feature constraints of each face, and the projection morphological features under typical views.

[0079] The packaging topological prior is transformed into a quantifiable set of parameters, including the minimum area threshold of the surface region, the maximum area threshold of the surface region, the angular constraints of adjacent surfaces in the image, and the smoothness threshold of the surface boundary, all calculated based on the actual packaging size and image resolution.

[0080] Edge gradient features are extracted from the first and second standard left views respectively. Gradient operators are used to perform multi-scale gradient calculations on the first and second standard left views to obtain the segmentation gradient magnitude and segmentation gradient direction for each pixel. The segmentation gradient magnitude represents the edge intensity and is used to distinguish the packaging boundary from the background area. The segmentation gradient direction represents the edge direction and is used to determine the boundary continuity. The scale parameter is dynamically adjusted according to the texture of the packaging surface and according to experimentally calibrated rules. The segmentation gradient magnitude and segmentation gradient direction are different from the gradient magnitude and gradient direction used to describe the local texture of the feature point in the feature point registration. The former is for all pixels in the image and focuses on the intensity representation of the overall edge of the packaging, serving the region boundary localization. The latter is only for pixels in the neighborhood of the feature point and focuses on the intensity of local texture changes, serving the feature point direction assignment and descriptor generation.

[0081] Non-maximum suppression is applied to the segmentation gradient magnitude to retain strong gradient pixels at the edge center and remove weak gradient pixels on both sides of the edge to achieve edge refinement. A dual threshold method is used to filter continuous edge contours. Strong edges are retained through a high threshold and weak edges are connected through a low threshold to finally form an initial edge map.

[0082] Based on the geometric constraints and edge gradient features of the packaging topology prior, the left view is initially segmented and candidate packaging regions are selected. A region growing algorithm is used, with strong edge pixels in the initial edge map as seed points. Regions are grown according to the consistency of the segmentation gradient direction and gray-level similarity to generate multiple connected regions. The generated connected regions are matched and verified with the packaging topology prior, including area verification, shape verification, and connectivity verification. Among them, area verification removes regions with areas smaller than the minimum area threshold or larger than the maximum area threshold. Shape verification extracts the region shape through the minimum bounding rectangle or convex hull algorithm and removes regions whose projection shape deviates from the corresponding surface shape in the topology prior by more than a preset range. Connectivity verification checks whether the adjacent regions of the retained regions satisfy the connectivity relationship in the topology prior and removes isolated regions that do not meet the connectivity relationship. After screening, a set of candidate packaging regions for the left view is obtained.

[0083] The candidate packaging region set is refined to ensure the accuracy and integrity of the region boundaries. In the boundary refinement, the boundary of the candidate region is located at the sub-pixel level based on the gradient direction in the edge gradient features, and the boundary offset is corrected to make it accurately aligned with the actual packaging edge. Then, morphological operations are used to remove small hole noise in the candidate region, and small noise regions with an area ratio less than the preset ratio are removed by connected component analysis. Finally, combined with the constraint of covering all faces in the packaging topology prior, it is checked whether the candidate region set covers all packaging faces that should be included in the left view. If there are any missing faces, the region growth and screening operations are re-executed until the integrity requirements are met, and finally the left packaging region of the first standard left view and the second standard left view are formed.

[0084] Based on the coordinates of each pixel in the left packaging area, the corresponding coordinates in the right view are calculated through the registration relationship of the feature points of the two standard view groups, forming the mapping area in the right view. Since the registration error may cause jagged edges or offsets in the boundary of the mapping area, the boundary of the mapping area is corrected based on the edge gradient features of the right view. The boundary in the mapping area that coincides with the strong edge of the right view is retained, and the boundary that deviates from the strong edge is adjusted to the nearest strong edge position to ensure that the boundary of the right packaging area is consistent with the actual packaging edge of the right view. The topological relationship between the right packaging area and the left packaging area is checked. If they are inconsistent, the registration relationship is re-optimized and the mapping is performed until the consistency requirement is met, and finally the right packaging area corresponding to the two view groups is obtained.

[0085] Furthermore, the parallax of the packaging area in the dual-view group is obtained through a semi-global block matching algorithm. Combined with calibration parameters, a single-view packaging point cloud is generated. The dual-view group point clouds are then stitched together based on the turntable rotation angle to obtain a complete packaging point cloud. This process includes the following steps:

[0086] For the left wrapper area of ​​a standard left view in a standard view group. The right packaging area of ​​a standard right view In the secondary standard view group, the left packaging area of ​​the secondary standard left view The right packaging area of ​​the second standard right view Parallax calculations are performed separately. Based on the surface texture density of the drug packaging, the matching window size is set according to the experimentally calibrated dynamic adjustment rules. The window adopts a rectangular structure and covers continuous pixel blocks within the packaging area. For each pixel in the left packaging area, within the preset parallax range of the right packaging area, the matching cost is calculated by combining the sum of the absolute grayscale differences with gradient consistency constraints to reduce the probability of mismatches in textured blurry areas. The preset parallax range is determined based on camera calibration parameters and the actual size of the packaging. The sum of the absolute grayscale differences is the sum of the absolute differences of the pixel-by-pixel grayscale values ​​of the matching window of the current pixel in the left packaging area and the candidate pixel window of the same size in the right packaging area. The smaller the sum, the higher the grayscale similarity. The gradient consistency constraint is the deviation of the segmentation gradient direction of the corresponding pixels in the two windows. A preset penalty value is assigned according to the magnitude of the deviation. The final matching cost is the weighted sum of the sum of the absolute grayscale differences and the gradient constraint penalty value.

[0087] A semi-global block matching algorithm is employed to optimize the matching cost and generate an effective disparity map. Based on a multi-path cost aggregation strategy, the matching cost is aggregated along the horizontal, vertical, and diagonal directions of the image to ensure that the global cost comprehensively reflects the reliability of pixel matching across different paths. The matching cost of the current pixel is composed of its original matching cost, the minimum of the matching costs of adjacent pixels under the same and adjacent disparities, multiplied by their corresponding distance weights. This balances disparity continuity and matching accuracy. The distance weights are preset based on the actual distance between pixels along the path and the smoothness of the drug packaging surface texture, and their specific values ​​are determined through experimental calibration. Global cost optimization balances local matching ambiguities and improves disparity continuity. A winner-takes-all strategy is adopted for the aggregated global cost matrix, selecting the disparity value with the lowest cost as the initial disparity. Invalid disparities are eliminated through a consistency check between the left and right packaging regions. That is, the disparity of a pixel in the left packaging region is... At that time, the pixel parallax at the corresponding position in the right packaging area should be If the deviation between the two exceeds a preset threshold, it is marked as invalid disparity. Interpolation is used to fill the invalid disparity area to ensure the integrity of the disparity map, and finally the disparity map of the first-order view group is obtained. disparity diagram of secondary view group ;

[0088] Using pre-calibrated binocular industrial camera parameters, the disparity map is converted into a 3D point cloud through triangulation to obtain the camera's intrinsic and extrinsic parameters after offline calibration. The intrinsic parameters include the focal lengths of the left and right cameras. Left camera principal point coordinates That is, the pixel coordinates of the image coordinate system origin in the left view, and the extrinsic parameters include the baseline distance of the binocular industrial camera. This refers to the length of the line connecting the optical centers of the left and right cameras. All parameters are fixed values ​​stored after offline calibration to ensure the accuracy of 3D reconstruction. For each effective disparity pixel in the left packaging area, its 3D spatial coordinates are calculated based on the principle of triangulation. The disparity is then determined according to the corresponding disparity of the pixels in the disparity map. Through geometric relationships The depth is obtained by the vertical distance from the pixel to the optical center of the left camera. Among them, depth equal to baseline distance Multiply by focal length Then divide by parallax This establishes a mapping relationship between parallax and spatial depth. A coordinate system is established with the optical center of the left camera as the origin, combined with the two-dimensional coordinates of pixels in the left view. ,focal length and depth Solving for three-dimensional spatial coordinates ,in, equal to pixel horizontal coordinate Subtract the horizontal coordinates of the left camera principal point Multiply by depth Divide by focal length , equal to pixel vertical coordinate Subtract the vertical coordinates of the left camera principal point Multiply by depth Divide by focal length , The coordinates are the depth values ​​calculated above, ensuring consistency between the coordinates and the actual spatial position of the packaging. Based on standard packaging design parameters, the maximum / minimum coordinate range of the packaging is preset in the coordinate system. If a certain three-dimensional point... , or Points with coordinates outside this range are identified as noise points and removed. Valid points within a reasonable spatial range are retained to form a packaged point cloud for a primary view group. Packaging point cloud of secondary view group Each point cloud contains several three-dimensional points, and each three-dimensional point is associated with corresponding spatial coordinate information;

[0089] right and Statistical filtering is performed separately to improve the accuracy and efficiency of subsequent stitching. A neighborhood search radius and a threshold for the number of neighboring points are set. For each 3D point, all other points within its neighborhood are searched, and the Euclidean distance between the 3D point and these neighboring points is calculated. The Euclidean distance is the square root of the sum of the squares of the differences between the three coordinates of two points. The mean and standard deviation of these distances are calculated. If the deviation of a point from the mean exceeds a preset standard deviation multiple, it is identified as a noise point and removed, preserving the main structure of the point cloud. A voxel grid size is set, dividing the point cloud space into several equally sized voxel grids. For all points within each voxel grid, the centroid coordinates are calculated. Average coordinates Average coordinates The average coordinates are used to replace all the original points with the centroid coordinates, which reduces the amount of point cloud data and improves the subsequent stitching speed while preserving the geometric features of the packaging surface.

[0090] Based on the turntable rotation angle during the capture of the two standard view groups, the point clouds of the two view groups are unified to the same coordinate system. The turntable rotation angle in the acquisition module is then retrieved, which is the rotation angle from the initial pose to the second capture pose. This angle is a precise value recorded by the turntable CNC system, used for point cloud coordinate transformation, to group point clouds in a single view. Using the coordinate system as a reference, where the origin is the optical center of the left camera, the point cloud of the secondary view group is... Perform a rotation transformation, converting the original coordinates to rotated coordinates using a rotation matrix. The rotation matrix is ​​constructed as follows: the turntable rotates around the coordinate system... If the axis is rotated, then the first row of elements in the rotation matrix is... The second line The third line ,in For angle cosine value, For angle The sine value, for Coordinates of each 3D point The new coordinates are calculated by multiplying the rotation matrix and the coordinate vector. ,Right now , , ,make coordinate system and To maintain consistency and ensure spatial pose matching between the two point cloud datasets, registration is performed using the overlapping surface of the two view groups. First, the features of the overlapping surface are extracted from the transformed data. and The three-dimensional feature points of the overlapping surface are extracted. The curvature of each point is calculated by the rate of change of the normal vector of the fitted plane between the point and its neighbors. The greater the curvature, the more likely the point is to be an edge or corner. The points with the highest curvature values ​​are selected as significant feature points, forming a series of features from... Overlapping surface feature point set and from of Calculate the Euclidean distance between the two sets of feature points. Each point in, The algorithm finds the points with the smallest distance as matching pairs, and minimizes the distance error of the matching point pairs through an iterative nearest-neighbor algorithm. During the iteration process, the average distance error of the current matching point pairs is calculated, and the optimal rotation matrix and translation vector are solved by singular value decomposition to update the algorithm. The coordinates are iterated repeatedly until the average distance error is less than a preset threshold, thereby eliminating minor deviations caused by rotation transformation and ensuring seamless connection of overlapping surfaces. The finely adjusted coordinates are then used... and The points are merged, retaining all valid 3D points, to form a preliminary complete packaged point cloud. ;

[0091] For the initial integration To optimize and ensure the point cloud quality meets subsequent inspection requirements, the first step is to... For 3D points in the overlapping region, calculate the Euclidean distance between adjacent points. If the distance between two points is less than a preset threshold, retain one of the points and remove redundant and duplicate points to reduce data redundancy. Simultaneously, combine this with packaging topology priors to check... Whether the entire outer surface of the packaging is covered is determined. If a point cloud is missing in a certain area, it is supplemented by interpolation using the point cloud features of adjacent surfaces. Valid points around the missing area are used to fit a plane equation, and the coordinates of points within the missing area are calculated based on the plane equation. This ensures that the complete packaging point cloud covers all the outer surfaces of the drug packaging, ultimately yielding a complete packaging point cloud. .

[0092] like Figure 4 As shown, further, a Poisson surface reconstruction algorithm is used to reconstruct a 3D mesh of the complete packaging point cloud to obtain a packaging model. Surface unevenness defects are detected by curvature abrupt change analysis combined with a preset tolerance threshold, and the pass / fail status is determined and linked to sorting. This includes the following steps:

[0093] Based on complete packaging point cloud A continuous and smooth 3D mesh model is constructed using the Poisson surface reconstruction algorithm, based on... To meet the point cloud density and detail accuracy requirements of pharmaceutical packaging, core parameters for Poisson reconstruction were set, including sampling density factor, reconstruction depth, and boundary weights, to ensure that the reconstructed mesh matches the point cloud details without generating redundancy. For each 3D point, a local normal vector is calculated based on its neighborhood point set. The direction of the normal vector is then unified through a global consistency optimization algorithm, providing accurate geometric guidance for surface reconstruction and thus fully packaging the point cloud. The spatial extent is defined by a 3D voxel mesh covering the point cloud space. Points with normal vectors are projected onto the 3D voxel mesh, and their gradient contributions are calculated. The Poisson equations are constructed and solved to obtain the 3D implicit function field. The zero isosurface of the implicit function is extracted to generate an initial triangular mesh. After applying the Laplace smoothing algorithm, a continuous and smooth 3D mesh model is generated. The model fully preserves the outer surface geometry of the drug packaging;

[0094] Based on 3D mesh model The surface curvature is calculated and a reference curvature is set to provide a basis for defect detection. First, for the three-dimensional mesh model... For each vertex of a triangular mesh, a local neighborhood is constructed by selecting a preset number of neighboring vertices. The reference plane of the neighborhood is fitted using the least squares method. By calculating the distance deviation from the triangular mesh vertex to the reference plane and the rate of change of the normal vector of the vertex in the neighborhood, the actual average curvature of the vertex is solved. The larger the average curvature, the more drastic the surface morphology change at that location. If it is close to 0, it indicates that the surface tends to be flat.

[0095] Based on the preset shape parameters of the drug packaging, a three-dimensional mesh model is generated. Each surface is given a reference curvature. If the packaging is a planar structure of cuboid or cube, the reference curvature of all outer surfaces is set to 0, that is, the ideal plane has no curvature change. If the packaging contains curved surface structures of cylindrical or spherical surfaces, the reference curvature is set to the theoretical curvature of the corresponding ideal surface.

[0096] Based on the material characteristics and manufacturing process requirements of pharmaceutical packaging, a curvature mutation threshold is set, and the packaging model is traversed. For all vertices, calculate the absolute value of the difference between the actual average curvature of each vertex and the reference curvature at the corresponding position. If the absolute value of this difference is greater than the curvature abrupt change threshold, the vertex is determined to be a curvature abrupt change point and marked as a suspected defect point. Using the suspected defect point as the initial seed point, execute the region growing algorithm. For the adjacent vertices of the seed point, determine whether they satisfy the curvature abrupt change consistency and mesh topological continuity. If they satisfy these conditions, the vertex is included in the suspected defect region, and the seed point set is updated. When the number of newly added vertices in a suspected defect region is less than a preset threshold for three consecutive times, growth stops, and an independent set of suspected defect regions is formed. ;

[0097] By combining preset tolerance thresholds, suspected defect areas are verified to confirm whether they are real defects and to distinguish the defect types. First, based on the quality standards of pharmaceutical packaging, preset tolerance thresholds for defect judgment are set, including the maximum allowable protrusion height, the maximum allowable indentation depth, and the maximum allowable area of ​​a single defect. For each area in the set of suspected defect areas, the vertical distance from the vertex to the corresponding reference plane is calculated. When the distance is positive, the maximum value is taken as the protrusion height, and when the distance is negative, the maximum absolute value is taken as the indentation depth. The total area of ​​the triangular mesh facets covered by the suspected defect area is calculated to obtain the actual area of ​​the area.

[0098] The parameters of each region are compared with the tolerance threshold to screen out the real defects that exceed the standard and eliminate false defects that do not exceed the standard. The real defects are divided into raised defects and sunken defects according to the positive and negative distance. The location, type, height / depth and actual area of ​​each defect are recorded to form a defect information set. Based on the defect information set and combined with the preset qualification standards, the qualification of drug packaging is judged and sorted in conjunction with the process.

[0099] This invention discloses an online inspection system for pharmaceutical packaging based on 3D reconstruction. It utilizes a CNC rotary table and a binocular industrial camera to acquire a set of two views covering the entire surface of the packaging, including overlapping areas. After Gaussian filtering for noise reduction, the system employs a polarization fusion SIFT algorithm to extract initial feature points from each view. Polarization feature vectors are generated by calculating the arithmetic mean of the polarization angle intervals and polarization degrees within the pixel neighborhood. These vectors are then weighted and fused with the initial descriptor to obtain a corrected feature point set. Initial matching pairs are selected using Euclidean distance, and the feature point registration relationship is optimized using a particle swarm optimization random sampling consensus algorithm. Based on packaging topological prior quantization constraints and edge gradient features, the system accurately identifies the left packaging region and maps and corrects it to obtain the right packaging region. Finally, it uses semi-global block matching... The algorithm, combined with pre-calibrated camera focal length and baseline distance parameters, transforms the data into a 3D point cloud using triangulation principles. After statistical filtering and voxel downsampling optimization, a rotation transformation is performed based on the turntable rotation angle. Then, feature point extraction of overlapping surfaces and fine-tuning through iterative nearest-point algorithms are used to stitch together a complete packaging point cloud. Based on the complete point cloud, local normal vectors are calculated and their directions are unified. A 3D mesh model is generated through Poisson surface reconstruction. Through curvature mutation analysis and tolerance threshold quantification, accurate identification and parameter quantification of concave and convex defects are achieved. This effectively compensates for the shortcomings of traditional detection methods, such as incomplete coverage and high false positive rates. It meets the high precision, high efficiency, and high adaptability requirements of industrial production for pharmaceutical packaging inspection, providing a reliable technical solution for pharmaceutical packaging quality control.

[0100] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A pharmaceutical packaging online inspection system based on three-dimensional reconstruction, characterized in that, It includes an acquisition module, an image registration module, a region segmentation module, a point cloud generation module, and a calculation and decision module. The acquisition module controls the turntable to rotate so that the binocular industrial camera can capture images of the drug packaging, obtaining a set of dual views covering the entire surface of the drug packaging and including overlapping surfaces; The image registration module filters and denoises the dual-view group and obtains polarization parameters. It then uses the polarization fusion SIFT algorithm to extract texture features and corrects them in conjunction with the polarization parameters to obtain corrected feature points. Finally, it determines the registration relationship of the feature points based on the particle swarm optimization random sampling consensus algorithm. Specifically, the polarization fusion SIFT algorithm obtains polarization parameters through the built-in polarization sensor of the binocular industrial camera, extracts the polarization angle interval ratio of the pixel neighborhood and the arithmetic mean of the polarization degree to construct a polarization feature sub-vector, and fuses it with the initial descriptor extracted by the SIFT algorithm according to dynamic weights to generate a corrected descriptor. The initial feature points are associated with the corrected descriptor to form corrected feature points, which participate in similarity calculation. After initial matching pairs are selected, they are input into the particle swarm optimization random sampling consensus algorithm. The reprojection error corresponding to the corrected descriptor is calculated to select inliers to determine the registration relationship of the feature points. The region segmentation module locates the left packaging region based on a segmentation model that integrates packaging topology priors and edge gradient constraints, and maps the right packaging region according to the registration relationship. Here, the packaging topology prior refers to a set of quantifiable parameters transformed from the standard design parameters of pharmaceutical packaging, including the minimum and maximum area thresholds of the surface region, the angle constraints of adjacent surfaces, and the surface boundary smoothness threshold. Using these topology prior parameters as geometric constraints, multi-scale edge gradient features are coupled, and the segmentation model is constructed through area, shape, and connectivity verification. The point cloud generation module obtains parallax through a semi-global block matching algorithm, and combines it with camera calibration parameters and turntable rotation angle to obtain a complete packaged point cloud; The calculation and judgment module obtains a three-dimensional mesh model through Poisson surface reconstruction, uses curvature mutation analysis and tolerance threshold comparison to detect defects, generates a defect information set, and links with the sorting mechanism to determine the pass / fail status.

2. The online drug packaging inspection system based on three-dimensional reconstruction as described in claim 1, characterized in that, The image registration module acquires polarization parameters, performs Gaussian filtering on each view in the dual-view group to obtain a standard view, uses the SIFT algorithm to extract initial feature points for each standard view and assigns orientation values ​​to generate initial descriptors, constructs polarization feature sub-vectors corresponding to each initial feature point, and weights and fuses the initial descriptors and polarization feature sub-vectors to obtain a corrected descriptor. The initial feature points and the corrected descriptors are associated to form a corrected feature point set, and the similarity of the corrected feature point set is calculated to screen initial matching pairs. The initial matching pairs are optimized using a particle swarm optimization random sampling consensus algorithm, the reprojection error of the initial matching pairs is calculated, inliers with reprojection errors less than a preset threshold are retained, and the homography matrix with the most inliers is selected as the optimal transformation model to finally determine the feature point registration relationship.

3. The online drug packaging inspection system based on three-dimensional reconstruction as described in claim 1, characterized in that, The region segmentation module constructs a packaging topology prior based on the standard design parameters of drug packaging and transforms it into a set of quantifiable parameters. For the standard left view, the gradient operator is used to calculate the multi-scale segmentation gradient magnitude and direction. Non-maximum suppression and double thresholding are applied to the segmentation gradient magnitude to obtain the initial edge map. A region growing algorithm is used to generate connected regions with strong edge pixels in the initial edge map as seed points. The connected regions are verified and screened in combination with the packaging topology prior parameters to obtain a set of candidate packaging regions. The set of candidate packaging regions is refined to obtain the left packaging region. The right packaging region is obtained by mapping and correcting according to the feature point registration relationship.

4. The online drug packaging inspection system based on three-dimensional reconstruction as described in claim 1, characterized in that, The point cloud generation module sets the matching window size for the left and right packaging regions of each view group, calculates the matching cost of pixels in the left packaging region within the preset disparity range of the right packaging region, optimizes the matching cost using a semi-global block matching algorithm to generate a disparity map, and removes invalid disparities by checking the consistency between the left and right packaging regions. Combining pre-calibrated camera intrinsic and extrinsic parameters, the disparity map is converted into a packaging point cloud using the triangulation principle. Statistical filtering and voxel downsampling are performed on the packaging point cloud. The rotation angle of the turntable is retrieved to perform rotation transformation on the packaging point cloud of the two view groups. The point cloud is fine-tuned using overlapping surface feature point extraction and iterative nearest point algorithm. Finally, the packaging point clouds of the two view groups are fused to obtain a complete packaging point cloud.

5. The online drug packaging inspection system based on three-dimensional reconstruction as described in claim 2, characterized in that, The image registration module calculates the interval proportion of polarization angle and the arithmetic mean of polarization degree for the pixel neighborhood corresponding to each initial feature point. The interval proportions are arranged in the order of interval division to form an interval proportion vector, which is then concatenated with the arithmetic mean of polarization degree to form a polarization feature sub-vector. Based on the arithmetic mean of polarization degree and the experimentally calibrated adjustment parameters, the fusion weight of the initial descriptor and the polarization feature sub-vector is determined. The initial descriptor and the polarization feature sub-vector are then fused according to the fusion weight to obtain the corrected descriptor.

6. The online pharmaceutical packaging inspection system based on three-dimensional reconstruction as described in claim 3, characterized in that, The quantifiable parameter set constructed by the region segmentation module includes the minimum area threshold, maximum area threshold, adjacent face angle constraints, and face boundary smoothness threshold, all calculated based on the actual size of the drug packaging and the image resolution. The verification of connected regions includes area verification, morphological verification, and connectivity verification. Area verification removes regions whose area exceeds the threshold range, morphological verification removes regions whose morphological deviation from the topological prior projection exceeds the range, and connectivity verification removes isolated regions that do not conform to the topological prior connectivity relationship.

7. The online inspection system for pharmaceutical packaging based on three-dimensional reconstruction as described in claim 4, characterized in that, The point cloud generation module sets the neighborhood search radius and the threshold for the number of neighborhood points. It searches for neighborhood points for each point in the packaged point cloud and calculates the Euclidean distance. It solves for the mean and standard deviation of the distance, calculates the deviation of each point in the packaged point cloud from the mean, and removes noise points whose deviation exceeds a preset standard deviation multiple. It sets the voxel grid size to divide the point cloud space into voxel grids, calculates the centroid coordinates of the points in each voxel grid, and replaces the original points in the voxel grid with the centroid coordinates to reduce the amount of point cloud data.

8. The online inspection system for pharmaceutical packaging based on three-dimensional reconstruction as described in claim 1, characterized in that, The calculation and judgment module sets Poisson reconstruction parameters based on the density of the complete packaging point cloud and the accuracy requirements of the drug packaging details. It calculates the local normal vector of each 3D point in the complete packaging point cloud and unifies the normal vector direction through a global consistency optimization algorithm. It constructs a 3D voxel mesh covering the point cloud space, projects the points with normal vectors onto the 3D voxel mesh and calculates the gradient contribution, constructs the Poisson equation system and solves it to obtain the 3D implicit function field, extracts the zero isosurface of the implicit function to generate the initial triangular mesh, and uses the Laplace smoothing algorithm to generate the 3D mesh model.

9. The online inspection system for pharmaceutical packaging based on three-dimensional reconstruction as described in claim 8, characterized in that, The calculation and judgment module selects the adjacent vertices of each triangular mesh vertex in the 3D mesh model to construct a local neighborhood. It uses the least squares method to fit the reference plane of the local neighborhood, calculates the distance deviation from the triangular mesh vertex to the reference plane and the rate of change of the vertex normal vector in the neighborhood, solves the actual average curvature of the vertex, sets the reference curvature for each surface of the 3D mesh model based on the preset shape parameters of the drug packaging, sets the curvature mutation threshold, calculates the absolute value of the difference between the actual average curvature of the vertex and the reference curvature, and marks vertices with an absolute value of the difference greater than the curvature mutation threshold as suspected defect points.

10. The online drug packaging inspection system based on three-dimensional reconstruction as described in claim 9, characterized in that, The calculation and judgment module uses suspected defect points as initial seed points to execute a region growth algorithm, judges whether adjacent vertices satisfy the consistency of curvature abrupt change and the continuity of mesh topology, and forms a set of suspected defect regions. Based on the tolerance threshold preset for defect judgment in the drug packaging quality standard, it calculates the vertical distance from the vertices of the suspected defect region to the reference plane and the total area of ​​the triangular mesh facets covering the region. It compares the distance, area and tolerance threshold to screen out real defects and distinguish the defect types, records the defect information to form a defect information set, and judges the qualification of drug packaging in combination with the qualification standard and links with the sorting mechanism.

Images