A curved surface label splicing detection system based on machine vision

Abstract

This invention relates to a machine vision-based curved label splicing and inspection system, belonging to the field of visual inspection technology, and is used to solve the problem of inspecting labels on cylindrical beverage bottles. It includes four high-speed, high-resolution CCD area array cameras for high-speed image capture of the inspected product; each camera is equipped with two LED light sources, positioned above and below the camera respectively, to illuminate the label inspection area; an image processing system for processing the image information and running the curved label splicing and inspection method; and a signal control system. Through an innovative curved label splicing and inspection method, the planar label images obtained by each camera are spliced ​​and fused to obtain a spliced ​​planar label image. This machine vision-based curved label splicing and inspection method achieves the detection and control strategy for the appearance labels, surface dirt, and label damage of cylindrical packaged products, greatly improving production quality and efficiency.

Description

[0001] This application is a divisional application. The original application has the application number 201911078594.7 and the original application date is November 6, 2019. The entire contents of the original application are incorporated herein by reference. Technical Field

[0002] This invention belongs to the field of visual inspection technology, and relates to an inspection system and method, particularly a curved surface label splicing inspection system and method based on machine vision. Background Technology

[0003] With the continuous improvement of living standards, the demand for bottled water, beverages, dairy products, beer, chewing gum, milk powder and canned products is increasing rapidly. The rapid development of intelligent manufacturing technology and the continuous increase in production line speed (>28,000 pieces / hour) mean that the traditional manual inspection methods on the production line can no longer meet the needs of modern production. The demand for appearance inspection of cylindrical packaging products will increase, and it is of great significance for equipment to replace manual labor.

[0004] Currently, label inspection software systems for cylindrical products on the market are all independent software inspection systems with four cameras. There is no inspection system that connects the content captured by the four cameras (after each camera image completes the surface correction plane, feature stitching and fusion are performed). The disadvantages of existing software on the market are that the inspection system detects only one type of content (label position, high and low labels, inverted labels and oblique labels), has low detection accuracy, and low detection efficiency. Summary of the Invention

[0005] The purpose of this invention is to address the aforementioned problems in existing technologies by proposing a machine vision-based curved label splicing detection system and method. The technical problem to be solved by this detection system and method is: how to achieve accurate and high-precision detection of cylindrical beverage bottle labels.

[0006] The objective of this invention can be achieved through the following technical solutions:

[0007] A machine vision-based system and method for detecting splicing defects in curved labels is disclosed. The system includes four high-speed, high-resolution CCD area array cameras, eight high-brightness LED light sources, an image processing system, and a signal control system. The four area array cameras are placed at 90-degree intervals to perform 360-degree blind-spot-free detection of cylindrical beverage bottle labels. Each camera is equipped with two LED light sources to illuminate the label area from both above and below. By combining top and bottom lighting, uniform brightness of the label area is ensured, significantly improving the accuracy of label printing defect detection. The method includes the following steps:

[0008] Step 1: Establish a four-eye joint calibration system to obtain the depth image of the product. Use the image containing depth information to obtain the deviation between the product outline and the template, and obtain the scaling ratio through linear scaling.

[0009] Step 2: The camera automatically captures images in real time, and the software crops the images in real time. After calibrating the images based on their left and right offset from the center position, the images are automatically radially transformed to the template position to achieve the same size as the template.

[0010] Step 3: Unfold the curved image into a planar image, and refine the planar image to subpixel accuracy;

[0011] Step 4: After unfolding the curved image into a planar image, obtain the boundary feature map of each planar image. At the boundary, fuse the images from the four cameras to ensure that there is no image overlap or vertical misalignment at the connection points of each camera image.

[0012] Step 5: To address different label detection needs, Halcon algorithm is used to extract and detect label wrinkles, skewness, and character blurring defects;

[0013] Step Six: The testing personnel observe the computer terminal and record the recognition results.

[0014] Multi-point checkerboard calibration was performed using a single camera. The field-of-view distortion of each camera was adjusted, and the intrinsic and extrinsic parameters of the cameras were determined. The entire system was calibrated using the joint calibration of two adjacent cameras. The multi-point checkerboard used the `SADDLE_POINTS_SUB_PIX` function to obtain saddle points, acquiring sub-pixel information of the checkerboard pattern. The direction information and grayscale variation information of the image were determined using the black-and-white order of the checkerboard. The image information was smoothed using a Gaussian filter, and the edge features of the checkerboard were extracted. The sequence relationship between the X and Y axes of each point was located using continuous quadratic polygon fitting. The weight information of each edge feature was determined using a Hessian matrix. The saddle point coordinates were determined by comparing the deviation with preset values. Adjacent cameras had the same intrinsic parameters: focal length, distortion, center coordinates, and pixel size. The centers of the two cameras coincided with the X-axis of the first camera. P(x c y c , z c Let P be a point in space, and Image1 and Image2 be two imaging planes. The point on the left imaging plane is P'. l (u l v l The point where the image is formed on the right imaging plane is P. r (u r v r If the cameras are installed at the same height, meaning point P is installed at the same height on both imaging planes, then:

[0015] v l =v r

[0016] From geometric relations, we can obtain:

[0017]

[0018] In the formula x c — Spatial coordinates of point P in the x-direction;

[0019] y c — Spatial coordinates of point P in the y-direction;

[0020] z c — Spatial coordinates of point P in the z-direction;

[0021] u l —The point on the left plane of the image is P. l The row coordinates;

[0022] v l —The point on the left plane of the image is P. l Column coordinates;

[0023] u r —Point P on the right plane image r The row coordinates;

[0024] v r —Point P on the right plane image r The row coordinates;

[0025] b — Baseline distance, i.e., the distance between the two cameras;

[0026] In a binocular vision system, the field difference is defined as the difference between the corresponding points P on the two imaging planes of point P. l and P r Positional difference;

[0027]

[0028] Therefore, the coordinates of a point in space in the left camera coordinate system can be calculated as follows:

[0029]

[0030] The extrinsic parameters of the camera, i.e., the relationship between the image coordinate system and the pixel coordinate system, are determined as follows:

[0031]

[0032] (1) World coordinate system (X) w ,Y w Z w The coordinates of ) are converted to the camera coordinate system (Xc,Yc,Z).c The coordinates of the world coordinate system need to be determined through a rigid body transformation. A rigid body transformation involves translating and rotating the world coordinate system to align it with the camera coordinate system. The transformation relationship is as follows:

[0033]

[0034] In the formula, R is the rotation matrix.

[0035] T—Translation vector

[0036] T(T x ,T y ,T z Let R(α,β,γ) be the translation vector, and R(α,β,γ) be the rotation matrix, where α,β,γ are the rotation angles about the camera's x, y, and z axes, respectively.

[0037]

[0038]

[0039]

[0040] R(α,β,γ)=R(α)R(β)R(γ)

[0041] (2) The following is the camera coordinate system (X) c ,Y c Z c The homogeneous transformation from the image coordinate system (x, y) to the image coordinate system (x, y), neglecting distortion, yields:

[0042]

[0043] (3) The above is the ideal situation, but in reality, due to processing errors, camera distortion is unavoidable. Taking distortion into account, the coordinates (x, y) of the distorted image and the coordinates (x′, y′) of the original image have the following relationship:

[0044]

[0045] In the formula, δx′ is the distortion of the original image along the x-axis in the coordinate system.

[0046] δy′ represents the distortion of the original image along the y-axis in the coordinate system;

[0047] (4) Without considering distortion, the image coordinate system (x,y) has the following homogeneous transformation relationship with the pixel coordinate system (u,v):

[0048]

[0049] Without considering camera lens distortion, the relationship between the world coordinate system and the image coordinate system constitutes a universal camera model.

[0050]

[0051] In the formula, M1 represents the camera's internal parameters.

[0052] M2 — Camera external parameters;

[0053] W h —The homogeneous coordinates of a point in three-dimensional space in the world coordinate system;

[0054] s—proportion factor;

[0055] Among them, parameters such as M1 are determined during camera manufacturing and are related to the camera's structure; these are called the camera's intrinsic parameters. M2 describes the positions of the two cameras and is related to their placement; these are commonly referred to as extrinsic parameters. W... h These are the homogeneous coordinates of an object in the world coordinate system, used to describe the object's spatial position.

[0056] Considering distortion, the camera model becomes a nonlinear model, which has additional distortion coefficients compared to the linear model. For example, if the ideal coordinates calculated by the pinhole camera model are (x, y), while the actual coordinates of the image points are (x′, y′), then:

[0057]

[0058] In the formula, k1 represents the distortion in the x-direction;

[0059] k2 — distortion coefficient in the y-direction;

[0060] During the calibration of the stereo system, initial values ​​for the camera intrinsic parameters are first required. The camera distortion coefficients are set to 0. The operator used to set the initial values ​​of the camera intrinsic parameters is `gen_cam_par_area_scan_division`. The spatial coordinates of the circular marker points on the calibration board can be read from the calibration board description file, using the operator `set_calib_data_calib_object`. The intrinsic parameter calibration results for the left and right cameras are shown in the table below.

[0061]

[0062] As can be seen from the table above, the distortion coefficient κ of the left camera is 1226.62, and the distortion coefficient κ of the right camera is 3425.91. The distortion of both cameras is positive, indicating that the distortion of the photographs is pincushion distortion.

[0063] The extrinsic parameters of the calibrated binoculars are shown in the table below.

[0064]

[0065] The rotation matrix obtained from the calibration provides the following information: the right camera rotates 0.95° around the X-axis, 13.67° around the Y-axis, and 0.75° around the Z-axis relative to the left camera. The rotations along the X and Z axes are much smaller than those along the Y-axis, indicating that the right camera has essentially rotated 13.67° around the Y-axis to coincide with the left camera. The translation vector provides the following information: the translation distance of the right camera relative to the left camera is:

[0066]

[0067] Based on the camera calibration results, the depth information of the template product in each camera is established when the template product is at the center position. Camera 1 is represented by Z1, Camera 2 by Z2, Camera 3 by Z3, and Camera 4 by Z4. When the product passes through the detection center position, the offset positions of the product center obtained by each camera are Z1', Z2', Z3', and Z4', respectively. Let Z' be the offset position of the product center obtained by the camera, and Z be the position of the template product center. According to the pre-defined template relationship, the diameter d of the product in the current image is derived. The change d' of the detected product diameter is obtained through linear scaling transformation.

[0068]

[0069] In the formula, d′ represents the diameter change of the product in the image, d represents the diameter of the template product, Z′ represents the offset position of the product center obtained by the camera, and Z represents the position of the template product center.

[0070] After obtaining the image edges, the horizontal X and vertical Y scaling ratio coefficients ψ are obtained through linear scaling transformation, thereby ensuring that the product maintains the same scale as the model when performing surface correction.

[0071] The surface image is unfolded into a planar image using methods such as saddle points, Hessian matrix, Gaussian kernel, or bilinear interpolation. The sub-pixel accuracy correction steps are as follows:

[0072] Step 1: Set the region of interest (ROI) containing the target object in the template image, exclude other unwanted parts, and construct a multi-layered pyramid for the ROI in the template;

[0073] Step 2: Perform edge filtering on each layer of the pyramid of the template to obtain the edge series of the template, and then rotate this edge series;

[0074] Step 3: Set the region of interest (ROI) in the target image as the search region to improve search speed;

[0075] Step 4: Similarly, generate a pyramid for the search region of the target image, and then perform edge filtering to obtain the edge-filtered image pyramid.

[0076] Step 5: Starting from the top of the pyramid, perform template matching on the processed target image layer by layer to obtain the corresponding similarity measure, and take the local maximum similarity measure as the matching point;

[0077] Step Six: Correct the final matching points obtained above using the least squares method to obtain sub-pixel precision matching.

[0078] There are two ways to extract features from the camera edges during the process of obtaining boundary feature maps: the first is based on shape matching, which manually creates templates for regions with obvious features; the second is based on corner matching, which extracts features through corner features and gray value changes.

[0079] The Halcon algorithm is as follows:

[0080] (1) Use Halcon's rgb1_to_gray operator to convert the RGB image to a grayscale image. A normal color image has three channels: R, G, and B, and is called a true color image. In order to improve the image processing speed, the color image needs to be converted to a grayscale image during image processing. When converting a color image to a grayscale image, it is necessary to calculate the effective brightness value of each channel. The calculation formula is as follows:

[0081] G(i,j)=0.3R(i,j)+0.59G(i,j)+0.11B(i,j)

[0082] In the formula, G(i,j) is the gray value of point (i,j).

[0083] R(i,j)——The gray value of the R channel at point (i,j) in the color image;

[0084] G(i,j)——Gen channel gray value of point (i,j) in a color image;

[0085] B(i,j) — The gray value of the B channel at point (i,j) in the color image;

[0086] (2) Set ROI. The purpose of setting ROI is to reduce the processing area. The ROI area is the area near the slab. Use the operator gen_rectangle1 to get a rectangle, and then use reduce_domain to remove the area outside the ROI. (3) Mean filtering. The operator used is mean_image. This operator uses the mean to smooth the image and remove noise. The size of the mean filter can be selected by yourself.

[0087] (4) Dynamic thresholding. The operator for dynamic thresholding is dyn_threshold. The input is the image after mean filtering and the original image. The output is the result of dynamic thresholding. The result of image segmentation can be adjusted by adjusting the size of the filter. The smaller the filter size, the more edges are segmented. As the filter size increases, the edges will become wider. In layman's terms, dynamic thresholding can be understood as contrast, that is, comparing a certain pixel with its surrounding pixels.

[0088] (5) Connectivity analysis: Connectivity analysis is to calculate the connected parts in the segmented region. The connected parts belong to a connected region. The operator used in this process is connection, which separates the region segmented by the above dynamic threshold into a single connected region.

[0089] (6) Shape selection. The operator used for shape selection is select_shape, which removes the interference region by using the area feature. The parameter of the area feature is "area". In addition, this operator has about 70 parameters, including "col", "row", "width", "height", etc.

[0090] (7) Shape filling, using the operator fill_up, which fills the holes in the region to make it a simply connected region.

[0091] (8) Opening operation. The operator for opening operation is opening_circle, which is to use a circular struct to perform an opening operation on the graphic. In addition, Halcon also has the operator opening_rectangle1, which is to use a rectangular region to perform an opening operation on the graphic. opening is to use a user-defined struct to perform an opening operation on the graphic.

[0092] (9) Calculate the difference between the two regions by using the preset features in the template information and the difference operator;

[0093] (10) Defect extraction is a shape transformation in Halcon. The operator used is shape_trans, with the parameter set to "convex" to transform the original shape into a convex shape. There are also parameters "ellipse" and "rectangle2", which transform the region of the original shape into an ellipse and a rectangle.

[0094] Compared with existing technologies, this machine vision-based curved surface label splicing detection device and method have the following advantages:

[0095] This paper proposes a multi-camera joint calibration technique for surface label stitching and detection. Four cameras are jointly calibrated, and then the cylindrical object is unfolded into a plane. Feature extraction is used to stitch and fuse the planar label images from each camera. After obtaining the stitched label planar image, it is possible to detect label seam deviation, main feature skew, displacement deviation between front and rear labels, label wrinkles, label damage, duplicate characters, missing characters, and improve the variety and accuracy of defects detected using traditional methods. Attached Figure Description

[0096] Figure 1 This is a workstation distribution diagram in this invention;

[0097] Figure 2 This is a flowchart of the detection method in this invention;

[0098] Figure 3 This is a calibration flowchart of the four-target calibration system in this invention;

[0099] Figure 4 This is a flowchart of the product scaling ratio and cutting ratio in this invention;

[0100] Figure 5 This is a flowchart of the chessboard unfolding process in this invention;

[0101] Figure 6 This is a flowchart of feature matching and edge fusion in this invention;

[0102] Figure 7 This is a flowchart of defect extraction and special location measurement in this invention. Detailed Implementation

[0103] The technical solution of this patent will be further described in detail below with reference to specific embodiments.

[0104] Please see Figure 1-7 This embodiment provides a machine vision-based method for detecting splicing defects in curved labels. The detection system includes four high-speed, high-resolution CCD area array cameras, eight high-brightness LED light sources, an image processing system, and a signal control system. The four area array cameras are placed at 90-degree intervals to perform 360-degree blind-spot-free detection of cylindrical beverage bottle labels. Each camera is equipped with two LED light sources, illuminating the label area from above and below. By combining top and bottom lighting, uniform brightness of the label area is ensured, greatly improving the accuracy of label printing defect detection. The method includes the following steps:

[0105] Step 1: Establish a four-eye joint calibration system to obtain the depth image of the product. Use the image containing depth information to obtain the deviation between the product outline and the template, and obtain the scaling ratio through linear scaling.

[0106] Step 2: The camera automatically captures images in real time, and the software crops the images in real time. After calibrating the images based on their left and right offset from the center position, the images are automatically radially transformed to the template position to achieve the same size as the template.

[0107] Step 3: Unfold the curved image into a planar image, and refine the planar image to subpixel accuracy;

[0108] Step 4: After unfolding the curved image into a planar image, obtain the boundary feature map of each planar image. At the boundary, fuse the images from the four cameras to ensure that there is no image overlap or vertical misalignment at the connection points of each camera image.

[0109] Step 5: To address different label detection needs, Halcon algorithm is used to extract and detect label wrinkles, skewness, and character blurring defects;

[0110] Step Six: The testing personnel observe the computer terminal and record the recognition results.

[0111] Multi-point checkerboard calibration was performed using a single camera. The field-of-view distortion of each camera was adjusted, and the intrinsic and extrinsic parameters of the cameras were determined. The entire system was calibrated using the joint calibration of two adjacent cameras. The multi-point checkerboard used the `SADDLE_POINTS_SUB_PIX` function to obtain saddle points, acquiring sub-pixel information of the checkerboard pattern. The direction information and grayscale variation information of the image were determined using the black-and-white order of the checkerboard. The image information was smoothed using a Gaussian filter, and the edge features of the checkerboard were extracted. The sequence relationship between the X and Y axes of each point was located using continuous quadratic polygon fitting. The weight information of each edge feature was determined using a Hessian matrix. The saddle point coordinates were determined by comparing the deviation with preset values. Adjacent cameras had the same intrinsic parameters: focal length, distortion, center coordinates, and pixel size. The centers of the two cameras coincided with the X-axis of the first camera. P(x c y c , z c Let P be a point in space, and Image1 and Image2 be two imaging planes. The point on the left imaging plane is P'. l (u l v l The point where the image is formed on the right imaging plane is P. r (u r v r If the cameras are installed at the same height, meaning point P is installed at the same height on both imaging planes, then:

[0112] v l =v r

[0113] From geometric relations, we can obtain:

[0114]

[0115] In the formula x c — Spatial coordinates of point P in the x-direction;

[0116] y c — Spatial coordinates of point P in the y-direction;

[0117] z c — Spatial coordinates of point P in the z-direction;

[0118] u l —The point on the left plane of the image is P. l The row coordinates;

[0119] v l —The point on the left plane of the image is P. l Column coordinates;

[0120] u r —Point P on the right plane image r The row coordinates;

[0121] v r —Point P on the right plane image r The row coordinates;

[0122] b — Baseline distance, i.e., the distance between the two cameras;

[0123] In a binocular vision system, the field difference is defined as the difference between the corresponding points P on the two imaging planes of point P. l and P r The positional difference.

[0124]

[0125] Therefore, the coordinates of a point in space in the left camera coordinate system can be calculated as follows:

[0126]

[0127] The extrinsic parameters of the camera, i.e., the relationship between the image coordinate system and the pixel coordinate system, are determined as follows:

[0128]

[0129] (1) World coordinate system (X) w ,Y w Z w The coordinates of the camera are converted to the camera coordinate system (X). c ,Y c Z cThe coordinates of the world coordinate system need to be determined through a rigid body transformation. A rigid body transformation involves translating and rotating the world coordinate system to align it with the camera coordinate system. The transformation relationship is as follows:

[0130]

[0131] In the formula, R is the rotation matrix.

[0132] T—Translation vector

[0133] T(T x ,T y ,T z Let R(α,β,γ) be the translation vector, and R(α,β,γ) be the rotation matrix, where α,β,γ are the rotation angles about the camera's x, y, and z axes, respectively.

[0134]

[0135]

[0136] R(α,β,γ)=R(α)R(β)R(γ)

[0137] (2) The following is the camera coordinate system (X) c ,Y c Z c The homogeneous transformation from the image coordinate system (x, y) to the image coordinate system (x, y), neglecting distortion, yields:

[0138]

[0139] (3) The above is the ideal situation, but in reality, due to processing errors, camera distortion is unavoidable. Taking distortion into account, the coordinates (x, y) of the distorted image and the coordinates (x′, y′) of the original image have the following relationship:

[0140]

[0141] In the formula, δx′ is the distortion of the original image along the x-axis in the coordinate system.

[0142] δy′ represents the distortion of the original image along the y-axis in the coordinate system;

[0143] (4) Without considering distortion, the image coordinate system (x,y) has the following homogeneous transformation relationship with the pixel coordinate system (u,v):

[0144]

[0145] Without considering camera lens distortion, the relationship between the world coordinate system and the image coordinate system constitutes a universal camera model.

[0146]

[0147] In the formula, M1 represents the camera's internal parameters;

[0148] M2 — Camera external parameters;

[0149] W h —The homogeneous coordinates of a point in three-dimensional space in the world coordinate system;

[0150] s — scaling factor;

[0151] Parameters such as M1 are determined during camera manufacturing and are related to the camera's structure; these are called the camera's intrinsic parameters. M2 describes the position of the two cameras and is related to their placement; these are commonly referred to as extrinsic parameters. W... h These are the homogeneous coordinates of an object in the world coordinate system, used to describe the object's spatial position.

[0152] Considering distortion, the camera model becomes a nonlinear model, which has additional distortion coefficients compared to the linear model. For example, if the ideal coordinates calculated by the pinhole camera model are (x, y), while the actual coordinates of the image points are (x′, y′), then:

[0153]

[0154] In the formula, k1 represents the distortion in the x-direction;

[0155] k2 — distortion coefficient in the y-direction;

[0156] During the calibration of the stereo system, initial values ​​for the camera intrinsic parameters are first required. The camera distortion coefficients are set to 0. The operator used to set the initial values ​​of the camera intrinsic parameters is `gen_cam_par_area_scan_division`. The spatial coordinates of the circular marker points on the calibration board can be read from the calibration board description file, using the operator `set_calib_data_calib_object`. The intrinsic parameter calibration results for the left and right cameras are shown in the table below.

[0157]

[0158] As can be seen from the table above, the distortion coefficient κ of the left camera is 1226.62, and the distortion coefficient κ of the right camera is 3425.91. The distortion of both cameras is positive, indicating that the distortion of the photographs is pincushion distortion.

[0159] The extrinsic parameters of the calibrated binoculars are shown in the table below.

[0160]

[0161] The rotation matrix obtained from the calibration provides the following information: the right camera rotates 0.95° around the X-axis, 13.67° around the Y-axis, and 0.75° around the Z-axis relative to the left camera. The rotations along the X and Z axes are much smaller than those along the Y-axis, indicating that the right camera has essentially rotated 13.67° around the Y-axis to coincide with the left camera. The translation vector provides the following information: the translation distance of the right camera relative to the left camera is:

[0162]

[0163] Based on the camera calibration results, the depth information of the template product in each camera is established when the template product is at the center position. Camera 1 is represented by Z1, Camera 2 by Z2, Camera 3 by Z3, and Camera 4 by Z4. The offset positions of the product center obtained by each camera when the product passes through the detection center position are Z1', Z2', Z3', and Z4', respectively. Let Z' be the offset position of the product center obtained by the camera, and Z be the position of the center of the template product. According to the pre-defined template relationship, the diameter d of the product in the current image is derived, and the change d' of the product diameter is obtained through linear scaling transformation.

[0164]

[0165] In the formula, d′ represents the diameter change of the product in the image, d represents the diameter of the template product, Z′ represents the offset position of the product center obtained by the camera, and Z represents the position of the template product center.

[0166] After obtaining the image edges, the horizontal X and vertical Y scaling ratio coefficients ψ are obtained through linear scaling transformation, thereby ensuring that the product maintains the same scale as the model when performing surface correction.

[0167] In this embodiment, the distance from point P to the binocular system is obtained by determining the intrinsic parameters of the two cameras and the position of the second camera relative to the first camera.

[0168] The surface image is unfolded into a planar image using methods such as saddle points, Hessian matrix, Gaussian kernel, or bilinear interpolation. The sub-pixel accuracy correction steps are as follows:

[0169] Step 1: Set the region of interest (ROI) containing the target object in the template image, exclude other unwanted parts, and construct a multi-layered pyramid for the ROI in the template;

[0170] Step 2: Perform edge filtering on each layer of the pyramid of the template to obtain the edge series of the template, and then rotate this edge series;

[0171] Step 3: Set the region of interest (ROI) in the target image as the search region to improve search speed;

[0172] Step 4: Similarly, generate a pyramid for the search region of the target image, and then perform edge filtering to obtain the edge-filtered image pyramid.

[0173] Step 5: Starting from the top of the pyramid, perform template matching on the processed target image layer by layer to obtain the corresponding similarity measure, and take the local maximum similarity measure as the matching point;

[0174] Step 6: Correct the final matching points obtained above using the least squares method to obtain sub-pixel precision matching.

[0175] There are two ways to extract features from the camera edges during the process of obtaining boundary feature maps: the first is based on shape matching, which manually creates templates for regions with obvious features; the second is based on corner matching, which extracts features through corner features and gray value changes.

[0176] The Halcon algorithm is as follows:

[0177] (1) Use Halcon's rgb1_to_gray operator to convert the RGB image to a grayscale image. A normal color image has three channels: R, G, and B, and is called a true color image. In order to improve the image processing speed, the color image needs to be converted to a grayscale image during image processing. When converting a color image to a grayscale image, it is necessary to calculate the effective brightness value of each channel. The calculation formula is as follows:

[0178] G(i,j)=0.3R(i,j)+0.59G(i,j)+0.11B(i,j)

[0179] In the formula, G(i,j) is the gray value of point (i,j).

[0180] R(i,j)——The gray value of the R channel at point (i,j) in the color image;

[0181] G(i,j)——Gen channel gray value of point (i,j) in a color image;

[0182] B(i,j) — The gray value of the B channel at point (i,j) in the color image;

[0183] (2) Set the ROI. The purpose of setting the ROI is to reduce the processing area. The ROI is set to the area near the slab. A rectangle is obtained using the operator gen_rectangle1, and then the area outside the ROI is removed using reduce_domain.

[0184] (3) Mean filtering, using the mean_image operator, which uses the mean to smooth the image and remove noise. The size of the mean filter can be selected by yourself.

[0185] (4) Dynamic thresholding. The operator for dynamic thresholding is dyn_threshold. The input is the image after mean filtering and the original image. The output is the result of dynamic thresholding. The result of image segmentation can be adjusted by adjusting the size of the filter. The smaller the filter size, the more edges are segmented. As the filter size increases, the edges will become wider. In layman's terms, dynamic thresholding can be understood as contrast, that is, comparing a certain pixel with its surrounding pixels.

[0186] (5) Connectivity analysis: Connectivity analysis is to calculate the connected parts in the segmented region. The connected parts belong to a connected region. The operator used in this process is connection, which separates the region segmented by the above dynamic threshold into a single connected region.

[0187] (6) Shape selection. The operator used for shape selection is select_shape, which removes the interference area by using the area feature. The parameter of the area feature is "area". In addition, this operator has about 70 parameters, including "col", "row", "width", "height", etc.

[0188] (7) Shape filling, using the operator fill_up, which fills the holes in the region to make it a simply connected region.

[0189] (8) Opening operation. The operator for opening operation is opening_circle, which is to use a circular struct to perform an opening operation on the graphic. In addition, Halcon also has the operator opening_rectangle1, which is to use a rectangular region to perform an opening operation on the graphic. opening is to use a user-defined struct to perform an opening operation on the graphic.

[0190] (9) Calculate the difference. Using the preset features in the template information, calculate the difference between the two regions using the difference operator;

[0191] (10) Defect extraction is a shape transformation in Halcon. The operator used is shape_trans, with the parameter set to "convex" to transform the original shape into a convex shape. There are also parameters "ellipse" and "rectangle2", which transform the region of the original shape into an ellipse and a rectangle.

[0192] The aforementioned "circular structuring element" and "structuring element" refer to an auxiliary tool used in morphological image processing. A structuring element is a set of pixels with a known shape and size. Depending on the image analysis purpose, commonly used structuring elements include rectangles, flat shapes, and circles. Furthermore, the operator type used in the opening operation in the Halcon machine vision library is determined by the shape of the structuring element. For example, the opening_circle operator uses a circular structuring element to perform the opening operation on a graphic; similarly, the opening_rectangle1 operator uses a rectangular structuring element to perform the opening operation, while the opening operator uses a structuring element of a user-defined shape. The names and specific meanings of these operators are based on the specifications of the Halcon machine vision library.

[0193] In summary, this paper proposes a multi-camera joint calibration technique for surface label stitching and detection. Four cameras are jointly calibrated, and then the cylindrical object is unfolded into a plane. Feature extraction is used to stitch and fuse the planar label images from each camera. After obtaining the stitched label planar image, it is possible to detect label seam deviation, main feature skew, displacement deviation between front and rear labels, label wrinkles, label damage, duplicate characters, missing characters, and improve the variety and accuracy of defects detected using traditional methods.

[0194] The preferred embodiments of this patent have been described in detail above. However, this patent is not limited to the above embodiments. Within the scope of knowledge possessed by those skilled in the art, various changes can be made without departing from the spirit of this patent.

Claims

1. A machine vision-based curved surface label splicing detection system, comprising the following modules: Four high-speed, high-resolution CCD area array cameras are used to capture images of the inspected products at high speed. The four area array cameras are placed at 90-degree intervals. Eight high-brightness LED light sources are used, with two LED light sources for each camera, positioned above and below the camera respectively, to illuminate the label detection area from above and below. An image processing system is used to process image information acquired by four high-speed, high-resolution CCD area array cameras and run a surface label stitching detection method. And a signal control system for coordinating and controlling the above modules; The method for detecting splicing of curved labels includes the following steps: Step 1: Establish a four-eye joint calibration system to obtain the depth image of the product. Use the image containing depth information to obtain the deviation between the product outline and the template, and obtain the scaling ratio through linear scaling. Step 2: The camera automatically captures images in real time, and the software crops the images in real time. After calibrating the images based on their left and right offset from the center position, the images are automatically transformed into the template position to achieve the same size as the template. Step 3: Unfold the curved image into a planar image, and refine the planar image to subpixel accuracy; Step 4: After unfolding the curved image into a planar image, obtain the boundary feature map of each planar image. At the boundary, fuse the images from the four cameras to ensure that there is no image overlap or vertical misalignment at the connection points of each camera image. Step 5: To address different label detection needs, Halcon algorithm is used to extract and detect label wrinkles, skewness, and character blurring defects; Step Six: The testing personnel observe the computer terminal and record the recognition results; In step one, the four-camera joint calibration system uses a single camera to perform multi-point checkerboard calibration. The field-of-view distortion of each camera is adjusted, and then the intrinsic and extrinsic parameters of the cameras are determined. The entire system is jointly calibrated using two adjacent cameras. The multi-point checkerboard uses the `SADDLE_POINTS_SUB_PIX` function to obtain saddle points, acquiring sub-pixel information of the checkerboard pattern. Furthermore, the direction information and grayscale variation information of the image are determined using the black-and-white order of the checkerboard. The image information is smoothed using a Gaussian filter, and the edge features of the checkerboard are extracted. The sequence relationship between the X and Y axes of each point is located using continuous quadratic polygon fitting. The weight information of each edge feature is determined using a Hessian matrix. The saddle point coordinates are determined by comparing the deviation with preset values. Two adjacent cameras have the same intrinsic parameters, namely focal length, distortion, center coordinates, and pixel size. The centers of the two cameras coincide with the X-axis of the first camera. P(x c y c , z c Let P be a point in space, and Image1 and Image2 be two imaging planes. The point on the left imaging plane is P'. l (u) l v l The point where the image is formed on the right imaging plane is P. r (u) r v r The cameras are installed at the same height, that is... If the points are installed at the same height on both imaging planes, then: From geometric relationships, we can obtain: ; In a binocular vision system, the field difference is defined as a point At corresponding points on both imaging planes and Positional difference; Therefore, the coordinates of a point in space in the left camera coordinate system can be calculated as follows: .

2. The machine vision-based curved surface label splicing detection system as described in claim 1, characterized in that, The determination of the camera's extrinsic parameters, namely the relationship between the image coordinate system and the pixel coordinate system, has the following relationship: (1) World coordinate system X w ,Y w , Z w The coordinates of ) are converted to the camera coordinate system ( X c ,Y c ,Z c The coordinates of the world coordinate system need to be determined through a rigid body transformation. A rigid body transformation involves translating and rotating the world coordinate system to align it with the camera coordinate system. This rigid body transformation is as follows: It is a translation vector. Let be a rotation matrix, where These are the angles of rotation around the camera's x, y, and z axes, respectively. (2) The following is the camera coordinate system To the image coordinate system The homogeneous transformation, neglecting distortion, yields: (3) Considering camera distortion, the coordinates of the distorted image are... Coordinates of the original image The following relationships exist: In the formula, δx´ represents the distortion of the original image along the x-axis in the coordinate system. δy´ represents the distortion of the original image along the y-axis in the coordinate system; (4) Image coordinate system Without considering distortion, its relationship with the pixel coordinate system The homogeneous transformation relationship is as follows: Without considering camera lens distortion, the relationship between the world coordinate system and the image coordinate system constitutes a universal camera model. parameter These are determined during camera manufacturing and are related to the camera's structure; they are called the camera's intrinsic parameters. This describes the positions of the two cameras, which are related to the camera placement and are considered external parameters. These are the homogeneous coordinates of an object in the world coordinate system, used to describe the object's spatial position. In this model, the camera model becomes a nonlinear model, which has an additional distortion coefficient compared to the linear model. The ideal coordinates calculated by the camera pinhole imaging model are ( x,y The coordinates of the actual image points are... Then we have: During the calibration of the binocular system, the initial values ​​of the camera intrinsic parameters must first be provided. The camera distortion coefficient is set to 0. The operator used to set the initial values ​​of the camera intrinsic parameters is: gen_cam_par_area_scan_division. The spatial coordinates of the circular marker points on the calibration board can be read from the calibration board description file. The operator used is set_calib_data_calib_object. The calibration results of the intrinsic parameters of the left and right cameras and the calibration results of the binocular system's extrinsic parameters are obtained. The information obtained from the rotation matrix in the extrinsic parameter calibration results includes: the rotation angle of the right camera relative to the left camera around the X-axis, the rotation angle around the Y-axis, and the rotation angle around the Z-axis. When the rotation angles of the X-axis and Z-axis are less than the rotation angle of the Y-axis, it is determined that the rotation of the right camera relative to the left camera is mainly concentrated in the Y-axis direction. This means that the right camera can coincide with the left camera by rotating around the Y-axis by a predetermined angle. The translation distance of the right camera relative to the left camera is obtained from the translation vector. Based on the camera calibration results, the depth information of the template product in each camera is established when the template product is at the center position. Camera 1 is represented by Z1, Camera 2 by Z2, Camera 3 by Z3, and Camera 4 by Z4. The product passes through the detection center position, and the offset positions of the product center obtained by each camera are Z1', Z2', Z3', and Z4', respectively. Let Z' be the offset position of the product center obtained by the camera, and Z be the position of the center of the template product. According to the pre-defined template relationship, the diameter d of the product in the current image is derived, and the change d' of the product diameter is obtained through linear scaling transformation. In the formula, d´ represents the diameter change of the product in the image, d represents the diameter of the template product, Z´ represents the offset position of the product center obtained by the camera, and Z represents the position of the template product center; After obtaining the image edges, the scaling ratio coefficients ψ for the horizontal X and vertical Y axes are obtained through linear scaling transformation.

3. The machine vision-based curved surface label splicing detection system as described in claim 1, characterized in that, Step 3 uses saddle points, Hessian matrix, Gaussian kernel, or bilinear interpolation to unfold the surface image into a planar image. The sub-pixel accuracy correction steps are as follows: Step 1: Set the region of interest (ROI) containing the target object in the template image, exclude other unwanted parts, and construct a multi-layered pyramid for the ROI in the template; Step 2: Perform edge filtering on each layer of the template pyramid to obtain the edge series of the template image, and then rotate the edge series; Step 3: Set the region of interest (ROI) in the target image as the search region to improve search speed; Step 4: Similarly, generate a pyramid for the search region of the target image, and then perform edge filtering to obtain the edge-filtered pyramid. Step 5: Starting from the top of the pyramid, perform template matching on the processed target image layer by layer to obtain the corresponding similarity measure, and take the local maximum similarity measure as the matching point; Step 6: The obtained matching points are corrected using the least squares method to obtain sub-pixel precision matching.

4. The machine vision-based curved surface label splicing detection system as described in claim 1, characterized in that, In step four, there are two ways to extract features from the camera edges during the process of obtaining the boundary feature map: the first is based on shape matching, and the second is based on corner matching, which extracts features through corner features and gray value changes.

5. The machine vision-based curved surface label splicing detection system as described in claim 1, characterized in that, The Halcon algorithm described in step five is as follows: (1) Use Halcon's rgb1_to_gray operator to convert the RGB image to a grayscale image. The formula for calculating the effective brightness values ​​of the R, G, and B channels is as follows: G(i,j)= 0.3 R(i,j)+ 0.59 G(i,j)+ 0.11 B(i,j) (2) Set the ROI. The ROI area is the area near the slab. Use the operator gen_rectangle1 to get a rectangle, and then use reduce_domain to remove the area outside the ROI. (3) Mean filtering, using the mean_image operator, which uses the mean to smooth the image and remove noise. The size of the mean filter can be determined by selection. (4) Dynamic thresholding: The operator for dynamic thresholding is dyn_threshold. The input is the image after mean filtering and the original image, and the output is the result of dynamic thresholding. The result of image segmentation is adjusted by adjusting the size of the filter. (5) Connectivity analysis: Connectivity analysis calculates the connected parts in the segmented region. The connected parts belong to a connected region. The operator used in this connectivity analysis process is connection, which separates the dynamically thresholded region into a single connected region. (6) Shape selection. The operator used for shape selection is select_shape, which removes the interference region by using the area feature. The parameter of the area feature is "area". (7) Shape filling, using the fill_up operator, which fills the holes in the region to make it a simply connected region; (8) Opening operation. The operator for opening operation is opening_circle, which is to use a circular struct to perform an opening operation on the graphic. In addition, Halcon also has the operator opening_rectangle1, which is to use a rectangular region to perform an opening operation on the graphic. opening is to use a user-defined struct to perform an opening operation on the graphic. (9) Calculate the difference: Using the preset features in the template information, use the difference operator to calculate the difference between the two regions; (10) Defect extraction is a shape transformation in Halcon. The operator used is shape_trans, with the parameter set to "convex" to transform the original shape into a convex shape. There are also parameters "ellipse" and "rectangle2", which transform the region of the original shape into an ellipse and a rectangle.

Images