A pump head tilt detection method based on shape analysis and image segmentation fusion

Abstract

The present invention discloses a pump head tilt detection method based on shape analysis and image segmentation fusion, including a method based on image feature enhancement and BiSeNetV2 backbone structure fusion to construct a fusion semantic segmentation model; the image to be detected is sent to the fusion semantic segmentation model to obtain a segmented image of the pump head area; all connected domains of the segmented image are extracted to obtain the key point coordinates of all connected domains, and the reference direction vector of the canned bottle; the angle between the key point coordinates and the reference direction vector of the canned bottle is obtained to obtain the pump head tilt angle; it is determined whether the pump head tilt angle is within a normal threshold range, if it is within the normal threshold range, the pump head tilt angle is determined to be normal, otherwise the pump head tilt angle is determined to be abnormal. The present invention can accurately and automatically detect the tilt defect of the pump head of the boxed product by analyzing the pump head shape of the boxed product and combining contour analysis with image segmentation algorithm through a non-contact visual detection method.

Description

Technical Field

[0001] The invention relates to the technical field of image recognition, and in particular to a pump head tilt detection method based on shape analysis and image segmentation fusion. Background Art

[0002] In the production process of shampoo, canned beverages, etc., packaging is an important link in product production. Abnormal situations such as poor packaging may occur. If such products cannot be removed in time, it will greatly affect the quality of the product and the brand image. Pump head tilt is a common defect in product packaging, and effective detection methods are urgently needed to complete efficient and automated detection of pump head tilt.

[0003] The shortcoming of the existing technology is that there is currently no good and efficient method to accurately and automatically detect and determine the tilt of the pump head of the packaged product, which is a common defect in the packaging of related products, such as the tilt of the pump head. Summary of the invention

[0004] The purpose of the present invention is to overcome the shortcomings of the prior art. To achieve the above purpose, a pump head tilt detection method based on shape analysis and image segmentation fusion is adopted to solve the problems raised in the above background technology.

[0005] A pump head tilt detection method based on shape analysis and image segmentation fusion, the specific steps include:

[0006] Step S1, establishing an image feature enhancement module based on an image enhancement method, and integrating the image feature enhancement module with BiSeNetV2 as the backbone structure of the model to construct a fusion semantic segmentation model;

[0007] Step S2, inputting the training set formed by acquiring the boxed cans and bottles images into the fused semantic segmentation model for training, and then sending the boxed cans and bottles images to be detected into the trained fused semantic segmentation model to obtain a segmented image of the pump head area;

[0008] Step S3, extracting all connected domains of the segmented image of the pump head area, obtaining key point coordinates of all connected domains, and a reference direction vector of the canned bottle, wherein the key point coordinates of the connected domains include the center coordinates of the maximum inscribed circle of each connected domain, the center of gravity coordinates of the contour of each connected domain, and the point coordinates on the contour of the connected domain farthest from the maximum inscribed circle;

[0009] Step S4, obtaining the angle between the key point coordinates and the reference direction vector of the canned bottle to obtain the inclination angle of the pump head;

[0010] Step S5, determining whether the pump head tilt angle is within a normal threshold range, if so, determining that the pump head tilt angle is normal, otherwise determining that the pump head tilt angle is abnormal.

[0011] As a further solution of the present invention: the specific steps of establishing the image feature enhancement module based on the image enhancement method in step S1 include:

[0012] The image feature enhancement module is composed of an edge layer constructed based on an image edge detection method, a filter layer constructed based on an image mean filter method, and a grayscale transformation layer constructed based on an image grayscale transformation method;

[0013] The input boxed can and bottle image is subjected to feature extraction of the edge layer and the filter layer respectively, and the feature extraction result of the edge layer is input to the grayscale transformation layer for feature extraction, and the feature extraction results of the grayscale transformation layer and the filter layer are connected by the Add method.

[0014] As a further solution of the present invention: the specific steps of obtaining the key point coordinates of all connected domains in step S3 include:

[0015] After performing Canny edge detection on the segmented image of the pump head area, the outline of each connected domain is extracted to obtain a circumscribed rectangle of the outline of each connected domain;

[0016] The coordinates of the contour center of each connected domain are obtained by accumulating and averaging the contour coordinates of each connected domain.

[0017] Taking the centroid coordinates of the contour as the center point, selecting the candidate area of ​​the center coordinates of the maximum inscribed circle, traversing and obtaining the closest distances from all pixel points in the candidate area to the contour of the connected domain, the pixel point corresponding to the maximum value in the obtained set of closest distances is the center coordinates of the maximum inscribed circle;

[0018] According to the center coordinates of the maximum inscribed circle, the distances from the pixel points on the contour of the connected domain to the center coordinates of the maximum inscribed circle are traversed to obtain the pixel point corresponding to the maximum value in the obtained distance set is the point coordinate on the contour of the connected domain farthest from the maximum inscribed circle.

[0019] As a further solution of the present invention: the specific steps of obtaining the reference direction vector of the canned bottle in step S3 include:

[0020] According to the prior information of the number of rows and columns of canned bottles in the canned bottle image, the center coordinates of the maximum inscribed circle of each connected domain are sorted to obtain the center coordinates of the maximum inscribed circle of all connected domains in each row;

[0021] Then, the least square method is used to perform straight line fitting on the center coordinates of the maximum inscribed circle of all connected domains in each row to obtain the reference direction vector of each row of canned bottles.

[0022] As a further solution of the present invention: the specific steps of obtaining the angle between the key point coordinates and the reference direction vector of the canned bottle in step S4 to obtain the inclination angle of the pump head include:

[0023] Calculate the angle formed by the direction vector from the center coordinates of the maximum inscribed circle of all connected domains in each row of the boxed canned bottle image to the center of gravity coordinates of the contour and the reference direction vector of the canned bottle in the current row; and

[0024] Calculate the angle formed by the direction vector from the contour centroid coordinates of all connected domains in each row of the boxed canned bottle image to the coordinates of the point on the contour farthest from the center coordinates of the maximum inscribed circle and the reference direction vector of the canned bottle in the current row;

[0025] Select the smaller of the two angles as the pump head inclination angle.

[0026] Compared with the prior art, the present invention has the following technical effects:

[0027] The above technical solution is adopted to establish an image feature enhancement module based on the image enhancement method, and integrate BisenetV2 as the model backbone structure to build a fusion semantic segmentation model; then the boxed canned bottle image to be detected is input into the trained model to obtain a segmented image; the connected domain contour of the image is extracted to obtain the coordinates of the key points, and the angle between the vector of each coordinate point and the reference direction vector of the canned bottle is calculated as the pump head tilt angle to determine whether it is within the normal threshold range. Through the above method design, the pump head shape of the boxed product is analyzed, and the contour analysis and image segmentation algorithm are combined to accurately and efficiently detect the common pump head skew defects in product packaging. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The specific embodiments of the present invention are described in detail below in conjunction with the accompanying drawings:

[0029] Figure 1 is a schematic diagram of the steps of a pump head tilt detection method according to some embodiments disclosed in the present application;

[0030] Figure 2 is a sobel edge operator graph of an embodiment of the present invention;

[0031] Figure 3 is a structural diagram of an image feature enhancement module according to an embodiment of the present invention;

[0032] Figure 4 is a structural diagram of a detail branch fusion module according to an embodiment of the present invention;

[0033] Figure 5 is a structural diagram of a fused semantic segmentation model according to an embodiment of the present invention;

[0034] Figure 6 This is a diagram of packing cans and bottles according to an embodiment of the present invention;

[0035] Figure 7 is a segmentation diagram of the pump head area of ​​an embodiment of the present invention;

[0036] Figure 8 This is a result diagram of the calculation of the center, center of gravity, and farthest point coordinates of the maximum inscribed circle of the contour of the embodiment of the present invention;

[0037] Fig. 9 is a result diagram of calculation of the reference direction vector of the canned bottle of the row to which the connected domain belongs according to an embodiment of the present invention;

[0038] Fig.10 Result diagram of calculation of direction vector of pump head angle according to an embodiment of the present invention;

[0039] Fig.11 4 is a result diagram of the calculation of the pump head tilt angle according to an embodiment of the present invention. DETAILED DESCRIPTION

[0040] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0041] Please refer to Figure 1 In an embodiment of the present invention, a pump head tilt detection method based on shape analysis and image segmentation fusion includes:

[0042] Step S1, establishing an image feature enhancement module based on an image enhancement method, and integrating the image feature enhancement module with BiSeNetV2 as the backbone structure of the model to construct a fusion semantic segmentation model;

[0043] In a specific implementation manner, the specific steps of establishing an image feature enhancement module based on the image enhancement method in step S1 include:

[0044] like Figure 2 As shown, the diagram shows two operator graphs of Sobel horizontal edge and vertical edge;

[0045] By introducing the image edge detection method, the edge layer is constructed. The edge layer can be regarded as the feature extraction of the image by the Sobel horizontal edge operator and the Sobel vertical edge operator respectively, and the absolute values ​​of the feature extraction results of the two are taken and added;

[0046] By introducing the image mean filtering method, a filtering layer is constructed, where the filtering layer can be regarded as a 3x3 average pooling process for the image;

[0047] By introducing the image grayscale transformation method, a grayscale transformation layer is constructed, where the grayscale transformation layer consists of a 1x1 depthwise separable convolutional layer with bias parameters, a BN layer, and a RELU layer.

[0048] like Figure 3 As shown, the diagram is a structural diagram of the image feature enhancement module;

[0049] The image feature enhancement module is composed of an edge layer constructed based on an image edge detection method, a filter layer constructed based on an image mean filter method, and a grayscale transformation layer constructed based on an image grayscale transformation method;

[0050] The input boxed can and bottle image is subjected to feature extraction of the edge layer and the filter layer respectively, and the feature extraction result of the edge layer is input to the grayscale transformation layer for feature extraction, and the feature extraction results of the grayscale transformation layer and the filter layer are connected by the Add method.

[0051] In this embodiment, Figure 4 As shown, the diagram is a structural diagram of the detail branch fusion module;

[0052] The established image feature enhancement module is integrated with the detail branch DetailBranch of the BisenetV2 model. Specifically, the output of the Stage1 module of the detail branch is connected to the image feature enhancement module for feature extraction, and the feature extraction result is input into the Stage2 module of the detail branch. The output of the Stage2 module is connected to the image feature enhancement module for feature extraction. The feature extraction result is input into the Stage3 module of the detail branch. The output of the Stage3 module is connected to the image feature enhancement module for feature extraction.

[0053] like Figure 5 As shown, the figure is a schematic diagram of the fusion semantic segmentation model.

[0054] Step S2, inputting the acquired labeled boxed can and bottle images into the fusion semantic segmentation model for training;

[0055] Specifically, the images of packed cans and bottles collected by video acquisition equipment are used to form a training set, which mainly includes two types of objects: pump heads and backgrounds, and the semantic segmentation training set is labeled;

[0056] The labeled training set is then input into the constructed fusion semantic segmentation model for model training.

[0057] Then, the boxed cans and bottles images to be inspected are fed into the trained fusion semantic segmentation model to obtain the segmented image of the pump head area.

[0058] like Figure 6 As shown, the figure shows the image of boxed cans and bottles to be inspected;

[0059] like Figure 7 As shown, the figure shows the segmented image of the pump head area obtained after feeding into the model;

[0060] Step S3, extracting all connected domains of the segmented image of the pump head area, obtaining key point coordinates of all connected domains, and a reference direction vector of the canned bottle, wherein the key point coordinates of the connected domains include the center coordinates C of the maximum inscribed circle of each connected domain, the center of gravity coordinates G of the contour of each connected domain, and the point coordinates F on the contour of the connected domain farthest from the maximum inscribed circle;

[0061] The specific steps of obtaining the key point coordinates of all connected domains in step S3 include:

[0062] In this embodiment, after performing Canny edge detection on the segmented image of the pump head area, the outline of each connected domain is extracted to obtain the circumscribed rectangle of the outline of each connected domain;

[0063] The contour coordinates of each connected domain are accumulated and averaged to obtain the contour centroid coordinates G of each connected domain. The specific calculation steps are as follows:

[0064] Assume that there are n pixels (x i ,y i )(i=1,2,3,...,n), the coordinates of the centroid of its contour for:

[0065]

[0066] Taking the contour centroid coordinate G as the center point, select the candidate area of ​​the maximum inscribed circle center coordinate C, traverse and find the closest distance from all pixels in the candidate area to the contour of the connected domain, and the pixel point corresponding to the maximum value in the obtained closest distance set is the maximum inscribed circle center coordinate C;

[0067] According to the center coordinate C of the maximum inscribed circle, the distance from the pixel point on the contour of the connected domain to the center coordinate C of the maximum inscribed circle is traversed to obtain the pixel point corresponding to the maximum value in the distance set, which is the point coordinate on the contour of the connected domain farthest from the maximum inscribed circle.

[0068] In this embodiment, Figure 8 As shown, the figure shows the calculation results of the center of the maximum inscribed circle of the contour, the centroid of the contour, and the coordinates of the point on the contour farthest from the maximum inscribed circle;

[0069] In this embodiment, the specific steps of obtaining the reference direction vector D of the canned bottle in step S3 include:

[0070] According to the prior information of the number of rows and columns of canned bottles in the boxed canned bottle image, the maximum inscribed circle center coordinates C of each connected domain are sorted to obtain the maximum inscribed circle center coordinates C of all connected domains in each row;

[0071] Then, the least square method is used to perform straight line fitting on the center coordinates of the maximum inscribed circle of all connected domains in each row to obtain the reference direction vector of each row of canned bottles.

[0072] like Fig. 9 As shown, the figure shows the calculation results of the reference direction vector of each row of canned bottles.

[0073] Step S4, obtaining the angle between the key point coordinates and the reference direction vector of the canned bottle to obtain the pump head tilt angle, the specific steps include:

[0074] Calculate the angle formed by the direction vector from the center coordinates of the maximum inscribed circle of all connected domains in each row of the boxed canned bottle image to the center of gravity coordinates of the contour and the reference direction vector of the canned bottle in the current row; and

[0075] Calculate the angle formed by the direction vector from the contour centroid coordinates of all connected domains in each row of the boxed canned bottle image to the coordinates of the point on the contour farthest from the center coordinates of the maximum inscribed circle and the reference direction vector of the canned bottle in the current row;

[0076] like Fig.10 The figure shows the calculated direction vector for each pump head angle.

[0077] Select the smaller of the two angles as the pump head inclination angle.

[0078] like Fig.11 The figure shows the calculated results for each pump head inclination angle.

[0079] Step S5, determining whether the pump head tilt angle is within a normal threshold range, if so, determining that the pump head tilt angle is normal, otherwise determining that the pump head tilt angle is abnormal.

[0080] In this embodiment, the normal threshold range of the pump head tilt angle is set to (0°, 30°). According to the specific calculation results, it can be obtained that the pump heads in the 2nd row and 3rd column and the 3rd row and 4th column of the boxed bottles in this embodiment are abnormally tilted.

[0081] Although the embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention. The scope of the present invention is defined by the appended claims and their equivalents, and all should be included within the scope of protection of the present invention.

Claims

1. A pump head tilt detection method based on shape analysis and image segmentation fusion, characterized in that: The specific steps include: Step S1, establishing an image feature enhancement module based on an image enhancement method, and integrating the image feature enhancement module with BiSeNetV2 as the backbone structure of the model to construct a fusion semantic segmentation model; Step S2, inputting the training set formed by acquiring the boxed cans and bottles images into the fused semantic segmentation model for training, and then sending the boxed cans and bottles images to be detected into the trained fused semantic segmentation model to obtain a segmented image of the pump head area; Step S3, extracting all connected domains of the segmented image of the pump head area, obtaining key point coordinates of all connected domains, and a reference direction vector of the canned bottle, wherein the key point coordinates of the connected domains include the center coordinates of the maximum inscribed circle of each connected domain, the center of gravity coordinates of the contour of each connected domain, and the point coordinates on the contour of the connected domain farthest from the maximum inscribed circle; Step S4, obtaining the angle between the key point coordinates and the reference direction vector of the canned bottle to obtain the inclination angle of the pump head; Step S5, determining whether the pump head tilt angle is within a normal threshold range, if so, determining that the pump head tilt angle is normal, otherwise determining that the pump head tilt angle is abnormal.

2. According to claim 1, a pump head tilt detection method based on shape analysis and image segmentation fusion is characterized in that: The specific steps of establishing the image feature enhancement module based on the image enhancement method in step S1 include: The image feature enhancement module is composed of an edge layer constructed based on an image edge detection method, a filter layer constructed based on an image mean filter method, and a grayscale transformation layer constructed based on an image grayscale transformation method; The input boxed can and bottle image is subjected to feature extraction of the edge layer and the filter layer respectively, and the feature extraction result of the edge layer is input to the grayscale transformation layer for feature extraction, and the feature extraction results of the grayscale transformation layer and the filter layer are connected by the Add method.

3. According to claim 1, a pump head tilt detection method based on shape analysis and image segmentation fusion is characterized in that: The specific steps of obtaining the key point coordinates of all connected domains in step S3 include: After performing Canny edge detection on the segmented image of the pump head area, the outline of each connected domain is extracted to obtain a circumscribed rectangle of the outline of each connected domain; The coordinates of the contour center of each connected domain are obtained by accumulating and averaging the contour coordinates of each connected domain. Taking the centroid coordinates of the contour as the center point, selecting the candidate area of ​​the center coordinates of the maximum inscribed circle, traversing and obtaining the closest distances from all pixel points in the candidate area to the contour of the connected domain, the pixel point corresponding to the maximum value in the obtained set of closest distances is the center coordinates of the maximum inscribed circle; According to the center coordinates of the maximum inscribed circle, the distances from the pixel points on the contour of the connected domain to the center coordinates of the maximum inscribed circle are traversed to obtain the pixel point corresponding to the maximum value in the distance set, which is the point coordinate on the contour of the connected domain farthest from the maximum inscribed circle.

4. According to claim 3, a pump head tilt detection method based on shape analysis and image segmentation fusion is characterized in that: The specific steps of obtaining the reference direction vector of the canned bottle in step S3 include: According to the prior information of the number of rows and columns of canned bottles in the boxed canned bottle image, the center coordinates of the maximum inscribed circle of each connected domain are sorted to obtain the center coordinates of the maximum inscribed circle of all connected domains in each row; Then, the least square method is used to perform straight line fitting on the center coordinates of the maximum inscribed circle of all connected domains in each row to obtain the reference direction vector of each row of canned bottles.

5. According to claim 1, a pump head tilt detection method based on shape analysis and image segmentation fusion is characterized in that: The specific steps of obtaining the angle between the key point coordinates and the reference direction vector of the canned bottle in step S4 to obtain the inclination angle of the pump head include: Calculate the angle formed by the direction vector from the center coordinates of the maximum inscribed circle of all connected domains in each row of the boxed canned bottle image to the center of gravity coordinates of the contour and the reference direction vector of the canned bottle in the current row; and Calculate the angle formed by the direction vector from the contour centroid coordinates of all connected domains in each row of the boxed canned bottle image to the coordinates of the point on the contour farthest from the center coordinates of the maximum inscribed circle and the reference direction vector of the canned bottle in the current row; Select the smaller of the two angles as the pump head inclination angle.

Images