A one-time chopstick quantity detection method based on image processing technology

Abstract

The application discloses a kind of disposable chopsticks quantity detection method based on image processing technology, comprising the following steps: step 1: industrial camera adopts hard trigger mode, receives the trigger command outside, output one frame disposable tableware package high-resolution image, based on this, establish the training dataset and detection image library of disposable tableware package.The application relates to the field of image processing technology, the beneficial effects of the application are that the scheme of using instance Mask does pixel-level NMS, improves the detection accuracy of chopsticks quantity.Secondly, this paper uses traditional method for pixel-level segmentation, and integrates adaptive threshold calculation method, and carries out secondary determination to chopsticks quantity.Finally, the results of deep learning and pixel-level segmentation are integrated through fully connected network, which can solve the problem of chopsticks quantity detection inaccuracy, and the final accuracy can reach 99.88%.

Description

Technical Field

[0001] This invention relates to the field of image processing, and in particular to a method for detecting the number of disposable chopsticks based on image processing technology. Background Technology

[0002] Currently, my country's manufacturing industry is gradually losing its traditional cost advantages and facing dual pressures from both domestic and international markets. The transformation of the manufacturing sector has become an urgent issue. On the one hand, rising domestic labor and raw material costs have impacted the initial large-scale development model; on the other hand, external economic policy factors and the inspection and balancing of high-tech manufacturing have reduced the export competitiveness of the manufacturing sector. Currently, the norm in manufacturing includes mixed production in factories and workshops, rapid updates to product line models, high similarity in product functions, and complex product labeling. However, if quality inspection continues to rely on manual assembly lines, problems such as high labor costs, high risk of false detection, and low efficiency will persist. For the manufacturing industry, product quality is a crucial indicator for evaluating enterprise capacity. Even minor defects can affect overall production efficiency and profitability. How to effectively improve the efficiency and accuracy of industrial quality inspection is key to the future development of the manufacturing industry.

[0003] Over the past few decades, smart manufacturing based on information systems has improved the productivity and quality of businesses of all sizes. In this smart manufacturing environment, the quality and trends of both machine-generated and human-generated data are constantly improving, and the use of data analysis, statistical modeling, and predictive algorithms is growing exponentially. Machine learning, especially deep learning, has opened up limitless possibilities for quality control tasks on large assembly lines. According to Forbes, machine learning-driven quality optimization lines are projected to grow by 35%, while process visualization and automation are expected to grow by 34%.

[0004] Specifically, in the scenario of quality monitoring for disposable tableware packages, which typically contain chopsticks, spoons, paper towels, and other commonly used disposable utensils, the current inspection method is manual sampling. After the tableware packages are packaged normally on the production line, workers subsequently conduct random checks, manually screening for missing packages. This method tends to rely on tactile judgment and is easily affected by external interference, leading to many misjudgments. Therefore, many companies choose to use machine vision equipment to replace manual labor for some repetitive tasks, enabling timely detection of product defects during the production process and subsequent disposal of waste packages.

[0005] The vision equipment needs to inspect each piece of tableware. If any item is missing, it is considered defective and must be removed. Usually, the disposable chopsticks in a tableware set are a single piece, so only the presence of missing chopsticks needs to be checked. However, higher-end tableware sets provide two separate chopsticks. Only if there are two chopsticks and all other tableware is present can the tableware set be considered acceptable.

[0006] While deep learning methods can detect ordinary conjoined chopsticks, detecting independent chopsticks using ordinary deep learning methods can be problematic. This is because the chopsticks severely occlude each other, resulting in the inclusion of features from other objects within the bounding box when annotating each chopstick. This leads to confusion during feature extraction by the network, causing chaotic and false detections, and ultimately resulting in low accuracy of the deep learning model. Summary of the Invention

[0007] The purpose of this invention is to solve the above-mentioned problems by designing a method for detecting the number of disposable chopsticks based on image processing technology.

[0008] The technical solution of the present invention to achieve the above objectives is a method for detecting the number of disposable chopsticks based on image processing technology, comprising the following steps:

[0009] Step 1: The industrial camera adopts a hard-triggered working mode. After receiving an external trigger command, it outputs a high-resolution image of a disposable tableware pack. Based on this, a training dataset and detection image library of disposable tableware packs are established.

[0010] Step 2: Manually annotate the cutlery sets in the cutlery package dataset images described in Step 1 using annotation tools to create an image label library for the cutlery packages;

[0011] Step 3: Using statistical methods, determine the proportion of occluded bounding boxes in the cutlery bag dataset. When the proportion of occluded bounding boxes is greater than 1 / 2, it indicates that there is a large amount of occlusion between objects in this dataset, and subsequent ensemble segmentation algorithms are required.

[0012] Step 4: Build a deep learning object detection model. The network structure consists of convolutional layers, batch normalization layers, pooling layers, and linear activation layers. Post-processing is changed to use instance masks for pixel-level NMS processing.

[0013] Step 5: Expand the tableware package image library acquired in Step 1 using data augmentation techniques;

[0014] Step 6: Train a deep learning model for object detection on the expanded tableware package image library from Step 5;

[0015] Step 7: Use a clustering algorithm to perform color clustering on the cutlery bag image to verify whether the color features of the chopsticks region are suitable for using an ensemble segmentation algorithm;

[0016] Step 8: Extract the target ROI from the cutlery bag image to reduce the interference of external noise on the target object features; convert the image from BGR color space to HSV color space, use a determined threshold to perform instance segmentation of disposable chopsticks to obtain a binary image with chopsticks as the foreground, and use morphological methods for further processing.

[0017] Step 9: Vertically segment the binary image with chopsticks as the foreground obtained in Step 8 using a fixed step size. Count the number of foreground pixels within each rectangle in the vertical direction and arrange them in descending order to obtain an ordered array. Make a preliminary judgment by calculating the segmented pixel values ​​in the vertical direction and count their intervals. Make a secondary judgment by dynamically adjusting the parameters to determine the number of chopsticks. Dynamic parameter adjustment means that when the system is running stably, the threshold used for segmentation also needs to be stable. If there is a very large deviation between the two, it indicates that the system has an error. Therefore, a dynamic adjustment mechanism is needed to continuously adjust the value of N based on the threshold of the previous segmentation and the value of the current segmentation. Finally, the value of N is used to determine whether the number of chopsticks is one, two, or more than three.

[0018] Step 10: Establish a fully connected network structure, fit the functional relationship between the target detection model results, the vertical segmentation pixel value determination results, and the real label values, and use the expanded cutlery bag image library from Step 5 for deep learning training to fit the relationship between the deep learning detection results of the number of chopsticks and the traditional method detection results of the number of chopsticks.

[0019] Step 11: Collect cutlery bag images online. Detect the cutlery bag images based on the target detection model trained in Step 6, including the number of tissues, spoons, and chopsticks. Then, use a vertical pixel segmentation algorithm combined with dynamically adjusted pixel thresholds to determine the number of chopsticks. Finally, output the final number of chopsticks through the fully connected network trained in Step 10.

[0020] Step 1 specifically includes:

[0021] The cutlery bag inspection image library was acquired by a 500W pixel industrial camera, with an image resolution of 2048*2448. The ratio of normal image samples to defective image samples in the cutlery bag inspection image library is close to 1:1, and the total number of samples in the image library is N, where N>4000.

[0022] Step 3 specifically includes:

[0023] Iterate through all the real labels in the database. If the Intersection over Union (IOU) between a bounding box and another bounding box exceeds a set threshold (IOU threshold is set to 0.3), it means that the bounding box is severely occluded. If the final statistical result shows that more than half of the bounding boxes are severely occluded, then the ensemble segmentation algorithm described above should be used for this dataset.

[0024] Step 4 specifically includes:

[0025] The object detection model consists of convolutional layers, batch normalization layers, pooling layers, and non-linear activation layers. Convolutional layers are used to extract depth features from the input image, which are represented by each pixel in the image through combination or independent methods. Pooling layers are used to select the features extracted by the convolutional layers. Convolutional layers and upsampling layers are used to ensure that the resolution of the resulting image is the same as that of the input image.

[0026] The improved NMS processing utilizes the instance mask output by the neural network for pixel-level NMS, using a more refined bounding box as the basis for NMS. This is used to calculate the intersection-union ratio (IUU) of two object instances. The bounding box is the maximum bounding rectangle of a target, so the minimum bounding rectangle of the target is used to describe the object. In YOLOv5-seg, the mask is described as a 0-1 binary matrix. Polygon bounding boxes are obtained through edge extraction, allowing the binary matrix of the mask to be obtained directly during the computation. Then, matrix calculations are used to calculate the intersection and union, thus performing pixel-level NMS. Using the same set of weight files for comparison, the mask-based NMS is more effective than the bounding box-based NMS in addressing the problem of missed detection of tilted and slender targets.

[0027] Step 5 specifically includes:

[0028] Data augmentation methods refer to methods that increase the amount of data by adding minor modifications to existing data or creating new synthetic data from existing data. These methods include image scaling, image cropping, image flipping, image blurring, affine transformation, random Gaussian noise perturbation, and random image rotation, with the random rotation angle ranging from ±30°.

[0029] Step 6 specifically includes:

[0030] First, the dataset augmented in step 5 is randomly divided into training set, validation set and test set in a ratio of 7:2:1.

[0031] Next, the deep learning object detection model from step 4 is iterated on the training set divided on S1 for a maximum of 300 rounds. If the loss value during training has converged, training is stopped. Finally, the model that performs best on the validation set is selected as the subsequent object detection model.

[0032] Step 7 specifically includes:

[0033] Clustering algorithms use color histograms to represent color features in an image. In this task, chopsticks, tissues, spoons, and the background are distinctly different colors. The histogram of the original image clustered by the clustering algorithm shows four color blocks, indicating that the algorithm successfully distinguished four completely different colors in the original image. Therefore, it was determined that the chopsticks region could be segmented using color features.

[0034] Step 8 specifically includes:

[0035] S1: Target ROI extraction, which involves cropping out the area outside the image conveyor belt and keeping only the area inside the conveyor belt;

[0036] S2: HSV color thresholding segmentation converts the image from the BGR color space to the HSV color space and extracts the chopsticks portion of the image according to a specified threshold, resulting in a mask image with the chopsticks as the foreground, i.e., the chopsticks portion is white and the background portion is black.

[0037] S3: Morphological method, which uses structuring elements with a certain shape to measure and extract the corresponding shape in the image, remove external interference points, fill the internal holes of the chopsticks, and preserve edge pixels as much as possible. The result is obtained by performing one opening operation and one closing operation on a struct with a kernel of 3.

[0038] Step 9 specifically includes:

[0039] Determining the number of chopsticks by vertically segmenting pixels is the foundation of a constrained chopstick pixel counting algorithm. Specifically, since the conveyor belt position is fixed, the vertical movement of the chopsticks is constrained. Assuming the conveyor belt width is H and the chopstick length is L, taking the horizontal direction as the positive direction, the maximum angle that the chopsticks can rotate in the direction of the chopsticks can be calculated as α = cos^(-1) (H / L). Therefore, the chopstick angle should be within the range of (-α, α). Even with occlusion, the spoon cannot completely cover the chopsticks in the most extreme case. Therefore, sampling in the vertical direction always results in all chopsticks being exposed. Based on this, an algorithm for segmentation based on a color threshold is designed. A preliminary judgment is made by calculating the pixel value of each block after vertical segmentation, and its interval is counted. A secondary judgment is made by dynamically adjusting the parameters. Then, in the processed image, the number of 255 pixels in each rectangle is counted in the vertical direction and arranged in descending order to obtain an ordered array. Based on the pre-set maximum pixel threshold, it can be determined whether it belongs to a single chopstick, two chopsticks, or three chopsticks. The specific calculation process is as follows:

[0040] S1: Assuming the camera imaging plane is parallel to the conveyor belt, the maximum number of pixels in the vertical direction of a single chopstick is N. This is because the chopstick is exactly in the same direction as the conveyor belt. Therefore, the minimum number of pixels can be considered to be N*cosα.

[0041] S2: Assuming a is the sampling step size, 1 ≪ a ≪ L, then based on the image width L, we can obtain n = L / a small rectangles for horizontal cutting. Count the number of pixels in each small rectangle and sort them to obtain the array Ci, where i is the index of the small rectangle.

[0042] S3: Assuming that the highest threshold is initially determined to be three chopsticks, the threshold interval should be [3 * a * N * cosα, 3 * a * N], denoted as A. The threshold interval that does not belong to single chopsticks or double chopsticks is [2 * a * N, 3 * a * N], denoted as B. Each of the three cases corresponds to its own A and B, and A and B should be at the maximum, that is, single chopsticks will not have the cases of 2 * a * N and 3 * a * N.

[0043] S4: Calculate the proportions within the above intervals based on the array, obtaining the proportion within region A as p1 and the proportion within region B as p2, if and only if p1 / p2 > p t 0 <p t <1, p t The outliers used to control segmentation errors are considered to be three chopsticks;

[0044] The purpose of introducing an adaptive dynamic threshold adjustment strategy is to address the problem that while pt is used to measure the impact of outliers on the results, the threshold cannot be dynamically adjusted. Specifically, if environmental factors change during program execution, causing deviations in the h, s, and v values ​​used for segmentation in the HSV color space, the results will inevitably be affected. Furthermore, the segmentation threshold needs to remain stable during stable system operation. If two very large deviations occur, it indicates a system error. Therefore, a dynamic adjustment mechanism is needed to continuously adjust the value of N based on the previous segmentation threshold and the current segmentation value. The specific process is as follows:

[0045] S1: First, assume there is only one chopstick. The initial statistical interval is A = [a * N0 * cosα, a * N0], B = [0, a * N0]. We need to calculate the probability P that the chopstick belongs to the same person. n = p1 / p2> p t That is, it is considered to belong to one chopstick;

[0046] S2: Calculate the average N at each segmentation. h= H(h, s, v), where H is the observation function, which characterizes the segmented observation results of N;

[0047] S3: Assume that the probability of each segmentation result falling within a segmentation interval follows a normal distribution. Let Nn-1 be the probability predicted based on the previous frame, and N be the probability predicted based on the previous result. p = a * N n-1 'a' is the adjustment factor;

[0048] S4: Define the error as e - = N n - N n-1 , where N n-1 = N p(n-1) + K(N h(n-1) – H * N p(n-1) ), representing the previous result, where K characterizes the proportion of the current segmentation observation result; then the average error can be obtained as E. n - =E(e - * e -T E was calculated. n = a * E n-1 * a T , a is the adjustment factor, E n-1 This is the error from the previous reading;

[0049] S5: The optimization error function, after differentiation, yields the extreme point K = E. n - * H T (H * E n * H T ) - ;

[0050] S6: From this, we obtain N n = N p + K(N h – H * N p ), and calculate E this time. n-1 ;

[0051] S7: The obtained Nn is the actual calculated value. It takes into account both the result of the previous segmentation and the result of the current segmentation. The most reliable result is obtained through the coefficient K. Because Nn is constrained by the previous result, it will not have a sudden jump. At the same time, it is affected by the current segmented image and can cope with small changes in the environment. If the calculated K value is significantly close to 0 or 1, it indicates that the system has an anomaly or the environment has changed significantly, and an error should be reported.

[0052] Step 10 specifically includes:

[0053] The fully connected network structure and training involves a five-layer fully connected network with the ReLU activation function added after the hidden layers. After the network framework is built, the neural network training model is established, which includes:

[0054] S1: Neural network model building. Generate a model using the constructed framework, given the number of neural units in each network layer;

[0055] S2: Optimizer setup, select the optimizer type, input network parameters, and set the learning rate;

[0056] S3: Definition of the loss function, using mean squared error (MSE);

[0057] S4: The results of the object detection model and the results of the vertical segmentation pixel value determination are used as input values ​​and then fed into the model for deep learning training.

[0058] This invention provides a method for detecting the number of disposable chopsticks based on image processing technology. It employs statistical methods to determine whether the current dataset is suitable for using the integrated algorithm of this patent. The method obtains the number of disposable chopsticks through a convolutional neural network algorithm and pixel-level NMS post-processing. Pixel-level segmentation is performed using traditional methods, and an adaptive threshold calculation method is integrated for secondary determination of the chopsticks number. Finally, a feedforward neural network with a low number of network layers is used to integrate the results of deep learning and traditional methods, achieving fast and accurate detection efficiency and performance. Attached Figure Description

[0059] Figure 1 This is a flowchart of a method for detecting the number of disposable chopsticks based on image processing technology as described in this invention;

[0060] Figure 2 This invention relates to the constrained chopstick pixel statistics algorithm principle of a disposable chopsticks quantity detection method based on image processing technology.

[0061] Figure 3 This is a comparison of the post-processing results of replacing the image processing-based disposable chopsticks quantity detection method described in this invention with pixel-level NMS;

[0062] Figure 4 This is the original image of a disposable tableware package and its color clustering results for a disposable chopsticks quantity detection method based on image processing technology as described in this invention;

[0063] Figure 5 This is a comparison image of the morphological processing results of the disposable chopsticks quantity detection method based on image processing technology described in this invention. Detailed Implementation

[0064] The present invention will now be described in detail with reference to the accompanying drawings, such as... Figure 1-5 As shown, a method for detecting the number of disposable chopsticks based on image processing technology specifically includes the following steps:

[0065] Step 1: The industrial camera adopts a hard-triggered working mode. After receiving an external trigger command, it outputs a high-resolution image of a disposable tableware pack. Based on this, a training dataset and detection image library of disposable tableware packs are established.

[0066] Step 2: Manually annotate the cutlery sets in the cutlery kit dataset images described in Step 1 using an annotation tool to form an image label library for the cutlery kits.

[0067] Step 3: Using statistical methods, analyze the proportion of occluded bounding boxes in the cutlery bag dataset. When the proportion of occluded bounding boxes is greater than 1 / 2, it indicates that there is a large amount of occlusion between objects in this dataset, and subsequent ensemble segmentation algorithms are required.

[0068] Step 4: Build a deep learning object detection model. The network structure consists of convolutional layers, batch normalization layers, pooling layers, linear activation layers, etc. Post-processing is changed to use instance masks for pixel-level NMS processing.

[0069] Step 5: Expand the tableware package image library acquired in Step 1 using data augmentation techniques;

[0070] Step 6: Train a deep learning model for object detection on the expanded tableware package image library from Step 5;

[0071] Step 7: Use a clustering algorithm to perform color clustering on the cutlery bag image to verify whether the color features of the chopsticks region are suitable for using a segmentation algorithm;

[0072] Step 8: Extract the target ROI from the cutlery bag image to reduce the interference of external noise on the target object features; convert the image from BGR color space to HSV color space, use a determined threshold to perform instance segmentation of disposable chopsticks to obtain a binary image with chopsticks as the foreground, and use morphological methods for further processing.

[0073] Step 9: Vertically segment the binary image with chopsticks as the foreground obtained in Step 8 using a fixed step size. Count the number of foreground pixels within each rectangle in the vertical direction and arrange them in descending order to obtain an ordered array. Make a preliminary judgment by calculating the segmented pixel values ​​in the vertical direction and count their intervals. Make a secondary judgment by dynamically adjusting the parameters to determine the number of chopsticks. Introduce a dynamic parameter adjustment strategy. When the system is running stably, the threshold used for segmentation also needs to be stable. If there is a very large deviation between the two, it indicates that the system has an error. Therefore, a dynamic adjustment mechanism is needed to continuously adjust the value of N based on the threshold of the previous segmentation and the value of the current segmentation. Finally, determine whether the number of chopsticks is one, two, or more than three based on the value of N.

[0074] Step 10: Establish a fully connected network structure, fit the functional relationship between the target detection model results, the vertical segmentation pixel value determination results, and the real label values, and use the expanded cutlery bag image library from Step 5 for deep learning training to fit the relationship between the deep learning detection results of the number of chopsticks and the traditional method detection results of the number of chopsticks.

[0075] Step 11: Collect cutlery bag images online. Detect the cutlery bag images based on the target detection model trained in Step 6, including the number of tissues, spoons, and chopsticks. Then, use a vertical pixel segmentation algorithm combined with dynamically adjusted pixel thresholds to determine the number of chopsticks. Finally, output the final number of chopsticks through the fully connected network trained in Step 10.

[0076] The cutlery bag detection image library in step 1 consists of high-resolution images captured by an industrial camera, with a resolution of 2048*2448. The ratio of normal image samples to defective image samples in the image library is close to 1:1, and the total number of samples in the image library is N, where N>4000.

[0077] The statistical method in step 3 involves traversing all real labels in the database. If the IOU between a bounding box and another bounding box exceeds a set threshold (the IOU threshold is set to 0.3), it indicates that the bounding box is severely occluded. If the final statistical result shows that more than half of the bounding boxes are severely occluded, then the dataset needs to use the ensemble algorithm of this invention.

[0078] The target detection model in step 4 consists of convolutional layers, batch normalization layers, pooling layers, and nonlinear activation layers. The convolutional layers are used to extract depth features from the input image, which are represented by each pixel in the image through combination or independent methods. The pooling layers are used to select the features extracted by the convolutional layers. The convolutional layers and upsampling layers are used to ensure that the resolution of the resulting image is the same as that of the input image.

[0079] The improved NMS processing in step 4 utilizes the instance mask output by the neural network for pixel-level NMS, using a more refined bounding box as the basis for NMS calculation to determine the intersection-union ratio (IUU) of two object instances. The bounding box (BBox) is the maximum bounding rectangle of an object, so the minimum bounding rectangle is used to describe the object. In YOLOv5-seg, the mask is described as a 0-1 binary matrix. Polygon bounding boxes are obtained through edge extraction, allowing the binary matrix of the mask to be directly obtained during computation. Then, matrix calculations are used to calculate the intersection and union, thus performing pixel-level NMS. The following figure shows a comparison using the same weight file; please refer to [link / reference]. Figure 3 The left image (a) shows NMS based on Mask, and the right image (b) shows NMS based on BBox. It can be seen that the NMS based on Mask is effective in solving the problem of missed detection of tilted and slender targets.

[0080] The data augmentation method in step 5 refers to the method of increasing the amount of data by adding minor modifications to existing data or creating new synthetic data from existing data. This includes image scaling, image cropping, image flipping, image blurring, affine transformation, random Gaussian noise perturbation, and random image rotation, with the random rotation angle ranging from ±30°.

[0081] The deep learning training in step 6 is performed according to the following steps:

[0082] S1: The dataset enhanced in step 5 is randomly divided into training set, validation set and test set in a ratio of 7:2:1.

[0083] S2: Iterate the deep learning object detection model from step 4 on the training set divided in S1 for up to 300 rounds. If the loss value during training has converged, stop training and finally select the model that performs best on the validation set as the subsequent object detection model.

[0084] The clustering algorithm in step 7 uses color histograms to represent the color features in the image. In this task, the chopsticks, tissues, spoons, and background are clearly different colors. Please refer to [link / reference needed]. Figure 4 (b) is the result of color clustering of the original image using a clustering algorithm. The histogram shows four color blocks, which means that the algorithm successfully distinguished four completely different colors in the original image. Therefore, it was determined that the chopsticks region could be segmented by color features.

[0085] The target ROI extraction in step 8 involves cropping out the area outside the image conveyor belt and keeping only the area inside the conveyor belt.

[0086] The HSV color threshold segmentation in step 8 involves converting the image from the BGR color space to the HSV color space and extracting the chopsticks portion of the image according to a specified threshold, resulting in a mask image with the chopsticks as the foreground (i.e., a binarized image where the chopsticks portion is white and the background portion is black).

[0087] Step 8 involves using structuring elements with a specific shape to measure and extract corresponding shapes from the image, removing external interference points, filling the holes inside the chopsticks, and preserving edge pixels as much as possible. This is achieved by performing one opening and one closing operation using a struct with a kernel of 3. Please refer to [link / reference]. Figure 5 (a) shows the HSV thresholding results. Please refer to [link / reference]. Figure 5 (b) shows the image after morphological processing.

[0088] The adaptive interval threshold determination in step 9 is based on a constrained chopstick pixel statistics algorithm. Since the conveyor belt position is fixed, the vertical movement of the chopsticks is constrained. Assuming the conveyor belt width is H and the chopstick length is L, taking the horizontal direction as the positive direction, the maximum angle that the chopsticks can rotate in the direction of the chopsticks can be calculated as α = cos^(-1) (H / L). Therefore, the chopstick angle should be within the range of (-α, α). Even with occlusion, the spoon cannot completely cover the chopsticks in the most extreme case. Therefore, sampling in the vertical direction always results in all chopsticks being exposed. Based on this, an algorithm for segmentation based on a color threshold is designed. A preliminary judgment is made by calculating the pixel value of each block after vertical segmentation, and its interval is counted. A secondary judgment is then made by dynamically adjusting parameters. In the processed image, the number of 255 pixels within each rectangle is counted in the vertical direction and arranged in descending order to obtain an ordered array. Based on the pre-set maximum pixel threshold, it can be determined whether it belongs to a single chopstick, two chopsticks, or three chopsticks. The specific calculation process is as follows:

[0089] S1: Assuming the camera imaging plane is parallel to the conveyor belt, the maximum number of pixels in the vertical direction of a single chopstick is N. This is because the chopstick is exactly in the same direction as the conveyor belt. Therefore, the minimum number of pixels can be considered to be N*cosα.

[0090] S2: Assuming a is the sampling step size (1 ≪ a ≪ L), then based on the image width L, we can obtain n = L / a small rectangles for horizontal cutting. Count the number of pixels in each small rectangle and sort them to obtain the array Ci (i is the index of the small rectangle).

[0091] S3: Assuming that based on the highest threshold, it is initially determined to be three chopsticks, the threshold interval should be [3 * a * N * cosα, 3 * a * N], denoted as A. The threshold interval for chopsticks that do not belong to single chopsticks or double chopsticks is [2 * a * N, 3 * a * N], denoted as B. Each of the three cases corresponds to its own A and B, and A and B should be at their maximum, meaning that single chopsticks will not result in the cases of 2 * a * N and 3 * a * N.

[0092] S4: Calculate the proportions within the above intervals based on the array, obtaining the proportion within region A as p1 and the proportion within region B as p2, if and only if p1 / p2 > p t (0 <p t <1), p t If an outlier is detected (used to control outliers caused by segmentation errors), it is considered to be three chopsticks.

[0093] The purpose of introducing adaptive interval threshold determination in step 9 is to solve the problem of introducing p. t This method is used to measure the impact of outliers on the results, but it cannot dynamically adjust the threshold value. If environmental factors change during program execution, causing deviations in the h, s, and v values ​​used for segmentation in the HSV color space, the results will inevitably be affected. Furthermore, the segmentation threshold needs to be stable during stable system operation. If two very large deviations occur, it indicates an error in the system. Therefore, a dynamic adjustment mechanism is needed to continuously adjust the value of N based on the previous segmentation threshold and the current segmentation value. The specific process is as follows:

[0094] S1: First, assume there is only one chopstick. The initial statistical interval is A = [a * N0 * cosα, a * N0], B = [0, a * N0]. If the probability value Pn = p1 / p2 > pt is calculated to belong to one chopstick, then it is considered to belong to one chopstick.

[0095] S2: At each segmentation, the average Nh = H(h, s, v) is calculated, where H is the observation function, which characterizes the segmentation observation results of N;

[0096] S3: Assume that the probability of each segmentation result falling within a segmentation interval follows a normal distribution. Let N be the probability predicted from the previous frame. n-1 N predicted based on the previous results p = a * N n-1 , where 'a' is the adjustment factor.

[0097] S4: Define the error as e - = N n - N n-1 , where Nn-1 = N p(n-1) + K(N h(n-1) – H * N p(n-1) ), representing the previous result, where K characterizes the proportion of the current segmentation observation result. The average error can then be obtained as E. n - =E(e - * e -T E was calculated. n = a * E n-1 * a T , a is the adjustment factor, E n-1 This is the error from the previous reading;

[0098] S5: The optimization error function, after differentiation, yields the extreme point K = E. n - * H T (H * E n * H T ) - ;

[0099] S6: From this, we obtain N n = N p + K(N h – H * N p ), and calculate E this time. n - ;

[0100] S7: The obtained Nn is the actual calculated value. It considers both the results of the previous segmentation and the results of the current segmentation, and obtains the most reliable result through the coefficient K. Nn is constrained by the previous result, preventing sudden jumps, and is also affected by the current segmented image, so it can cope with small changes in the environment. In particular, if the calculated K value is significantly close to 0 or 1, it indicates that the system has encountered an anomaly or that the environment has changed significantly, and an error should be reported.

[0101] The fully connected network structure and training in step 10 is a five-layer fully connected network structure with a ReLU activation function added after the hidden layers. After the network framework is built, the neural network training model is established, which mainly includes three parts:

[0102] S1: Neural network model building. Generate a model using the constructed framework, given the number of neural units in each network layer;

[0103] S2: Optimizer setup, select the optimizer type, input network parameters, and set the learning rate;

[0104] S3: Definition of the loss function, using mean squared error (MSE);

[0105] S4: The results of the object detection model and the results of the vertical segmentation pixel value determination are used as input values ​​and then used for deep learning training after being input into the model.

[0106] The above technical solutions only embody the preferred technical solutions of the present invention. Any modifications that may be made by those skilled in the art to certain parts thereof embody the principles of the present invention and fall within the protection scope of the present invention.

Claims

1. A method for detecting the number of disposable chopsticks based on image processing technology, comprising the following steps: Step 1: The industrial camera adopts a hard-triggered working mode. After receiving an external trigger command, it outputs a high-resolution image of a disposable tableware pack. Based on this, a training dataset and detection image library of disposable tableware packs are established. Step 2: Manually annotate the cutlery sets in the cutlery package dataset images described in Step 1 using annotation tools to create an image label library for the cutlery packages; Step 3: Using statistical methods, determine the proportion of occluded bounding boxes in the cutlery bag dataset. When the proportion of occluded bounding boxes is greater than 1 / 2, it indicates that there is a large amount of occlusion between objects in this dataset, and subsequent ensemble segmentation algorithms are required. Step 4: Build a deep learning object detection model. The network structure consists of convolutional layers, batch normalization layers, pooling layers, and linear activation layers. Post-processing is changed to use instance masks for pixel-level NMS processing. Step 5: Expand the tableware package image library acquired in Step 1 using data augmentation techniques; Step 6: Train a deep learning model for object detection on the expanded tableware package image library from Step 5; Step 7: Use a clustering algorithm to perform color clustering on the cutlery bag image to verify whether the color features of the chopsticks region are suitable for using an ensemble segmentation algorithm; Step 8: Extract the target ROI from the cutlery bag image to reduce the interference of external noise on the target object features; convert the image from BGR color space to HSV color space, use a determined threshold to perform instance segmentation of disposable chopsticks to obtain a binary image with chopsticks as the foreground, and use morphological methods for further processing. Step 9: Vertically segment the binary image with chopsticks as the foreground obtained in Step 8 using a fixed step size. Count the number of foreground pixels within each rectangle in the vertical direction and arrange them in descending order to obtain an ordered array. Make a preliminary judgment by calculating the segmented pixel values ​​in the vertical direction and count their intervals. Make a secondary judgment by dynamically adjusting the parameters to determine the number of chopsticks. Dynamic parameter adjustment means that when the system is running stably, the threshold used for segmentation also needs to be stable. If there is a very large deviation between the two, it indicates that the system has an error. Therefore, a dynamic adjustment mechanism is needed to continuously adjust the value of N based on the threshold of the previous segmentation and the value of the current segmentation. Finally, the value of N is used to determine whether the number of chopsticks is one, two, or more than three. Step 10: Establish a fully connected network structure, fit the functional relationship between the target detection model results, the vertical segmentation pixel value determination results, and the real label values, and use the expanded cutlery bag image library from Step 5 for deep learning training to fit the relationship between the deep learning detection results of the number of chopsticks and the traditional method detection results of the number of chopsticks. Step 11: Collect cutlery bag images online. Detect the cutlery bag images based on the target detection model trained in Step 6, including the number of tissues, spoons, and chopsticks. Then, use a vertical pixel segmentation algorithm combined with dynamically adjusted pixel thresholds to determine the number of chopsticks. Finally, output the final number of chopsticks through the fully connected network trained in Step 10.

2. The method for detecting the number of disposable chopsticks based on image processing technology according to claim 1, characterized in that, Step 1 specifically includes: The cutlery bag inspection image library was acquired by a 500W pixel industrial camera, with an image resolution of 2048*2448. The ratio of normal image samples to defective image samples in the cutlery bag inspection image library is close to 1:1, and the total number of samples in the image library is N, where N>4000.

3. The method for detecting the number of disposable chopsticks based on image processing technology according to claim 1, characterized in that, Step 3 specifically includes: Iterate through all the real labels in the database. If the Intersection over Union (IOU) between a bounding box and another bounding box exceeds a set threshold (IOU threshold is set to 0.3), it means that the bounding box is severely occluded. If the final statistical result shows that more than half of the bounding boxes are severely occluded, then the ensemble segmentation algorithm described above should be used for this dataset.

4. The method for detecting the number of disposable chopsticks based on image processing technology according to claim 1, characterized in that, Step 4 specifically includes: The object detection model consists of convolutional layers, batch normalization layers, pooling layers, and non-linear activation layers. Convolutional layers are used to extract depth features from the input image, which are represented by each pixel in the image through combination or independent methods. Pooling layers are used to select the features extracted by the convolutional layers. Convolutional layers and upsampling layers are used to ensure that the resolution of the resulting image is the same as that of the input image. The improved NMS processing utilizes the instance mask output by the neural network for pixel-level NMS, using a more refined bounding box as the basis for NMS. This is used to calculate the intersection-union ratio (IUU) of two object instances. The bounding box is the maximum bounding rectangle of a target, so the minimum bounding rectangle of the target is used to describe the object. In YOLOv5-seg, the mask is described as a 0-1 binary matrix. Polygon bounding boxes are obtained through edge extraction, allowing the binary matrix of the mask to be obtained directly during the computation. Then, matrix calculations are used to calculate the intersection and union, thus performing pixel-level NMS. Using the same set of weight files for comparison, the mask-based NMS is more effective than the bounding box-based NMS in addressing the problem of missed detection of tilted and slender targets.

5. The method for detecting the number of disposable chopsticks based on image processing technology according to claim 1, characterized in that, Step 5 specifically includes: Data augmentation methods refer to methods that increase the amount of data by adding minor modifications to existing data or creating new synthetic data from existing data. These methods include image scaling, image cropping, image flipping, image blurring, affine transformation, random Gaussian noise perturbation, and random image rotation, with the random rotation angle ranging from ±30°.

6. The method for detecting the number of disposable chopsticks based on image processing technology according to claim 1, characterized in that, Step 6 specifically includes: First, the dataset augmented in step 5 is randomly divided into training set, validation set and test set in a ratio of 7:2:

1. Next, the deep learning object detection model from step 4 is iterated on the training set divided on S1 for a maximum of 300 rounds. If the loss value converges during the training process, the training is stopped. Finally, the model that performs best on the validation set is selected as the subsequent object detection model.

7. The method for detecting the number of disposable chopsticks based on image processing technology according to claim 1, characterized in that, Step 7 specifically includes: Clustering algorithms use color histograms to represent color features in an image. In this task, chopsticks, tissues, spoons, and the background are distinctly different colors. The histogram of the original image clustered by the clustering algorithm shows four color blocks, indicating that the algorithm successfully distinguished four completely different colors in the original image. Therefore, it was determined that the chopsticks region could be segmented using color features.

8. The method for detecting the number of disposable chopsticks based on image processing technology according to claim 1, characterized in that, Step 8 specifically includes: S1: Target ROI extraction, which involves cropping out the area outside the image conveyor belt and keeping only the area inside the conveyor belt; S2: HSV color thresholding segmentation converts the image from the BGR color space to the HSV color space and extracts the chopsticks portion of the image according to a specified threshold, resulting in a mask image with the chopsticks as the foreground, i.e., the chopsticks portion is white and the background portion is black. S3: Morphological method, which uses structuring elements with certain shapes to measure and extract corresponding shapes in an image, remove external interference points, fill the holes inside the chopsticks, and preserve edge pixels as much as possible. The result is obtained by performing one opening operation and one closing operation on a struct with a kernel of 3.

9. The method for detecting the number of disposable chopsticks based on image processing technology according to claim 1, wherein step 9 specifically comprises: Determining the number of chopsticks by vertically segmenting pixels is the foundation of a constrained chopstick pixel counting algorithm. Specifically, since the conveyor belt position is fixed, the vertical movement of the chopsticks is constrained. Assuming the conveyor belt width is H and the chopstick length is L, taking the horizontal direction as the positive direction, the maximum angle that the chopsticks can rotate in the direction of the chopsticks can be calculated as α = cos^(-1) (H / L). Therefore, the chopstick angle should be within the range of (-α, α). Even with occlusion, the spoon cannot completely cover the chopsticks in the most extreme case. Therefore, sampling in the vertical direction always results in all chopsticks being exposed. Based on this, an algorithm for segmentation based on a color threshold is designed. A preliminary judgment is made by calculating the pixel value of each block after vertical segmentation, and its interval is counted. A secondary judgment is made by dynamically adjusting the parameters. Then, in the processed image, the number of 255 pixels in each rectangle is counted in the vertical direction and arranged in descending order to obtain an ordered array. Based on the pre-set maximum pixel threshold, it can be determined whether it belongs to a single chopstick, two chopsticks, or three chopsticks. The specific calculation process is as follows: S1: Assuming the camera imaging plane is parallel to the conveyor belt, the maximum number of pixels in the vertical direction of a single chopstick is N. This is because the chopstick is exactly in the same direction as the conveyor belt. Therefore, the minimum number of pixels can be considered to be N*cosα. S2: Assuming a is the sampling step size, 1 ≪ a ≪ L, then based on the image width L, we can obtain n = L / a small rectangles for horizontal cutting. Count the number of pixels in each small rectangle and sort them to obtain the array Ci, where i is the index of the small rectangle. S3: Assuming that the highest threshold is initially determined to be three chopsticks, the threshold interval should be [3 * a * N * cosα, 3 * a * N], denoted as A. The threshold interval that does not belong to single chopsticks or double chopsticks is [2 * a * N, 3 * a * N], denoted as B. Each of the three cases corresponds to its own A and B, and A and B should be at the maximum, that is, single chopsticks will not have the cases of 2 * a * N and 3 * a * N. S4: Calculate the proportions within the above intervals based on the array, obtaining the proportion within region A as p1 and the proportion within region B as p2, if and only if p1 / p2 > p t 0 <p t <1, p t The outliers used to control segmentation errors are considered to be three chopsticks; The purpose of introducing an adaptive dynamic threshold adjustment strategy is to address the problem that while pt is used to measure the impact of outliers on the results, the threshold cannot be dynamically adjusted. Specifically, if environmental factors change during program execution, causing deviations in the h, s, and v values ​​used for segmentation in the HSV color space, the results will inevitably be affected. Furthermore, the segmentation threshold needs to remain stable during stable system operation. If two very large deviations occur, it indicates a system error. Therefore, a dynamic adjustment mechanism is needed to continuously adjust the value of N based on the previous segmentation threshold and the current segmentation value. The specific process is as follows: S1: First, assume there is only one chopstick, and the initial statistical interval is A = [a * N0 * cosα, a * N0], B = [0, a * N0]. If we calculate the probability P that the chopstick belongs to the same person... n = p1 / p2> p t That is, it is considered to belong to one chopstick; S2: Calculate the average N at each segmentation. h = H(h, s, v), where H is the observation function, which characterizes the segmented observation results of N; S3: Assume that the probability of each segmentation result falling within a segmentation interval follows a normal distribution. Let Nn-1 be the probability predicted based on the previous frame, and N be the probability predicted based on the previous result. p = a * N n-1 'a' is the adjustment factor; S4: Define the error as e - = N n - N n-1 , where N n-1 = N p(n-1) + K(N h(n-1) – H * N p(n-1) ), representing the previous result, where K characterizes the proportion of the current segmentation observation result; then the average error can be obtained as E. n - =E(e - * e -T ), calculate E n = a * E n-1 *a T , a is the adjustment factor, E n-1 This is the error from the previous reading; S5: The optimization error function, after differentiation, yields the extreme point K = E. n - * H T (H * E n * H T ) - ; S6: From this, we obtain N n = N p + K(N h – H * N p ), and calculate E this time. n-1 ; S7: The obtained Nn is the actual calculated value. It takes into account both the result of the previous segmentation and the result of the current segmentation. The most reliable result is obtained through the coefficient K. Because Nn is constrained by the previous result, it will not have a sudden jump. At the same time, it is affected by the current segmented image and can cope with small changes in the environment. If the calculated K value is significantly close to 0 or 1, it indicates that the system has an anomaly or the environment has changed significantly, and an error should be reported.

10. The method for detecting the number of disposable chopsticks based on image processing technology according to claim 1, characterized in that, Step 10 specifically includes: The fully connected network structure and training involves a five-layer fully connected network with the ReLU activation function added after the hidden layers. After the network framework is built, the neural network training model is established, which includes: S1: Neural network model building. Generate a model using the constructed framework, given the number of neural units in each network layer; S2: Optimizer setup, selecting the optimizer type, passing in network parameters, and setting the learning rate; S3: Definition of the loss function, using mean squared error (MSE); S4: The results of the object detection model and the results of the vertical segmentation pixel value determination are used as input values ​​and then used for deep learning training after being input into the model.

Images