Method and system for detecting carton state in sheet material packaging process

Abstract

The invention provides a sheet material packaging process carton state detection method comprising the following steps: collecting carton images and weight information of different stations, and obtaining robot position information; marking the collected carton image to obtain a carton image data set; constructing a carton region semantic segmentation model, inputting the carton image data set into the carton region semantic segmentation model for model training and optimization, and obtaining an optimal carton region semantic segmentation model; according to the method, carton positioning, cover plate angle and filling information in the sheet material packaging process can be accurate, meanwhile, the phenomenon of non-uniform blanking can be avoided, non-uniform density, caking and caking of the sheet materials are prevented, and the quality of sheet material packaged finished products is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of carton detection for packaging of thin sheet materials, and in particular to a method and system for detecting carton status in a process of packaging thin sheet materials. Background Art

[0002] Reconstituted tobacco, also known as tobacco flakes or reconstituted tobacco, is a type of flake material. It mainly utilizes tobacco by-products and tobacco waste such as tobacco stems, tobacco dust, fragments, shreds, and even low-grade tobacco leaves generated during the processing of tobacco leaves and tobacco products. It is processed according to certain formula requirements and through a specific process to produce a kraft paper-like reconstituted tobacco product. It is made from tobacco-flavored raw materials and its physical and chemical properties are close to or even better than those of natural tobacco leaves. It is usually used as a leaf group formula raw material or filler in tobacco products.

[0003] Finished sheet material sheet packaging is an important process in the production of papermaking sheet materials. After the finished sheet is cut, it is transported to the baler through a conveyor belt for compression and packaging. The packaging process is as follows: first seal the bottom of the inner carton, then put on the outer carton, then adjust the angle of the outer carton cover, and then put a plastic bag in the carton, fill it with sheet fragments again, and finally seal the plastic bag and the outer carton cover.

[0004] A tobacco intelligent packaging machine and quality tracing method with publication number CN108272123B includes a frame, a movable pressing head is provided on the frame, a camera for photographing tobacco is provided on the pressing head or on the frame on one side of the pressing head, during the loading process, the camera is started to record a video of the tobacco loading process; the video is compressed and stored in a database, and an identification code is established, and the identification code is bound to the video and tobacco data information; after packaging is completed, the identification code is output and associated with the packaged tobacco, and the remote terminal retrieves and obtains relevant video data through the identification code to facilitate quality tracing in subsequent processes.

[0005] The existing packaging process has many steps, and the position of cartons at each workstation is mainly determined by limiters. There is no targeted research on the status of cartons. During the packaging process, the normal packaging process is often affected due to inaccurate carton positioning, incorrect cover angles and lack of filler information, and even rework or shutdown. In addition, when the finished slices are transported from the conveyor belt to the packaging machine, uneven material falling will lead to high and low packaging after packaging, uneven density, agglomeration and cake formation, which will affect the quality of the packaged products. Summary of the invention

[0006] In view of this, the present invention proposes a method and system for detecting the status of a carton in a thin material packaging process, which solves the problems of inaccurate carton positioning, incorrect cover angle and lack of filler information in the thin material packaging process. At the same time, it avoids uneven material falling, prevents the thin material from having uneven density, agglomeration and cake formation, and improves the quality of the finished product of thin material packaging.

[0007] In one aspect, the present invention provides a method for detecting a carton status during a sheet material packaging process, comprising the following steps:

[0008] S1, collects carton images and weight information at different workstations, and obtains robot position information;

[0009] S2, annotating the collected carton images to obtain a carton image dataset;

[0010] S3, constructing a carton area semantic segmentation model, inputting the carton image dataset into the carton area semantic segmentation model for model training and optimization, and obtaining the best carton area semantic segmentation model;

[0011] S4, set the standard feature information and weight range threshold of the carton image corresponding to each workstation, use the optimal carton area semantic segmentation model to identify the carton images collected at each workstation, extract the carton image feature information, and judge whether the extracted carton image feature information and weight information are consistent with the standard feature information and weight range threshold of the carton image at the corresponding workstation. If not, feedback is given to the robot for adjustment.

[0012] On the basis of the above technical scheme, preferably, step S1 collects carton images and weight feature information of different workstations, and obtains robot position information, wherein the thin material packaging includes 4 workstations, workstation No. 1 is the carton state after the bottom of the inner box is sealed, workstation No. 2 is the carton state after the outer box is put into the inner box, workstation No. 3 is the carton state after the thin material is filled, and workstation No. 4 is the carton state after the top of the outer box is sealed, and each workstation is provided with an industrial camera and a weight sensor, and the overhead view image of the carton at each workstation is collected by the industrial camera, and the weight information of the carton at each workstation is collected by the weight sensor.

[0013] On the basis of the above technical solution, preferably, the step S1 also includes preprocessing the collected carton image, converting the carton image coordinate axis into the robot coordinate axis with the robot position as the origin, wherein the preprocessing includes cropping the ROI area, the ROI area includes the carton area, and the ROI area of ​​the image is determined according to the edge of the carton, and the size of the ROI area is 50-100 pixels larger than the edge of the carton.

[0014] On the basis of the above technical solution, preferably, in step S2, the collected carton images are annotated to obtain a carton image dataset, and the annotating of the collected carton images includes manual annotation and automatic annotation, wherein the automatic annotation includes the following sub-steps:

[0015] S21, extracting channels from the preprocessed carton image and converting it into a grayscale image;

[0016] S22, normalizing the pixel values ​​of the grayscale image, calculating the sum of the pixel values ​​of the grayscale images of each channel, and determining the grayscale image of the channel with the largest sum of pixel values;

[0017] S23, calculating a segmentation threshold according to the total value of the pixel values ​​divided by the number of pixels;

[0018] S24, segmenting the grayscale image of the determined channel according to the segmentation threshold, extracting pixels greater than the segmentation threshold, obtaining the carton area, filling all the pixel values ​​of the carton area with 1, and obtaining the semantic segmentation label of the carton area;

[0019] S25, matching and associating the preprocessed carton images and the corresponding carton area semantic segmentation labels, respectively filling them into the data set, and dividing the data set into a training set, a test set, and a validation set.

[0020] On the basis of the above technical solution, preferably, the carton area semantic segmentation model is constructed in step S3, and the carton image data set is input into the carton area semantic segmentation model for model training and optimization to obtain the best carton area semantic segmentation model, which includes the following sub-steps:

[0021] S31, builds a carton area semantic segmentation model based on DBS deep separable convolution, CBAM attention mechanism and SegFormer network model;

[0022] S32, using deep learning-based image generation network GAN to enhance the data set, and randomly input the enhanced training set data into the carton area semantic segmentation model for training;

[0023] S33, based on the LS-SOMA algorithm, adjust the parameters of the carton area semantic segmentation model to obtain the optimal carton area semantic segmentation model.

[0024] On the basis of the above technical solution, preferably, the network structure of the carton area semantic segmentation model in step S31 includes an encoder and a decoder, wherein:

[0025] The encoder consists of 4 stacked Transformer Block modules. The convolution in each Transformer Block module adopts DBS depth-separable convolution. The input carton image is sampled by each Transformer Block module. Different Transformer Block modules have different sampling feature depths. The fourth TransformerBlock module has the deepest sampling feature depth.

[0026] The decoder includes a CBAM attention mechanism module, a DBS depth-separable convolution unit and a fusion unit; the carton image is processed by the four Transformer Block modules of the encoder, and feature maps of different sampling depths are output respectively; the feature map is enhanced by the CBAM attention mechanism, and the feature map is upsampled by the DBS module; the upsampled features are fused with the features extracted by the TransformerBlock module, and the four TransformerBlock module branches are upsampled and fused to obtain a fused feature map. The fused feature map is adjusted to its output channel by two convolutions with a kernel size of 1×1, and then upsampled to the original image size, and finally a semantic segmentation image is output.

[0027] On the basis of the above technical solution, preferably, the method described in step S33 is to adjust the parameters of the carton area semantic segmentation model based on the LS-SOMA algorithm to obtain the optimal carton area semantic segmentation model, which includes the following sub-steps:

[0028] S331, based on the LS-SOMA self-organizing migration algorithm, the population is initialized to generate NP particles, the particle encoding method is X i (r,h i1 ,h i2 ,h i3 ,h i4 ,ts i , bs i , lr i , mi i ), where i represents the i-th particle, r represents the image reduction factor, and h i1 Indicates the number of neurons in the hidden layer of the first TransformerBlock module, h i2 Indicates the number of neurons in the hidden layer of the second TransformerBlock module, h i3 represents the number of neurons in the hidden layer of the third Transformer Block module, h i4 Indicates the number of neurons in the hidden layer of the fourth Transformer Block module, ts i Indicates the time step of the model input, bsi is the batch size, lr i is the initial learning rate size, mi i is the maximum number of training iterations;

[0029] S332, preset the random perturbation coefficient prt, generate a corresponding perturbation vector PRTVector for each particle in the population, decode each particle in the population, and configure its parameter values ​​in the SegFormer network model;

[0030] S333, input the enhanced test set data into the carton area semantic segmentation model in sequence for training and verification. Whenever the number of repetitions is an integer multiple of the verification threshold, compare the test results output by the model with the images with the same labels in the verification set data, calculate the MIoU value and the MPA value, and return the MIoU and MPA values ​​as the fitness function values ​​of the LS-SOMA algorithm to the LS-SOMA self-organizing migration algorithm;

[0031] S324, sorting the fitness function values ​​of the obtained population, selecting the particle with the smallest fitness function value as the leader, and migrating the particles except the leader, the migration expression is:

[0032] x i+1 =λ*x i *(1-x i )+(4-λ)*sin(π*x i ) / 4;

[0033] In the formula, x is the population particle solution, λ is the chaos coefficient, and its value is 0.5;

[0034] S325, when the maximum number of iterations of the LS-SOMA self-organizing migration algorithm is reached, the value of the last generation leader is the hyperparameter of the optimal semantic segmentation model, that is, the optimal carton area semantic segmentation model is obtained.

[0035] On the basis of the above technical scheme, preferably, the standard feature information of the carton image and the weight range threshold corresponding to each workstation are set in step S4, and the carton images collected at each workstation are recognized by using the optimal carton area semantic segmentation model, and the carton image feature information is extracted. The extracted carton image feature information and weight information are judged to be consistent with the standard feature information of the carton image and the weight range threshold of the corresponding workstation. If they do not meet, feedback is given to the robot for adjustment, including the following sub-steps:

[0036] S41, manually collecting images of the standard state of cartons corresponding to each workstation, obtaining standard feature information of the carton images of each workstation by manual marking, and setting a weight range threshold of the carton corresponding to each workstation;

[0037] S42, inputting the collected carton images into the semantic segmentation model in sequence according to the workstations, and outputting semantic segmentation images of the corresponding workstations;

[0038] S43, performing segmentation processing on the obtained semantic segmentation image by using a corresponding segmentation threshold to obtain a binary image;

[0039] S44, converting the binary image coordinate axis into the robot coordinate axis with the robot position as the origin;

[0040] S45, extract the center and contour features of the inner carton body, the angle of the outer carton cover and the contour features of the closed carton in the corresponding binary images according to different workstations, collect the weight of the cartons at different workstations according to the weight sensor, and judge whether they are consistent with the standard feature information of the carton image of the corresponding workstation and the set carton weight range threshold. If not, it is determined to be an abnormal state and fed back to the robot for adjustment.

[0041] On the basis of the above technical scheme, preferably, the method also includes storing the carton image data of each workstation in sequence and generating QR code information, associating the QR code with all the carton image data in the corresponding carton packaging and storing them in a database, pasting the QR code on the inner and outer surfaces of the corresponding carton, and tracing the carton packaging data according to the QR code.

[0042] On the other hand, the present invention also provides a system for detecting carton status during a sheet material packaging process, the system comprising:

[0043] The acquisition module is used to collect carton images and weight information at different workstations and obtain robot position information;

[0044] Annotation module, used to annotate the collected carton images to obtain a carton image dataset;

[0045] The model training module is used to build a carton area semantic segmentation model, input the carton image dataset into the carton area semantic segmentation model for model training and optimization, and obtain the best carton area semantic segmentation model;

[0046] The judgment module is used to set the standard feature information and weight range threshold of the carton image corresponding to each workstation, use the optimal carton area semantic segmentation model to identify the carton images collected at each workstation, extract the carton image feature information, and judge whether the extracted carton image feature information and weight information are consistent with the standard feature information and weight range threshold of the carton image at the corresponding workstation. If they do not meet the requirements, feedback is given to the robot for adjustment.

[0047] The data transmission method and control system of an embedded remote experiment platform provided by the present invention have the following beneficial effects compared with the prior art:

[0048] 1) By combining the optimal carton area semantic segmentation model with manually set standard feature information and weight range thresholds, the automated, high-precision recognition and anomaly detection of carton packaging on the production line are achieved; this not only improves the efficiency and accuracy of the production line, but also ensures the quality consistency of the cartons and improves the quality of the finished product of thin material packaging;

[0049] 2) By setting the ROI area, the amount of calculation can be reduced, the processing speed can be improved, and the impact of background noise on the recognition results can be reduced. The image coordinate axis can be converted into the robot coordinate axis, which can improve the accuracy and efficiency of subsequent carton recognition and state judgment;

[0050] 3) By combining DBS deep separable convolution, CBAM attention mechanism and SegFormer network model, as well as using GAN for data enhancement and LS-SOMA algorithm for model parameter adjustment, an efficient and accurate carton area semantic segmentation model can be constructed, which can accurately segment the carton area in a given carton image and improve the recognition accuracy;

[0051] 4) Through the LS-SOMA self-organizing migration algorithm, the model can quickly find the optimal hyperparameter combination in the search space, thereby improving the efficiency of model training. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0053] Figure 1 It is a schematic flow chart of a method for detecting carton status in a thin material packaging process of the present invention;

[0054] Figure 2 This is a structural block diagram of a carton status detection system for a thin material packaging process of the present invention;

[0055] Figure 3 It is a schematic diagram of a carton status detection system for a thin-sheet material packaging process of the present invention;

[0056] Figure 4 It is an overall block diagram of the carton area semantic segmentation model of the carton state detection method in the thin material packaging process of the present invention;

[0057] Figure 5 It is a schematic diagram of the robot and ROI area of ​​the carton status detection method in the thin material packaging process of the present invention. DETAILED DESCRIPTION

[0058] The following will be combined with the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0059] like Figure 1-5 The present invention provides a method for detecting the state of a carton in a thin material packaging process, comprising the following steps:

[0060] S1, collects carton images and weight information at different workstations, and obtains robot position information;

[0061] Among them, the packaging of thin materials includes 4 workstations. Workstation No. 1 is the carton state after the bottom of the inner box is sealed, workstation No. 2 is the carton state after the outer box is put into the inner box, workstation No. 3 is the carton state after the thin material is filled, and workstation No. 4 is the carton state after the top of the outer box is sealed. Each workstation is equipped with an industrial camera and a weight sensor. The overhead view image of the carton at each workstation is collected by the industrial camera, and the weight information of the carton at each workstation is collected by the weight sensor.

[0062] Image acquisition: An industrial camera with a lens of appropriate focal length is set up at each workstation to ensure that high-resolution overhead images are obtained in situ from the top of the carton; these images can clearly show the outline, shape, position and possible defects or abnormalities of the carton.

[0063] Weight collection: A weight sensor is installed under each workstation to measure the weight of the carton and the tobacco flakes inside it in real time; this weight information is key to determining whether the carton is filled evenly and whether there are too many or too few tobacco flakes.

[0064] In this embodiment, step S1 also includes preprocessing the collected carton image, converting the carton image coordinate axis into the robot coordinate axis with the robot position as the origin, wherein the preprocessing includes cropping the ROI area, the ROI area includes the carton area, and the ROI area of ​​the image is determined according to the edge of the carton, and the size of the ROI area is 50-100 pixels larger than the edge of the carton.

[0065] It should be noted that the approximate position of the carton in the image is determined first, and a ROI area containing the carton is determined and cropped according to the edge of the carton. The size of the ROI area is 50-100 pixels larger than the edge of the carton to ensure that the carton can be completely contained and has a certain buffer zone. In this embodiment, the size of the ROI area is preferably 70 pixels larger than the edge of the carton. The setting of the ROI area can reduce the amount of calculation, improve the processing speed, and reduce the impact of background noise on the recognition results.

[0066] In this embodiment, the robot is used to carry, position and adjust cartons during the packaging process. Therefore, it is necessary to obtain the robot's position information and use the robot's position as the origin to convert the image coordinate axis into the robot coordinate axis, which can improve the accuracy and efficiency of subsequent carton recognition and status judgment.

[0067] S2, annotating the collected carton images to obtain a carton image dataset;

[0068] The labeling of the collected carton image in step S2 includes manual labeling and automatic labeling, wherein the pre-processed image is automatically labeled by an image processing algorithm.

[0069] It should be noted that the collected carton images are labeled, including manual labeling and automatic labeling. Manual labeling uses Lableme software to manually label the preprocessed images, uses polygonal boxes to mark the carton areas, and obtains label information. Automatic labeling uses image processing algorithms to automatically label the preprocessed images. The carton images collected at each workstation during the packaging process are automatically labeled, and the standard feature information of the carton images at each workstation is manually labeled corresponding to the collected carton images.

[0070] It is understandable that manual labeling is more accurate, but the labeling efficiency is lower. Automatic labeling can quickly generate a large number of labels, but its accuracy is lower than manual labeling. The combination of the two can balance the speed and accuracy of label acquisition, ensuring the model training effect and the accuracy of subsequent judgments.

[0071] Automatic labeling includes the following sub-steps:

[0072] S21, extracting channels from the preprocessed carton image and converting it into a grayscale image;

[0073] S22, normalizing the pixel values ​​of the grayscale image, calculating the sum of the pixel values ​​of the grayscale images of each channel, and determining the grayscale image of the channel with the largest sum of pixel values;

[0074] S23, calculating a segmentation threshold according to the total value of the pixel values ​​divided by the number of pixels;

[0075] S24, segmenting the grayscale image of the determined channel according to the segmentation threshold, extracting pixels greater than the segmentation threshold, obtaining the carton area, filling all the pixel values ​​of the carton area with 1, and obtaining the semantic segmentation label of the carton area;

[0076] S25, matching and associating the preprocessed carton images and the corresponding carton area semantic segmentation labels, respectively filling them into the data set, and dividing the data set into a training set, a test set, and a validation set.

[0077] It should be noted that channel extraction is performed on the preprocessed carton image. The collected carton image in this embodiment is an RGB image, and the red, green and blue channels are extracted respectively. The image of each channel is converted into a grayscale image, and the pixel value of the grayscale image is normalized, and the pixel value is scaled to between 0 and 1. The sum of the pixel values ​​of the grayscale images of each channel is counted, and the channel with the largest sum of pixel values ​​is determined, that is, the channel with the highest brightness. The segmentation threshold is calculated based on the total sum of the pixel values ​​divided by the number of pixels; the segmentation threshold is used for subsequent image segmentation to distinguish the carton area from the background area.

[0078] S3, constructing a carton area semantic segmentation model, inputting the carton image dataset into the carton area semantic segmentation model for model training and optimization, and obtaining the best carton area semantic segmentation model;

[0079] Wherein, step S3 includes the following sub-steps:

[0080] S31, builds a carton area semantic segmentation model based on DBS deep separable convolution, CBAM attention mechanism and SegFormer network model;

[0081] S32, using deep learning-based image generation network GAN to enhance the data set, and randomly input the enhanced training set data into the carton area semantic segmentation model for training;

[0082] S33, based on the LS-SOMA algorithm, adjust the parameters of the carton area semantic segmentation model to obtain the optimal carton area semantic segmentation model.

[0083] It should be noted that DBS deep separable convolution is used to improve the computational efficiency of the model and reduce the number of parameters, and the CBAM attention mechanism is introduced to enhance the model's attention to important areas in the image and improve segmentation accuracy. Based on the SegFormer network model, the advantages of Transformer and convolution are combined to provide efficient and accurate semantic segmentation results; the deep learning-based image generation network GAN is used to enhance the carton image dataset; GAN can generate images similar to the original data, thereby increasing the diversity and quantity of the dataset, which helps to improve the generalization ability of the model. LS-SOMA is used to dynamically adjust the parameter settings until the best carton area semantic segmentation model is found. By combining DBS deep separable convolution, CBAM attention mechanism and SegFormer network model, as well as using GAN for data enhancement and LS-SOMA algorithm for model parameter adjustment, an efficient and accurate carton area semantic segmentation model can be constructed, which can accurately segment the carton area in a given carton image.

[0084] In step S31, the network structure of the carton area semantic segmentation model includes an encoder and a decoder, wherein the encoder includes 4 stacked TransformerBlock modules, and the convolution in each Transformer Block module adopts DBS depth-separable convolution. The input carton image is feature sampled by each TransformerBlock module, and different Transformer Block modules sample feature depths at different depths, and the fourth TransformerBlock module samples features at the deepest depth; the decoder includes a CBAM attention mechanism module, a DBS depth-separable convolution unit and a fusion unit; the carton image is processed by the four Transformer Block modules of the encoder and the feature maps of different sampling depths are output respectively; the feature map is feature enhanced by the CBAM attention mechanism, and the feature map is upsampled by the DBS module; the upsampled features are fused with the features extracted by the Transformer Block module, and the 4 Transformer Block module branches are upsampled, spliced ​​and fused to obtain a fused feature map, and the fused feature map is adjusted to its output channel by two convolutions with a convolution kernel size of 1×1, and then upsampled to the original image size, and finally a semantic segmentation image is output.

[0085] It should be noted that the carton area semantic segmentation model in this embodiment adopts a network structure including an encoder and a decoder. The encoder part includes 4 stacked Transformer Block modules. Different TransformerBlock modules perform feature sampling of different depths on the input image. As the network depth increases, the resolution of the feature map gradually decreases, but the degree of abstraction of the feature gradually increases, especially the fourth Transformer Block module, which has the deepest sampling feature depth and contains the richest semantic information. The convolution in each TransformerBlock module adopts DBS deep separable convolution. This convolution method can significantly reduce the computational amount and parameter amount of the model while maintaining performance. The decoder first enhances the feature map output by the encoder through the CBAM attention mechanism module. The CBAM module can recalibrate features in both spatial and channel dimensions, allowing the model to pay more attention to important areas in the image. The enhanced feature map is upsampled through the DBS deep separable convolution unit to gradually restore the resolution of the feature map. The upsampled features are fused with the features extracted by the corresponding TransformerBlock module, and then the four TransformerBlock module branches are upsampled and spliced ​​to obtain the final fused feature map. The fused feature map is convoluted twice with a convolution kernel size of 1×1 to adjust its output channel number. Finally, the feature map is upsampled to the original image size to obtain the final semantic segmentation image.

[0086] According to the carton area semantic segmentation model in this embodiment, the encoder extracts features of different depths, the decoder uses the CBAM attention mechanism and DBS deep separable convolution to perform feature enhancement and upsampling, and fuses features of different levels through the fusion unit, and finally outputs an accurate semantic segmentation image; this structure combines the advantages of Transformer and deep separable convolution, and uses the CBAM attention mechanism to improve the performance of the model, so that the model can accurately segment the carton area in the carton image.

[0087] The deep learning image generation network GAN in step S32 includes two models, one is the generation model G module, and the other is the discriminant model D module; the task of the generation model is to generate instances that look natural and real and similar to the original data; the task of the discriminant model is to determine whether a given instance looks natural and real or artificially forged, where the real instance comes from the data set and the forged instance comes from the generation model; G module and D module; the G module is a network that generates pictures, it receives a random noise z, and generates pictures through this noise, denoted as G(z). The D module is a discriminant network that determines whether a picture is real or not, its input is n, n represents a picture, and the output D(n) represents the probability that n is a real picture, if it is 1, it means that it is 100% a real picture, and the output is 0, it means that it cannot be a real picture.

[0088] Step S33 includes the following sub-steps:

[0089] S331, based on the LS-SOMA self-organizing migration algorithm, the population is initialized to generate NP particles, the particle encoding method is X i (r,h i1 ,h i2 ,h i3 ,h i4 ,ts i , bs i , lr i , mi i ), where i represents the i-th particle, r represents the image reduction factor, and h i1 Indicates the number of neurons in the hidden layer of the first TransformerBlock module, h i2 Indicates the number of neurons in the hidden layer of the second TransformerBlock module, h i3 represents the number of neurons in the hidden layer of the third Transformer Block module, h i4 Indicates the number of neurons in the hidden layer of the fourth Transformer Block module, ts i Indicates the time step of the model input, bs i is the batch size, lr i is the initial learning rate size, mi i is the maximum number of training iterations;

[0090] S332, preset the random perturbation coefficient prt, generate a corresponding perturbation vector PRTVector for each particle in the population, decode each particle in the population, and configure its parameter values ​​in the SegFormer network model;

[0091] S333, input the enhanced test set data into the carton area semantic segmentation model in sequence for training and verification. Whenever the number of repetitions is an integer multiple of the verification threshold, compare the test results output by the model with the images with the same labels in the verification set data, calculate the MIoU value and the MPA value, and return the MIoU and MPA values ​​as the fitness function values ​​of the LS-SOMA algorithm to the LS-SOMA self-organizing migration algorithm;

[0092] S324, sorting the fitness function values ​​of the obtained population, selecting the particle with the smallest fitness function value as the leader, and migrating the particles except the leader, the migration expression is:

[0093] x i+1 =λ*x i *(1-x i )+(4-λ)*sin(π*x i ) / 4;

[0094] In the formula, x is the population particle solution, λ is the chaos coefficient, and its value is 0.5;

[0095] S325, when the maximum number of iterations of the LS-SOMA self-organizing migration algorithm is reached, the value of the last generation leader is the hyperparameter of the optimal semantic segmentation model, that is, the optimal carton area semantic segmentation model is obtained.

[0096] It should be noted that through the LS-SOMA self-organizing migration algorithm, the model can quickly find the best hyperparameter combination in the search space, thereby improving the efficiency of model training. This automated hyperparameter adjustment process is more efficient and accurate than the traditional manual adjustment method. The LS-SOMA algorithm can ensure that the carton area semantic segmentation model obtains a higher mean intersection-over-union (MIoU) and mean pixel accuracy (MPA) on the test set, and the model can capture boundaries and details more accurately and finely when segmenting carton areas.

[0097] S4, set the standard feature information and weight range threshold of the carton image corresponding to each workstation, use the optimal carton area semantic segmentation model to identify the carton images collected at each workstation, extract the carton image feature information, and judge whether the extracted carton image feature information and weight information are consistent with the standard feature information and weight range threshold of the carton image at the corresponding workstation. If not, feedback is given to the robot for adjustment.

[0098] Step S4 includes the following sub-steps:

[0099] S41, manually collecting images of the standard state of cartons corresponding to each workstation, obtaining standard feature information of the carton images of each workstation by manual marking, and setting a weight range threshold of the carton corresponding to each workstation;

[0100] S42, inputting the collected carton images into the semantic segmentation model in sequence according to the workstations, and outputting semantic segmentation images of the corresponding workstations;

[0101] S43, performing segmentation processing on the obtained semantic segmentation image by using a corresponding segmentation threshold to obtain a binary image;

[0102] S44, converting the binary image coordinate axis into the robot coordinate axis with the robot position as the origin;

[0103] S45, extract the center and contour features of the inner carton body, the angle of the outer carton cover and the contour features of the closed carton in the corresponding binary images according to different workstations, collect the weight of the cartons at different workstations according to the weight sensor, and judge whether they are consistent with the standard feature information of the carton image of the corresponding workstation and the set carton weight range threshold. If not, it is determined to be an abnormal state and fed back to the robot for adjustment.

[0104] It should be noted that the optimal carton area semantic segmentation model is used to identify the carton images collected at each workstation, and whether the carton meets the standard status is judged based on the extracted carton image feature information and weight information. By setting the standard feature information and weight range threshold of each workstation, the quality consistency of the cartons is ensured, and abnormal conditions are identified and fed back in time, so that the robot can make quick adjustments, reduce production losses, and improve the quality of the finished products of thin material packaging.

[0105] This embodiment also includes storing the carton image data of each workstation in sequence and generating QR code information, associating the QR code with all the carton image data in the corresponding carton packaging and storing them in a database, pasting the QR code on the inner and outer surfaces of the corresponding carton, and tracing the carton packaging data according to the QR code.

[0106] It should be noted that after packaging, the image features and weight features of the carton status information are recorded and traced through a QR code, and a detailed inspection report is output, providing a reliable basis for intelligent evaluation and traceability of packaging quality.

[0107] The present invention also provides a carton status detection system for a thin material packaging process, the system comprising:

[0108] The acquisition module is used to collect carton images and weight information at different workstations and obtain robot position information;

[0109] Annotation module, used to annotate the collected carton images to obtain a carton image dataset;

[0110] The model training module is used to build a carton area semantic segmentation model, input the carton image dataset into the carton area semantic segmentation model for model training and optimization, and obtain the best carton area semantic segmentation model;

[0111] The judgment module is used to set the standard feature information and weight range threshold of the carton image corresponding to each workstation, use the optimal carton area semantic segmentation model to identify the carton images collected at each workstation, extract the carton image feature information, and judge whether the extracted carton image feature information and weight information are consistent with the standard feature information and weight range threshold of the carton image at the corresponding workstation. If they do not meet the requirements, feedback is given to the robot for adjustment.

[0112] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for detecting carton status during the packaging process of thin materials, characterized in that: The following steps are involved: S1, collects carton images and weight information at different workstations, and obtains robot position information; S2, annotating the collected carton images to obtain a carton image dataset; S3, constructing a carton area semantic segmentation model, inputting the carton image dataset into the carton area semantic segmentation model for model training and optimization, and obtaining the best carton area semantic segmentation model; S4, set the standard feature information and weight range threshold of the carton image corresponding to each workstation, use the optimal carton area semantic segmentation model to identify the carton images collected at each workstation, extract the carton image feature information, and judge whether the extracted carton image feature information and weight information are consistent with the standard feature information and weight range threshold of the carton image at the corresponding workstation. If not, feedback is given to the robot for adjustment.

2. The method for detecting carton status during the sheet material packaging process according to claim 1, characterized in that: Step S1 describes collecting carton images and weight feature information of different workstations, and obtaining robot position information, wherein the thin material packaging includes 4 workstations, No. 1 workstation is the carton state after the bottom of the inner box is sealed, No. 2 workstation is the carton state after the outer box is put into the inner box, No. 3 workstation is the carton state after the thin material is filled, and No. 4 workstation is the carton state after the top of the outer box is sealed. Each workstation is equipped with an industrial camera and a weight sensor, and the overhead view image of the carton at each workstation is collected by the industrial camera, and the weight information of the carton at each workstation is collected by the weight sensor.

3. The method for detecting carton status during the sheet material packaging process according to claim 2, characterized in that: The step S1 also includes preprocessing the collected carton image, converting the carton image coordinate axis into the robot coordinate axis with the robot position as the origin, wherein the preprocessing includes cropping the ROI area, the ROI area includes the carton area, and the ROI area of ​​the image is determined according to the edge of the carton, and the size of the ROI area is 50-100 pixels larger than the edge of the carton.

4. The method for detecting carton status during the sheet material packaging process according to claim 3, characterized in that: In step S2, the collected carton images are annotated to obtain a carton image dataset. The annotating of the collected carton images includes manual annotation and automatic annotation, wherein the automatic annotation includes the following sub-steps: S21, extracting channels from the preprocessed carton image and converting it into a grayscale image; S22, normalizing the pixel values ​​of the grayscale image, calculating the sum of the pixel values ​​of the grayscale images of each channel, and determining the grayscale image of the channel with the largest sum of pixel values; S23, calculating a segmentation threshold according to the total value of the pixel values ​​divided by the number of pixels; S24, segmenting the grayscale image of the determined channel according to the segmentation threshold, extracting pixels greater than the segmentation threshold, obtaining the carton area, filling all the pixel values ​​of the carton area with 1, and obtaining the semantic segmentation label of the carton area; S25, matching and associating the preprocessed carton images and the corresponding carton area semantic segmentation labels, respectively filling them into the data set, and dividing the data set into a training set, a test set, and a validation set.

5. The method for detecting carton status during the sheet material packaging process according to claim 4, characterized in that: In step S3, a carton area semantic segmentation model is constructed, and the carton image dataset is input into the carton area semantic segmentation model for model training and optimization to obtain the best carton area semantic segmentation model, which includes the following sub-steps: S31, builds a carton area semantic segmentation model based on DBS deep separable convolution, CBAM attention mechanism and SegFormer network model; S32, using deep learning-based image generation network GAN to enhance the data set, and randomly input the enhanced training set data into the carton area semantic segmentation model for training; S33, based on the LS-SOMA algorithm, adjust the parameters of the carton area semantic segmentation model to obtain the optimal carton area semantic segmentation model.

6. The method for detecting carton status during the sheet material packaging process according to claim 5, characterized in that: The carton area semantic segmentation model network structure in step S31 includes an encoder and a decoder, wherein: The encoder consists of 4 stacked Transformer Block modules. The convolution in each Transformer Block module adopts DBS depth-separable convolution. The input carton image is sampled by each Transformer Block module. Different Transformer Block modules have different sampling feature depths. The fourth TransformerBlock module has the deepest sampling feature depth. The decoder includes a CBAM attention mechanism module, a DBS depth-separable convolution unit and a fusion unit; the carton image is processed by the four Transformer Block modules of the encoder, and feature maps of different sampling depths are output respectively; the feature map is enhanced by the CBAM attention mechanism, and the feature map is upsampled by the DBS module; the upsampled features are fused with the features extracted by the TransformerBlock module, and the four TransformerBlock module branches are upsampled and fused to obtain a fused feature map. The fused feature map is adjusted to its output channel by two convolutions with a kernel size of 1×1, and then upsampled to the original image size, and finally a semantic segmentation image is output.

7. The method for detecting carton status during the sheet material packaging process according to claim 6, characterized in that: In step S33, the carton area semantic segmentation model is adjusted based on the LS-SOMA algorithm to obtain the optimal carton area semantic segmentation model, which includes the following sub-steps: S331, based on the LS-SOMA self-organizing migration algorithm, the population is initialized and NP particles are generated. The particle encoding method is X i (r,h i1 ,h i2 ,h i3 ,h i4 ,ts i , bs i , lr i , mi i ), where i represents the i-th particle, r represents the image reduction factor, and h i1 Indicates the number of neurons in the hidden layer of the first TransformerBlock module, h i2 represents the number of neurons in the hidden layer of the second Transformer Block module, h i3 represents the number of neurons in the hidden layer of the third Transformer Block module, h i4 Indicates the number of neurons in the hidden layer of the fourth Transformer Block module, ts i Indicates the time step of the model input, bs i is the batch size, lr i is the initial learning rate size, mi i is the maximum number of training iterations; S332, preset the random perturbation coefficient prt, generate a corresponding perturbation vector PRTVector for each particle in the population, decode each particle in the population, and configure its parameter values ​​in the SegFormer network model; S333, input the enhanced test set data into the carton area semantic segmentation model in sequence for training and verification. Whenever the number of repetitions is an integer multiple of the verification threshold, compare the test results output by the model with the images with the same labels in the verification set data, calculate the MIoU value and the MPA value, and return the MIoU and MPA values ​​as the fitness function values ​​of the LS-SOMA algorithm to the LS-SOMA self-organizing migration algorithm; S324, sorting the fitness function values ​​of the obtained population, selecting the particle with the smallest fitness function value as the leader, and migrating the particles except the leader. The migration expression is: x i+1 =λ*x i *(1-x i )+(4-λ)*sin(π*x i ) / 4; In the formula, x is the population particle solution, λ is the chaos coefficient, and its value is 0.5; S325, when the maximum number of iterations of the LS-SOMA self-organizing migration algorithm is reached, the value of the last generation leader is the hyperparameter of the optimal semantic segmentation model, that is, the optimal carton area semantic segmentation model is obtained.

8. The method for detecting carton status during the sheet material packaging process according to claim 7, characterized in that: Step S4 sets the standard feature information and weight range threshold of the carton image corresponding to each workstation, uses the optimal carton area semantic segmentation model to recognize the carton images collected at each workstation, extracts the carton image feature information, and determines whether the extracted carton image feature information and weight information are consistent with the standard feature information and weight range threshold of the carton image at the corresponding workstation. If they do not meet the requirements, feedback is given to the robot for adjustment, including the following sub-steps: S41, manually collecting images of the standard state of cartons corresponding to each workstation, obtaining standard feature information of the carton images of each workstation by manual marking, and setting a weight range threshold of the carton corresponding to each workstation; S42, inputting the collected carton images into the semantic segmentation model in sequence according to the workstations, and outputting semantic segmentation images of the corresponding workstations; S43, performing segmentation processing on the obtained semantic segmentation image by using a corresponding segmentation threshold to obtain a binary image; S44, converting the binary image coordinate axis into the robot coordinate axis with the robot position as the origin; S45, extract the center and contour features of the inner carton body, the angle of the outer carton cover and the contour features of the closed carton in the corresponding binary images according to different workstations, collect the weight of the cartons at different workstations according to the weight sensor, and judge whether they are consistent with the standard feature information of the carton image of the corresponding workstation and the set carton weight range threshold. If not, it is determined to be an abnormal state and fed back to the robot for adjustment.

9. The method for detecting carton status during the sheet material packaging process according to claim 1, characterized in that: The method also includes storing the carton image data of each workstation in sequence and generating QR code information, associating the QR code with all carton image data in the corresponding carton packaging and storing them in a database, pasting the QR code on the inner and outer surfaces of the corresponding carton, and tracing the carton packaging data according to the QR code.

10. A carton status detection system for thin material packaging process, characterized in that: The system comprises: The acquisition module is used to collect carton images and weight information at different workstations and obtain robot position information; A labeling module is used to label the collected carton images to obtain a carton image dataset; The model training module is used to build a carton area semantic segmentation model, input the carton image dataset into the carton area semantic segmentation model for model training and optimization, and obtain the best carton area semantic segmentation model; The judgment module is used to set the standard feature information and weight range threshold of the carton image corresponding to each workstation, use the optimal carton area semantic segmentation model to identify the carton images collected at each workstation, extract the carton image feature information, and judge whether the extracted carton image feature information and weight information are consistent with the standard feature information and weight range threshold of the carton image at the corresponding workstation. If they do not meet the requirements, feedback is given to the robot for adjustment.

Images