A method and system for detecting the state of a carton in a process of packing a sheet material

Abstract

The present application provides a kind of sheet material packing process carton state detection method, comprising the following steps: collecting the carton image and weight information of different stations, and obtaining robot position information;The collected carton image is labeled, and the carton image dataset is obtained;Carton area semantic segmentation model is constructed, and the carton image dataset is input into the carton area semantic segmentation model for model training and optimization, to obtain the best carton area semantic segmentation model;The method can make the sheet material packing process carton positioning, cover plate angle and filling information accurate, while avoiding uneven material drop phenomenon, preventing sheet material from uneven density, caking and cake, improve the sheet material packing product quality.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of sheet material packaging carton detection, and particularly relates to a sheet material packaging process carton state detection method and system. BACKGROUND

[0002] Reconstituted tobacco, also known as tobacco sheet or reconstituted tobacco, is a kind of sheet material, which is mainly made of tobacco stems, tobacco dust, scraps, broken filaments, even low-grade tobacco leaves and other tobacco by-products and waste generated during the processing of tobacco leaves and tobacco products, according to certain formula requirements, through specific process processing, it is a kind of reconstituted tobacco product similar to kraft paper, which is processed from tobacco flavor raw materials and its physicochemical properties are close to or even better than natural tobacco leaves, and it is usually used as a leaf formula raw material or filler in tobacco products.

[0003] Finished sheet material sheet packaging is an important process in the production of papermaking sheet material. After the finished sheet is cut, it is conveyed to the packaging machine by the conveying belt for compression packaging. The packaging process is as follows: first, seal the bottom of the inner carton, then put on the outer carton, then adjust the angle of the outer carton cover plate, then lay the plastic bag in the carton, then fill the sheet scraps, and finally seal the plastic bag and the outer carton cover plate.

[0004] A tobacco leaf intelligent packaging machine and quality tracing method with publication number CN108272123B includes a rack, a movable pressure head on the rack, a camera on the pressure head or the rack on one side of the pressure head for shooting tobacco leaves, during loading, the camera is started to record the video during the loading of tobacco leaves; the video is compressed and stored in a database, and an identification code is established, the identification code is bound with the video and the data information of the tobacco leaves; after packaging is completed, the identification code is outputted and associated with the packaged tobacco leaves, a remote terminal retrieves and obtains related video data through the identification code, so as to facilitate quality tracing in subsequent processes.

[0005] The existing packaging process has many procedures, and the position of the carton at each station is mainly determined by the limit stopper. There is no targeted research on the state of the carton. During the packaging process, the normal progress of the packaging process is often affected, and even rework or shutdown is caused due to inaccurate positioning of the carton, incorrect angle of the cover plate and lack of filler information. In addition, during the conveying of the finished sheet from the conveying belt to the packaging machine, uneven material falling will cause high and low packages after packaging, resulting in uneven density, caking and caking problems, which affect the quality of the packaged products. SUMMARY

[0006] Therefore, the present application provides a sheet material packaging process carton state detection method and system, which solves the problems of inaccurate positioning of the carton, incorrect cover plate angle and lack of filling information in the sheet material packaging process, avoids uneven material falling, prevents uneven density, caking and caking of the sheet material, and improves the quality of the sheet material packaging finished product.

[0007] In one aspect, the present application provides a sheet material packaging process carton state detection method, comprising the following steps:

[0008] S1, collecting carton images and weight information of different stations, and obtaining robot position information;

[0009] S2, labeling the collected carton images to obtain a carton image dataset;

[0010] S3, constructing a carton region semantic segmentation model, inputting the carton image dataset into the carton region semantic segmentation model for model training and optimization, and obtaining the best carton region semantic segmentation model;

[0011] S4, setting the standard feature information and weight range threshold of the carton image corresponding to each station, identifying the carton image collected by each station using the best carton region semantic segmentation model, extracting the feature information of the carton image, and judging whether the extracted feature information and weight information of the carton image are consistent with the standard feature information and weight range threshold of the corresponding station carton image. If not, feedback to the robot for adjustment.

[0012] On the basis of the above technical scheme, preferably, the step S1 collects the carton image and weight feature information of different stations, and obtains the robot position information, wherein the sheet material packaging includes four stations, the No. 1 station is the state of the carton after the inner box bottom is sealed, the No. 2 station is the state of the carton after the outer box is sleeved on the inner box, the No. 3 station is the state of the carton after the sheet material is filled, and the No. 4 station is the state of the carton after the outer box top is sealed. Industrial cameras and weight sensors are arranged on each station, the carton overhead image of each station is collected by the industrial camera, and the carton weight information of each station is collected by the weight sensor.

[0013] On the basis of the above technical scheme, preferably, the step S1 further includes preprocessing the collected carton image, and converting the carton image coordinate axis to the robot coordinate axis with the robot position as the origin, wherein the preprocessing includes cropping the ROI region, the ROI region contains the carton region, the ROI region is determined according to the edge of the carton, and the size of the ROI region is greater than the edge of the carton by 50-100 pixels.

[0014] Based on the above technical scheme, preferably, the step S2 comprises manual labeling and automatic labeling, wherein the automatic labeling comprises the following sub-steps:

[0015] S21, performing channel extraction on the preprocessed carton image and converting it into a grayscale image;

[0016] S22, normalizing the pixel value of the grayscale image, counting the sum of the pixel values of the grayscale image of each channel, and determining the grayscale image of the channel with the maximum sum of pixel values;

[0017] S23, calculating the segmentation threshold value according to the total value of the sum of pixel values divided by the number of pixel points;

[0018] S24, segmenting the grayscale image of the channel according to the segmentation threshold value, extracting the pixel points greater than the segmentation threshold value, obtaining the carton region, filling all the pixel values of the carton region as 1, and obtaining the carton region semantic segmentation label;

[0019] S25, matching and associating the preprocessed carton image and the corresponding carton region semantic segmentation label, respectively filling them into the data set, and dividing the data set into a training set, a test set and a validation set.

[0020] Based on the above technical scheme, preferably, the step S3 comprises the following sub-steps:

[0021] S31, constructing a carton region semantic segmentation model based on DBS deep separable convolution, CBAM attention concentration mechanism and SegFormer network model;

[0022] S32, using a deep learning-based image generation network GAN to enhance the data set, and randomly inputting the training set data after enhancement processing into the carton region semantic segmentation model for training;

[0023] S33, adjusting the parameters of the carton region semantic segmentation model based on the LS-SOMA algorithm to obtain the best carton region semantic segmentation model.

[0024] Based on the above technical scheme, preferably, the network structure of the carton region semantic segmentation model in step S31 comprises an encoder and a decoder, wherein,

[0025] The encoder comprises four stacked Transformer Block modules, the convolution in each Transformer Block module adopts DBS deep separable convolution, the input carton image is subjected to feature sampling by each Transformer Block module, the sampling feature depths of different Transformer Block modules are different, and the sampling feature depth of the fourth Transformer Block module is the deepest;

[0026] The decoder comprises a CBAM attention mechanism module, a DBS deep separable convolution unit and a fusion unit; the carton image is output with feature maps of different sampling depths after being processed by the four Transformer Block modules of the encoder; the feature maps are subjected to feature enhancement by the CBAM attention mechanism, and are subjected to upsampling by the DBS module; the upsampled features and the features extracted by the Transformer Block modules are fused, and the fusion feature maps are obtained by upsampling and splicing fusion of the four Transformer Block module branches; after the fusion feature maps are subjected to convolution with a convolution kernel size of 1*1 twice to adjust the output channels thereof, the fusion feature maps are upsampled to the original image size, and finally a semantic segmentation image is output.

[0027] On the basis of the above technical scheme, preferably, the paper box region semantic segmentation model is parameterized based on the LS-SOMA algorithm in step S33 to obtain an optimal paper box region semantic segmentation model, and the method comprises the following sub-steps:

[0028] S331, initializing a population based on an LS-SOMA self-organizing migration algorithm to generate NP particles, and the particle coding mode is X i (r, h i1 , h i2 , h i3 , h i4 , ts i , bs i , lr i , mi i ), wherein i represents the i-th particle, r represents an image reduction factor, h i1 represents the number of neurons in the hidden layer of the first Transformer Block module, h i2 represents the number of neurons in the hidden layer of the second Transformer Block module, h i3 represents the number of neurons in the hidden layer of the third Transformer Block module, h i4 represents the number of neurons in the hidden layer of the fourth Transformer Block module, ts i represents a time step of model input, and bsi is a batch size, lr i is a initial learning rate size, mi i is a maximum number of iterations for training;

[0029] S332, a preset random disturbance coefficient prt is generated for each particle in the population to generate a corresponding disturbance vector PRTVector, and each particle in the population is decoded to configure each parameter value in the SegFormer network model;

[0030] S333, the test set data after enhancement processing is sequentially input into the carton region semantic segmentation model for training and verification, and each time the repetition number is an integer multiple of the verification threshold, the test result output by the model is compared with the image with the same label in the verification set data, the MIoU value and the MPA value are calculated, and the MIoU and MPA values are returned to the LS-SOMA self-organizing migration algorithm as the fitness function value of the LS-SOMA self-organizing migration algorithm;

[0031] S324, the fitness function value of the obtained population is sorted, the particle with the minimum fitness function value is selected as the leader, and the particles except the leader are migrated, and the migration expression is:

[0032] x i+1 =λ*x i *(1-x i )+(4-λ)*sin(π*x i ) / 4;

[0033] In the formula, x is a population particle solution, and λ is a chaotic coefficient, which is 0.5;

[0034] S325, when the maximum number of iterations of the LS-SOMA self-organizing migration algorithm is reached, the value of the last leader is the hyperparameter of the best semantic segmentation model, that is, the best carton region semantic segmentation model is obtained.

[0035] On the basis of the above technical scheme, preferably, in step S4, the paper box image standard feature information and the weight range threshold corresponding to each station are set, the best carton region semantic segmentation model is used to identify the paper box image collected by each station, the paper box image feature information is extracted, and it is judged whether the extracted paper box image feature information and weight information are consistent with the corresponding paper box image standard feature information and weight range threshold, if not, feedback is given to the robot for adjustment, including the following substeps:

[0036] S41, the standard state of the paper box corresponding to each station is imaged by artificial image collection, the standard feature information of the paper box image of each station is obtained by manual marking, and the paper box weight range threshold corresponding to each station is set;

[0037] S42, the collected carton images are sequentially input into a semantic segmentation model according to the stations, and semantic segmentation images corresponding to the stations are output;

[0038] S43, the obtained semantic segmentation images are segmented by using corresponding segmentation thresholds, and binary images are obtained;

[0039] S44, the binary image coordinate axes are converted into robot coordinate axes with the robot position as the origin;

[0040] S45, the inner carton body center and contour features, the outer carton cover plate angle and the carton closed contour features in the corresponding binary images are extracted according to different stations, the weights of the cartons in different stations are collected by the weight sensor, and whether the collected weight information is consistent with the standard feature information of the carton images in the corresponding station and the set carton weight range threshold is judged, if not, it is determined as an abnormal state, and feedback is given to the robot for adjustment.

[0041] On the basis of the above technical scheme, preferably, the method further comprises sequentially storing the carton image data of each station, generating two-dimensional code information, associating the two-dimensional code with all carton image data in the corresponding carton packing, and storing the two-dimensional code into the database, pasting the two-dimensional code on the inner and outer surfaces of the corresponding carton, and tracing the carton packing data according to the two-dimensional code.

[0042] On the other hand, the present application also provides a sheet material packing process carton state detection system, which comprises:

[0043] The acquisition module is used for acquiring carton images and weight information of different stations, and obtaining robot position information;

[0044] The labeling module is used for labeling the collected carton images to obtain a carton image dataset;

[0045] The model training module is used for constructing a carton region semantic segmentation model, inputting the carton image dataset into the carton region semantic segmentation model for model training and optimization, and obtaining an optimal carton region semantic segmentation model;

[0046] The judgment module is used for setting the standard feature information of the carton images corresponding to each station and the weight range threshold, identifying the collected carton images of each station by using the optimal carton region semantic segmentation model, extracting the feature information of the carton images, and judging whether the extracted feature information and weight information of the carton images are consistent with the standard feature information of the carton images corresponding to the station and the weight range threshold, if not, feedback is given to the robot for adjustment.

[0047] The data transmission method and control system of the embedded remote experiment platform provided by the present application have the following beneficial effects compared with the prior art:

[0048] 1) By combining the best carton region semantic segmentation model and the standard feature information set by artificial and the weight range threshold, the automatic, high-precision identification and abnormal detection of the carton packaging on the production line are realized; not only the efficiency and accuracy of the production line are improved, but also the quality consistency of the carton is ensured, and the quality of the sheet material packaging product is improved;

[0049] 2) The ROI region setting can reduce the calculation amount, improve the processing speed, and reduce the influence of background noise on the recognition result, and the image coordinate axis is converted into the robot coordinate axis, which can improve the accuracy and efficiency of subsequent carton recognition and state judgment;

[0050] 3) By combining DBS deep separable convolution, CBAM attention concentration mechanism and SegFormer network model, and using GAN for data enhancement and LS-SOMA algorithm for model parameter adjustment, a high-efficiency and accurate carton region semantic segmentation model can be constructed, which can accurately segment the carton region in the given carton image, and improve the recognition accuracy;

[0051] 4) Through the LS-SOMA self-organizing migration algorithm, the model can quickly find the best super parameter combination in the search space, thereby improving the efficiency of model training. BRIEF DESCRIPTION OF DRAWINGS

[0052] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0053] Figure 1 The flowchart of the sheet material packaging process carton state detection method of the present application;

[0054] Figure 2 The structural block diagram of the sheet material packaging process carton state detection system of the present application;

[0055] Figure 3 The schematic diagram of the sheet material packaging process carton state detection system of the present application;

[0056] Figure 4 The overall block diagram of the carton region semantic segmentation model of the sheet material packaging process carton state detection method of the present application;

[0057] Figure 5 The schematic diagram of the robot and the ROI region of the sheet material packaging process carton state detection method of the present application. DETAILED DESCRIPTION

[0058] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present application.

[0059] As Figures 1-5 The present application provides a method for detecting the state of a carton in a sheet material packaging process, including the following steps:

[0060] S1, collecting the image and weight information of the carton at different stations, and obtaining the position information of the robot;

[0061] Among them, the sheet material packaging includes 4 stations, the No. 1 station is the state of the carton after the inner box bottom is sealed, the No. 2 station is the state of the carton after the outer box is sleeved on the inner box, the No. 3 station is the state of the carton after the sheet material is filled, and the No. 4 station is the state of the carton after the outer box top is sealed. Industrial cameras and weight sensors are arranged on each station, the overhead images of the cartons at each station are collected by the industrial cameras, and the weight information of the cartons at each station is collected by the weight sensors.

[0062] Image acquisition: industrial cameras are arranged on each station, and lenses with appropriate focal lengths are equipped to ensure that high-resolution overhead images are obtained from the top of the carton in situ; these images can clearly show the outline, shape, position of the carton, and possible defects or abnormal conditions.

[0063] Weight acquisition: weight sensors are installed below each station to measure the weight of the carton and its internal tobacco sheet in real time; these weight information is key information for determining whether the carton is evenly filled, whether there is too much or too little tobacco sheet.

[0064] In step S1 of the present embodiment, the collected carton images are also preprocessed, and the carton image coordinate axis is converted to the robot coordinate axis with the robot position as the origin. The preprocessing includes cropping the ROI region, and the ROI region contains the carton region. The ROI region of the image is determined according to the carton edge, and the size of the ROI region is greater than 50-100 pixels of the edge of the carton.

[0065] It should be noted that the approximate position of the carton in the image is first determined, and according to the edge of the carton, an ROI region containing the carton is determined and cut out, and the size of the ROI region is greater than the edge of the carton by 50-100 pixels, so as to ensure that the carton can be completely contained and there is a certain buffer zone. In the embodiment, the size of the ROI region is preferably greater than the edge of the carton by 70 pixels. The setting of the ROI region can reduce the amount of calculation, improve the processing speed, and reduce the influence of background noise on the recognition result.

[0066] In the embodiment, the robot is used for carrying, positioning and adjusting the carton in the packaging process, so the position information of the robot needs to be obtained, and the image coordinate axis is converted into the robot coordinate axis with the position of the robot as the origin, which can improve the accuracy and efficiency of subsequent carton recognition and state judgment.

[0067] S2, labeling the collected carton image to obtain a carton image dataset;

[0068] The labeling of the collected carton image in step S2 includes manual labeling and automatic labeling, wherein the preprocessed image is automatically labeled by an image processing algorithm.

[0069] It should be noted that the labeling of the collected carton image includes manual labeling and automatic labeling, wherein the preprocessed image is manually labeled by using Lableme software, the carton region is labeled by using a polygon frame, and label information is obtained. The automatic labeling is automatically regionally labeled by an image processing algorithm, wherein the collected carton image at each station in the packaging process is labeled automatically, and the standard feature information of the carton image at each station is labeled by manual labeling.

[0070] It can be understood that manual labeling has higher accuracy but lower labeling efficiency, and automatic labeling can quickly generate a large number of labels but has lower accuracy than manual labeling. The combination of the two can balance the labeling speed and accuracy, and ensure the model training effect and the accuracy of subsequent judgment.

[0071] The automatic labeling includes the following sub-steps:

[0072] S21, performing channel extraction on the preprocessed carton image and converting it into a grayscale image;

[0073] S22, normalizing the pixel value of the grayscale image, counting the sum of the pixel values of the grayscale image of each channel, and determining the grayscale image of the channel with the maximum sum of pixel values;

[0074] S23, calculating the segmentation threshold value according to the total value of the sum of pixel values divided by the number of pixel points;

[0075] S24, according to the segmentation threshold, the gray image of the channel is segmented, the pixel points greater than the segmentation threshold are extracted, the paper box region is obtained, the pixel values of the paper box region are filled as 1, and the paper box region semantic segmentation label is obtained;

[0076] S25, the preprocessed paper box image and the corresponding paper box region semantic segmentation label are matched and associated, respectively filled into the data set, and the data set is divided into a training set, a test set and a validation set.

[0077] It should be noted that the preprocessed paper box image is channel extracted, the paper box image collected in the embodiment is an RGB image, and the red, green and blue three channels are extracted, the image of each channel is converted into a gray image, the pixel value of the gray image is normalized, the pixel value is scaled to 0-1, the sum of the pixel values of each channel of the gray image is counted, and the channel with the maximum sum of pixel values, i.e. the channel with the highest brightness, is determined. According to the total value of the sum of pixel values divided by the number of pixel points, the segmentation threshold is calculated. The segmentation threshold is used for subsequent image segmentation to distinguish the paper box region and the background region.

[0078] S3, constructing a paper box region semantic segmentation model, inputting the paper box image data set into the paper box region semantic segmentation model for model training and optimization, and obtaining the best paper box region semantic segmentation model;

[0079] Among them, step S3 includes the following sub-steps:

[0080] S31, based on DBS deep separable convolution, CBAM attention concentration mechanism and SegFormer network model, a paper box region semantic segmentation model is constructed;

[0081] S32, using the deep learning based image generation network GAN to enhance the data set, and inputting the training set data after enhancement processing into the paper box region semantic segmentation model for training;

[0082] S33, based on LS-SOMA algorithm, the paper box region semantic segmentation model is adjusted, and the best paper box region semantic segmentation model is obtained.

[0083] It should be noted that the DBS deep separable convolution is used to improve the calculation efficiency and reduce the parameter amount of the model, the CBAM attention concentration mechanism is introduced to enhance the attention of important areas in the image, improve the segmentation accuracy, and based on the SegFormer network model, the advantages of Transformer and convolution are combined to provide efficient and accurate semantic segmentation results; the image generation network GAN based on deep learning is used to enhance the carton image dataset; GAN can generate images similar to the original data, thereby increasing the diversity and quantity of the dataset, which helps to improve the generalization ability of the model, LS-SOMA is used to dynamically adjust the parameter settings until the best carton region semantic segmentation model is found, by combining DBS deep separable convolution, CBAM attention concentration mechanism and SegFormer network model, and using GAN for data enhancement and LS-SOMA algorithm for model parameter adjustment, an efficient and accurate carton region semantic segmentation model can be constructed, which can accurately segment the carton region in the given carton image.

[0084] The network structure of the carton region semantic segmentation model in step S31 includes an encoder and a decoder, wherein the encoder includes 4 stacked TransformerBlock modules, and the convolution in each Transformer Block module adopts DBS deep separable convolution; the input carton image is subjected to feature sampling by each TransformerBlock module, and the sampling feature depths of different Transformer Block modules are different, and the sampling feature depth of the 4th TransformerBlock module is the deepest; the decoder includes a CBAM attention mechanism module, a DBS deep separable convolution unit and a fusion unit; the carton image is processed by the 4 Transformer Block modules of the encoder, and feature maps with different sampling depths are output respectively; the feature maps are subjected to feature enhancement by the CBAM attention mechanism, and are subjected to upsampling by the DBS module; the upsampled features are fused with the features extracted by the Transformer Block modules, and the fusion feature maps are obtained by upsampling and splicing fusion of the 4 Transformer Block module branches; after the fusion feature maps are subjected to convolution with a convolution kernel size of 1x1 twice to adjust the output channel thereof, the fusion feature maps are upsampled to the original image size, and finally a semantic segmentation image is output.

[0085] It should be noted that the carton region semantic segmentation model in this embodiment adopts a network structure including an encoder and a decoder. The encoder part includes 4 stacked Transformer Block modules. Different Transformer Block modules sample features of different depths for the input image. With the increase of network depth, the resolution of the feature map gradually decreases, but the degree of abstraction of the features gradually increases. In particular, the fourth Transformer Block module samples the deepest features and contains the richest semantic information. The convolution in each Transformer Block module adopts DBS depth separable convolution. This convolution method can significantly reduce the computational load and parameter amount of the model while maintaining performance. The decoder first enhances the feature map output by the encoder through a CBAM attention mechanism module. The CBAM module can recalibrate the features in the spatial and channel dimensions, so that the model pays more attention to important regions in the image and enhances the features. The enhanced feature map is upsampled by a DBS depth separable convolution unit to gradually restore the resolution of the feature map. The upsampled features are fused with the features extracted by the corresponding Transformer Block module. Then, the final fused feature map is obtained by upsampling and splicing the features of the four Transformer Block module branches. The fused feature map is subjected to two convolution operations with a convolution kernel size of 1x1 to adjust the output channel number thereof. Finally, the feature map is upsampled to the original image size to obtain the final semantic segmentation image.

[0086] According to the carton region semantic segmentation model in this embodiment, different depth features are extracted by the encoder, the CBAM attention mechanism and DBS depth separable convolution are used by the decoder to enhance and upsample the features, and the features at different levels are fused through a fusion unit to finally output an accurate semantic segmentation image. This structure combines the advantages of Transformer and depth separable convolution, and simultaneously uses the CBAM attention mechanism to improve the performance of the model, so that the model can accurately segment the carton region in the carton image.

[0087] The image generation network GAN in step S32 includes two models, a generation model G module and a discrimination model D module; the task of the generation model is to generate natural and real instances similar to the original data; the task of the discrimination model is to judge whether a given instance looks natural and real or artificially fake, wherein the real instance is derived from the data set, and the fake instance is derived from the generation model; the G module and the D module; the G module is a network for generating pictures, which receives a random noise z and generates a picture through the noise, denoted as G(z). The D module is a discrimination network for discriminating whether a picture is real or not, and its input is n, which represents a picture, and the output D(n) represents the probability that n is a real picture. If it is 1, it means that 100% is a real picture, and if it is 0, it means that it is not a real picture.

[0088] Step S33 includes the following sub-steps:

[0089] S331, initialize the population based on the LS-SOMA self-organizing migration algorithm, generate NP particles, and the particle coding mode is X i (r, h i1 , h i2 , h i3 , h i4 , ts i , bs i , lr i , mi i ), wherein i represents the ith particle, r represents the image reduction factor, h i1 represents the number of neurons in the first TransformerBlock module hidden layer, h i2 represents the number of neurons in the second TransformerBlock module hidden layer, h i3 represents the number of neurons in the third Transformer Block module hidden layer, h i4 represents the number of neurons in the fourth Transformer Block module hidden layer, ts i represents the time step of the model input, bs i is the batch size, lr i is the initial learning rate, and mi i is the maximum number of iterations for training.

[0090] S332, preset a random disturbance coefficient prt, generate a corresponding disturbance vector PRTVector for each particle in the population, and decode each particle in the population to configure its parameter values in the SegFormer network model;

[0091] S333, input the test set data after enhancement processing into the carton region semantic segmentation model in turn, and perform training and verification, and whenever the repetition number is an integer multiple of the verification threshold, compare the test results output by the model with the images in the verification set data with the same label, calculate the MIoU value and the MPA value, and return the MIoU and MPA values as the fitness function value of the LS-SOMA algorithm to the LS-SOMA self-organizing migration algorithm;

[0092] S324, sort the fitness function values of the obtained population, select the particle with the minimum fitness function value as the leader, and migrate the particles except the leader, and the migration expression is:

[0093] x i+1 =λ*x i *(1-x i )+(4-λ)*sin(π*x i ) / 4;

[0094] In the formula, x is the population particle solution, and λ is a chaotic coefficient, and the value is 0.5;

[0095] S325, when the maximum iteration number of the LS-SOMA self-organizing migration algorithm is reached, the value of the last leader is the hyperparameter of the best semantic segmentation model, that is, the best carton region semantic segmentation model is obtained.

[0096] It should be noted that through the LS-SOMA self-organizing migration algorithm, the model can quickly find the best hyperparameter combination in the search space, thereby improving the efficiency of model training. This automatic hyperparameter adjustment process is more efficient and accurate than the traditional manual adjustment method. By using the LS-SOMA algorithm, the carton region semantic segmentation model can obtain higher average intersection over union MIoU and average pixel accuracy MPA on the test set, and the model can more accurately and finely capture boundaries and details when segmenting the carton region.

[0097] S4, set the standard feature information and weight range threshold of the carton image corresponding to each station, use the best carton region semantic segmentation model to identify the carton image collected by each station, extract the feature information of the carton image, and judge whether the extracted feature information and weight information of the carton image are consistent with the standard feature information and weight range threshold of the corresponding station, and if not, feedback to the robot for adjustment.

[0098] The step S4 includes the following sub-steps:

[0099] S41, manually collect the standard state of the carton corresponding to each station, obtain the standard feature information of the carton image of each station by manual marking, and set the weight range threshold of the carton corresponding to each station;

[0100] S42, sequentially input the collected carton images according to the stations into a semantic segmentation model, and output semantic segmentation images corresponding to the stations;

[0101] S43, segment the obtained semantic segmentation images using corresponding segmentation thresholds to obtain binary images;

[0102] S44, convert the coordinate axes of the binary images into robot coordinate axes with the robot position as the origin;

[0103] S45, extract the inner carton body center and contour features, outer carton cover plate angle, and carton closed contour features in the corresponding binary images according to different stations, collect the weights of the cartons in different stations using a weight sensor, and judge whether the standard feature information of the corresponding station and the set weight range threshold of the carton image are consistent, if not, it is determined as an abnormal state, and feedback to the robot for adjustment.

[0104] It should be noted that the best carton region semantic segmentation model is used to identify the carton images collected in each station, and the extracted carton image feature information and weight information are used to judge whether the carton meets the standard state, by setting the standard feature information and weight range threshold of each station, the quality consistency of the carton is ensured, the abnormal state is identified and fed back in time, so that the robot can quickly adjust, reduce production loss, and improve the quality of the sheet material packaging product.

[0105] In this embodiment, the carton image data of each station is sequentially stored, and a two-dimensional code information is generated, the two-dimensional code is associated with all carton image data in the corresponding carton packaging and stored in the database, the two-dimensional code is pasted on the inner and outer surfaces of the corresponding carton, and the carton packaging data is traced according to the two-dimensional code.

[0106] It should be noted that the image features and weight features of the carton state information after packaging are recorded and traced through the two-dimensional code, and a detailed detection report is output, which provides a reliable basis for intelligent evaluation and tracing of packaging quality.

[0107] The present application also provides a carton state detection system for sheet material packaging process, the system comprises:

[0108] The acquisition module is used for acquiring carton images and weight information of different stations, and obtaining robot position information;

[0109] The labeling module is used for labeling the collected carton images to obtain a carton image dataset;

[0110] The model training module is configured to construct a carton region semantic segmentation model, input a carton image dataset into the carton region semantic segmentation model, and perform model training and optimization to obtain an optimal carton region semantic segmentation model.

[0111] The judgment module is configured to set carton image standard feature information and a weight range threshold corresponding to each station, identify carton images collected by each station by using the optimal carton region semantic segmentation model, extract carton image feature information, and judge whether the extracted carton image feature information and weight information are consistent with the corresponding station carton image standard feature information and weight range threshold. If not, the robot is fed back for adjustment.

[0112] The above merely describes preferred embodiments of the present application and is not intended to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method of detecting the state of a carton in a process of packing a sheet material, characterized by, Comprise the following steps: S1, collect the paper box image and weight information of different stations, and obtain the robot position information; S2, labeling the collected paper box image, obtaining the paper box image dataset; S3, constructing a paper box region semantic segmentation model, inputting the paper box image dataset into the paper box region semantic segmentation model for model training and optimization, and obtaining the best paper box region semantic segmentation model; Step S3 comprises: S331, initializing the population based on the LS-SOMA self-organizing migration algorithm, generating NP particles, the particle coding mode is , wherein i represents the i-th particle, r represents an image reduction factor, represents the number of neurons of the first Transformer Block module hidden layer, represents the number of neurons of the second Transformer Block module hidden layer, represents the number of neurons of the third Transformer Block module hidden layer, represents the number of neurons of the fourth Transformer Block module hidden layer, represents the time step of the model input, is a batch size, is a primary learning rate size, is a maximum number of iterations for training; S332, presetting a random disturbance coefficient prt, generating a corresponding disturbance vector PRTVector for each particle in the population, and decoding each particle in the population, configuring its parameter values in the SegFormer network model; S333, input the enhanced test set data into the paper box region semantic segmentation model in turn, and perform training and verification, compare the test results output by the model with the images with the same label in the validation set data, calculate the MIoU value and the MPA value, and return the MIoU and MPA values to the LS-SOMA self-organizing migration algorithm as the fitness function value of the LS-SOMA algorithm; S334, sort the fitness function values of the obtained population, select the particle with the minimum fitness function value as the leader, and migrate the particles except the leader, the migration expression is: ； In the formula, x is the population particle solution, λ is the chaos coefficient, and the value is 0.5; S335, when the maximum iteration number of the LS-SOMA self-organizing migration algorithm is reached, the value of the last leader is the hyperparameter of the best semantic segmentation model, that is, the best paper box region semantic segmentation model is obtained; S4, set the paper box image standard feature information and weight range threshold corresponding to each station, use the best paper box region semantic segmentation model to identify the paper box image collected by each station, extract the paper box image feature information, and judge whether the extracted paper box image feature information and weight information are consistent with the corresponding station paper box image standard feature information and weight range threshold, if not, feedback to the robot for adjustment.

2. The method of sheet material packing process carton condition detection according to claim 1, characterized in that, The paper box image and weight feature information of different stations are collected, and the robot position information is obtained, wherein the sheet material packaging comprises four stations, the No. 1 station is the state of the paper box after the inner box bottom is sealed, the No. 2 station is the state of the paper box after the outer box is sleeved on the inner box, the No. 3 station is the state of the paper box after the sheet material is filled, and the No. 4 station is the state of the paper box after the outer box top is sealed. Industrial cameras and weight sensors are arranged on each station, the industrial cameras are used to collect the overhead images of the paper boxes at each station, and the weight sensors are used to collect the weight information of the paper boxes at each station.

3. The method of sheet material packing process carton condition detection according to claim 2, characterized in that, The step S1 further comprises preprocessing the collected paper box image, and converting the paper box image coordinate axis to the robot coordinate axis with the robot position as the origin, wherein the preprocessing comprises cropping the ROI region, the ROI region contains the paper box region, the ROI region is determined according to the paper box edge, and the size of the ROI region is greater than the edge of the paper box by 50-100 pixels.

4. The method of claim 3, wherein, The paper box image data set is obtained by labeling the collected paper box image, and the labeling of the collected paper box image comprises manual labeling and automatic labeling, wherein the automatic labeling comprises the following sub-steps: S21, performing channel extraction on the preprocessed paper box image and converting it into a gray scale image; S22, normalizing the pixel value of the gray scale image, counting the sum of the pixel values of the gray scale image of each channel, and determining the gray scale image of the channel with the maximum sum of pixel values; S23, calculating the segmentation threshold value according to the total value of the pixel value divided by the number of pixel points; S24, segmenting the gray scale image of the channel according to the segmentation threshold value, extracting the pixel points greater than the segmentation threshold value, obtaining the paper box region, filling all the pixel values of the paper box region as 1, and obtaining the paper box region semantic segmentation label; S25, matching and associating the preprocessed paper box image and the corresponding paper box region semantic segmentation label, respectively filling them into the data set, and dividing the data set into a training set, a test set and a validation set.

5. The method of sheet material packing process carton condition detection according to claim 4, characterized in that, The paper box region semantic segmentation model is constructed in step S3, and the paper box image data set is input into the paper box region semantic segmentation model for model training and optimization to obtain the best paper box region semantic segmentation model, wherein the following sub-steps are included: S31, constructing a paper box region semantic segmentation model based on DBS deep separable convolution, CBAM attention concentration mechanism and SegFormer network model; S32, using a deep learning-based image generation network GAN to enhance the data set, and randomly inputting the training set data after enhancement processing into the paper box region semantic segmentation model for training; S33, adjusting the parameters of the paper box region semantic segmentation model based on the LS-SOMA algorithm to obtain the best paper box region semantic segmentation model.

6. The method of sheet material packing process carton condition detection according to claim 5, wherein, The network structure of the paper box region semantic segmentation model in step S31 comprises an encoder and a decoder, wherein The encoder comprises 4 stacked Transformer Block modules, and the convolution in each Transformer Block module adopts DBS deep separable convolution; the input paper box image is subjected to feature sampling by each Transformer Block module, the sampling feature depths of different Transformer Block modules are different, and the sampling feature depth of the 4th Transformer Block module is the deepest; The decoder comprises a CBAM attention mechanism module, a DBS deep separable convolution unit and a fusion unit; the carton image is processed by the four Transformer Block modules of the encoder, and feature maps of different sampling depths are output respectively; the feature maps are subjected to feature enhancement by the CBAM attention mechanism, and are up-sampled by the DBS module; the up-sampled features are fused with the features extracted by the Transformer Block modules, and the up-sampling spliced fusion of the four branches of the Transformer Block modules is obtained to obtain a fusion feature map; the fusion feature map is subjected to convolution with a 1*1 convolution kernel twice to adjust the output channel, and is then up-sampled to the original image size, and finally a semantic segmentation image is output.

7. The method of sheet material packing process carton condition detection of claim 1, wherein, In step S4, the standard feature information of the carton image corresponding to each station and the weight range threshold are set, the carton image collected at each station is recognized by using the best carton region semantic segmentation model, the feature information of the carton image is extracted, and whether the extracted feature information and weight information of the carton image are consistent with the standard feature information and weight range threshold of the corresponding station is judged, if not, the robot is fed back for adjustment, including the following sub-steps: S41, the standard state of the carton corresponding to each station is collected by using manual image collection, the standard feature information of the carton image of each station is obtained by manual marking, and the weight range threshold of the carton corresponding to each station is set; S42, the collected carton images are sequentially input into the semantic segmentation model according to the stations, and the semantic segmentation images of the corresponding stations are output; S43, the obtained semantic segmentation images are segmented by using the corresponding segmentation threshold, and a binary image is obtained; S44, the binary image coordinate axis is converted into the robot coordinate axis with the robot position as the origin; S45, the center of the inner carton body and the contour feature, the outer carton cover plate angle and the carton closed contour feature in the corresponding binary image of each station are extracted respectively, the weight of the carton at each station is collected by the weight sensor, and whether the collected weight is consistent with the standard feature information of the carton image of the corresponding station and the set weight range threshold of the carton is judged, if not, it is determined as an abnormal state, and the robot is fed back for adjustment.

8. The method of sheet material packing process carton condition detection of claim 1, wherein, The method further comprises sequentially storing the carton image data of each station, generating two-dimensional code information, associating the two-dimensional code with all carton image data in the corresponding carton packaging, and storing the two-dimensional code into the database, pasting the two-dimensional code on the inner and outer surfaces of the corresponding carton, and tracing the carton packaging data according to the two-dimensional code.

9. A system for detecting the condition of cartons in a process for packing a sheet material, characterized by: The system comprises: The acquisition module is used for acquiring the carton images and weight information of different stations, and obtaining the robot position information; The labeling module is used for labeling the collected carton images to obtain a carton image dataset; The model training module is used for constructing a carton region semantic segmentation model, inputting the carton image dataset into the carton region semantic segmentation model for model training and optimization, and obtaining the best carton region semantic segmentation model; Step S3 comprises: The population is initialized based on the LS-SOMA self-organizing migration algorithm, NP particles are generated, and the particle coding mode is , wherein i represents the i th particle, r represents an image reduction factor, represents the number of neurons of the first Transformer Block module hidden layer, represents the number of neurons of the second Transformer Block module hidden layer, represents the number of neurons of the third Transformer Block module hidden layer, represents the number of neurons of the fourth Transformer Block module hidden layer, represents the time step of model input, is a batch size, is a primary learning rate size, is a maximum number of iterations for training; A preset random disturbance coefficient prt is used to generate a corresponding disturbance vector PRTVector for each particle in the population, and each particle in the population is decoded and its parameter values are configured in the SegFormer network model; The enhanced test set data is sequentially input into the carton region semantic segmentation model for training and verification. Whenever the number of repetitions is an integer multiple of the verification threshold, the test results output by the model are compared with the images in the validation set data that have the same labels, and the MIoU value and MPA value are calculated. The MIoU and MPA values are returned to the LS-SOMA self-organizing migration algorithm as the fitness function values of the LS-SOMA algorithm; The fitness function values of the obtained population are sorted, the particle with the smallest fitness function value is selected as the leader, and the particles other than the leader are migrated. The migration expression is: ； In the formula, x is the population particle solution, λ is the chaos coefficient, and the value is 0.5; When the maximum number of iterations of the LS-SOMA self-organizing migration algorithm is reached, the value of the last leader is the hyperparameter of the best semantic segmentation model, i.e., the best carton region semantic segmentation model is obtained; The judgment module is used to set the standard feature information and weight range threshold of the carton image corresponding to each station, recognize the carton image collected by each station using the best carton region semantic segmentation model, extract the feature information of the carton image, and judge whether the extracted feature information and weight information of the carton image are consistent with the standard feature information and weight range threshold of the corresponding station carton image. If not, feedback is given to the robot for adjustment.

Images