Article packaging defect detection method and system

Abstract

The invention relates to the field of article packaging defect detection, in particular to an article packaging defect detection method and system. An infrared image and an RGB image are shot to calculate a difference feature and a fusion feature, the difference feature is converted into an attention weight through an activation function, the attention weight is multiplied by the fusion feature and decoded to obtain a film feature, whether a film exists in each area on an article package or not is recognized, and then an intact image with the film on an article is recognized. The method comprises the following steps: respectively extracting an interference image on a thin film and a package image on an article package, finally repairing the package image based on the interference image to obtain a to-be-detected image of the surface of the article package without the thin film, and finally further realizing identification of the to-be-detected image based on a current defect detection algorithm. The problem that whether a film is damaged or not and whether patterns on an object package are abnormal or not are difficult to accurately recognize when packaging defects of the object package containing the film are recognized in an image recognition mode at present is solved.

Description

Technical Field

[0001] The present invention relates to the field of article packaging defect detection, and in particular to an article packaging defect detection method and system. Background Art

[0002] In terms of item packaging design, products are often packaged in packaging boxes, and then the packaging boxes are further sealed with transparent films such as plastic films to achieve the purpose of waterproofing and dustproofing. In the final stage of the actual production process, it is necessary to detect the integrity of the product. At this time, it is necessary to check whether the film is intact and whether the packaging has defects such as damage, deformation, and abnormal patterns.

[0003] At present, there are two main methods for inspecting packaging containing films: manual and image recognition. The manual method requires a lot of manpower costs. According to feedback from some cooperative factories, there are cases of recognition errors when recognizing packaging containing films through image recognition. After retrieving the packaging images used for recognition in the image recognition process, it was found that, on the one hand, because the film is very thin and transparent, it is difficult to accurately identify when there are holes and other defects in the film. On the other hand, Figure 3 As shown, the transparent film surface reflects images of other items in the factory, such as Figure 3 The reflected image shown in the white box in the middle causes the image recognition system to misjudge the pattern on the item packaging as abnormal, resulting in a false alarm. Summary of the invention

[0004] In view of the deficiencies in the prior art, the present invention provides a method and system for detecting article packaging defects, which solves the problem that it is difficult to accurately identify whether the film is damaged and whether there is any abnormality in the pattern on the article packaging when identifying packaging defects through image recognition.

[0005] To achieve the above object, the present invention provides the following technical solutions: A method for detecting article packaging defects, the method specifically comprising the following steps: S1, obtaining several groups of infrared images and RGB images containing object packaging as input images; S2, obtaining film features based on the difference features and fusion features of each area in the corresponding position of the surface of the packaging article in the input image; S3, judging whether each area on the input image belongs to a complete image according to the film characteristics; If yes, mark the area as a good image; If not, the area is marked as a defective area and output; S4, extracting interference features and intermediate features from the intact image, and multiplying the intermediate features with the intermediate features after passing through the ECA channel to obtain the interference image; S5, extracting packaging features of the intact image, and converting the packaging features into two-dimensional features after passing through several layers of RSTB modules to obtain a packaging image; S6. Use the interference image to repair the packaging image to obtain the image to be detected, and identify defects in the image to be detected.

[0006] Preferably, in step S1, the following steps are specifically included: S11, capturing an RGB image containing the item packaging through an RGB camera; S12, arranging a plurality of infrared cameras, and measuring a line connecting a lens of the infrared camera and a surface of the packaging of the item and a maximum angle between the surface of the packaging of the item; S13, setting an angle threshold, and using each infrared camera to photograph each area on the item packaging where the maximum angle is less than the angle threshold to obtain a plurality of groups of infrared images; S14. Mark each infrared image and the RGB image containing the corresponding position as a group of input images to obtain several groups of input images.

[0007] Preferably, in step S13, the angle threshold is 70 degrees.

[0008] Preferably, in step S2, the following steps are specifically included: S21. The encoder based on the neural network model extracts the one-dimensional features of the corresponding positions on the surface of the item packaging in each group of infrared images and RGB images at different scales respectively; the calculation formula for extracting the one-dimensional features of the corresponding positions on different scales of the infrared image and the RGB image is: ; In the above formula, represents the one-dimensional features of the infrared image or RGB image extracted at the i-th scale, The range is , represents the pre-trained backbone network, Represents the input infrared image or RGB image; S22, converting each group of one-dimensional features into two-dimensional features, and then connecting the converted two-dimensional features of the infrared image and the RGB image through channels to obtain fusion features; the calculation formula of the fusion features is as follows: ; In the above formula, represents the one-dimensional feature of the extracted RGB image at the i-th scale, represents the one-dimensional feature of the extracted infrared image at the i-th scale, Indicates that the kernel is The convolutional layer of the convolutional neural network, represents the channel attention module, Indicates that the elements in the brackets are connected by channels. express function, and They represent the one-dimensional features of the RGB image and infrared image processed by the SE channel at the i-th scale, Indicates fusion features; S23, calculating the difference between the one-dimensional features of the RGB image and the infrared image after passing through the SE channel at the i-th scale to obtain the difference feature; the calculation formula of the difference feature is: ; In the above formula, Indicates differential characteristics; S24. The difference feature is converted into attention weight through activation function and then multiplied with the fusion feature to obtain film features at different scales. The calculation formula of film features at different scales is: ; In the above formula, represents the film characteristics at the i-th scale, represents the activation function; S25, connect the film features at different scales according to the channels and send them to the convolution layer to obtain the film features; the calculation formula of the film features is as follows: ; In the above formula, represents the film detection result at the i-th scale, is the upsampling operation of the convolutional neural network, represents the final obtained film characteristics, The kernel of the convolutional neural network is represented by The convolutional layer.

[0009] Preferably, in step S4, the following steps are specifically included: S41. Extract interference features from intact images based on a convolutional neural network model; the expression of interference features is: ; In the above formula, The kernel of the convolutional neural network is represented by The convolutional layer, I represents the input intact image, represents the ReLU activation function, represents the batch normalization function, and Represent the interference features extracted at the i-th and i-1-th scales respectively; S42, the interference features at each scale are passed through several convolutional layers and activation functions to obtain intermediate features; the calculation formula of the intermediate features is: ; In the above formula, represents the intermediate features at the i-th scale, The range is , represents the ReLU activation function, The kernel of the convolutional neural network is represented by The convolutional layer; Indicated in scale Interference features extracted from S43, multiplying the intermediate feature by the intermediate feature after passing through the ECA channel and then adding the intermediate feature to the interference feature at the corresponding scale to obtain an intermediate interference image; the calculation formula of the intermediate interference image is as follows: ; In the above formula, represents the intermediate features at the i-th scale, represents the intermediate interference image at the i-th scale, represents an efficient channel attention layer for a convolutional neural network model, Represents the interference features extracted at the i-th scale; S44, setting the number of extractions, and counting whether the current number of times of step S42 is greater than the number of extractions; If yes, the intermediate interference image is marked as an interference image and the process proceeds to step S5; If not, the intermediate interference image is marked as an interference feature, and then the process returns to step S42.

[0010] Preferably, in step S5, the following steps are specifically included: S51. Extract the packaging features of the intact image through the VGG-19 convolutional neural network model; the expression of the packaging features is: ; In the above formula, , , and They represent the packaging features of intact images at the 1st, 2nd, 3rd, and 4th scales extracted by the VGG-19 convolutional neural network model. Indicates that a complete image is extracted through the VGG-19 convolutional neural network model Packaging characteristics; S52, passing the packaging features through several layers of RSTB to obtain one-dimensional packaging features, and converting them into two-dimensional packaging features to obtain a packaging image; the calculation formula of the two-dimensional packaging features is as follows: ; In the above formula, Indicated in scale The two-dimensional packaging characteristics on The range is , Indicates that the intact image is extracted by the VGG-19 convolutional neural network model at scale The packaging features on Indicates that the packaging features are passed through the A layer RSTB, Indicates that the elements in brackets are converted into two-dimensional packaging features through the Patch Unembed function.

[0011] Preferably, in step S6, the following steps are specifically included: S61, input the interference image into several dilation rate hole convolution layers, and connect the hole convolution layers of different scales and fuse them through the convolution layer to obtain the interference fusion image; the calculation formula of the interference fusion image is: ; In the above formula, represents the interference fusion image at the i-th scale, The range is , Indicates channel connection, represents the dilation rate of the dilated convolution, , and They are the values ​​of the three dilation rates of the hole convolution. represents the interference image at the i-th scale; S62, connecting the interference fusion image with the interference features at the corresponding scale and then upsampling to obtain an upsampled interference image; the calculation formula of the upsampled interference image is: ; In the above formula, represents the upsampled interference image at the i-1th scale, represents upsampling, Represents the interference feature at the i-th scale; S63, after passing the packaging image through the convolution layer, removing the interference fusion image at the corresponding scale, connecting it with the packaging feature at the i-th scale, and upsampling it to obtain an upsampled packaging image; the calculation formula of the upsampled packaging image is: ; In the above formula, represents the upsampled packed image at the i-1th scale, Indicated in scale The packaging image on represents the interference fusion image at the i-th scale, represents the packaging characteristics of the intact image at the i-th scale; represents the packaging characteristics of the intact image at the i-th scale; S64, acquiring an upsampled packed image and an upsampled interference image at scale 1 to obtain an image to be detected and a standard interference image; S65, setting a loss function for training a convolutional neural network model for acquiring an image to be detected; S66. Identify defects in the image to be detected through a defect detection algorithm and output the defects.

[0012] Preferably, in step S61, the , and They are 1, 2 and 5 respectively.

[0013] Preferably, in step S65, the loss function is expressed as: ; In the above formula, L represents the loss function of the convolutional neural network model training for acquiring the image to be detected, , , and They represent packaging loss, interference loss, packaging perception loss and image reconstruction loss respectively. , , and Respectively represent the first loss coefficient, the second loss coefficient, the third loss coefficient and the fourth loss coefficient, represents the image to be detected, This is an actual image of the product packaging without the film. represents the standard interference image on the film, represents the true image of the interference on the film, represents the input intact image, represents the reconstructed image obtained by adding the standard interference image and the image to be detected, represents a function for calculating the structural similarity between the image to be detected and the actual image of the product packaging without the film, is a hyperparameter that balances the mean squared error loss and the structural similarity loss, Represents the packaging features of images at several different scales extracted by the VGG-19 convolutional neural network model. The range is .

[0014] The technical solution also provides a system for implementing the method for detecting article packaging defects, the system comprising: a processor and a memory, the memory being used to store a computer program, and the computer program implementing the method for detecting article packaging defects when executed by the processor.

[0015] Compared with the prior art, the present invention provides a method and system for detecting article packaging defects, which has the following beneficial effects: 1. The present invention first shoots infrared images and RGB images, then calculates difference features and fusion features, and converts the difference features into attention weights through activation functions, and then multiplies and decodes the fusion features to obtain film features, so as to realize the recognition of whether there is a film in each area on the object packaging. Then, for the intact image with a film on the object, the interference image on the film and the packaging image on the object packaging are respectively extracted. Finally, the packaging image is finally repaired based on the interference image to obtain the image to be detected of the object packaging surface without a film. Finally, based on the current defect detection algorithm, the recognition of the image to be detected can be further realized.

[0016] 2. In order to reduce the influence of infrared light emission on the infrared image, the present invention sets an angle threshold to control the maximum angle between the line connecting the infrared camera lens and the surface of the article packaging and the surface of the article packaging to be less than or equal to the angle threshold, thereby ensuring the accuracy of subsequent identification of film features to the greatest extent.

[0017] 3. In order to more accurately compare the infrared image and the RGB image to realize the recognition of the thin film features, the present invention connects the two-dimensional features of the converted infrared image and the RGB image through channels through an activation function to obtain fusion features, thereby realizing the mining of deep features without losing shallow features. Subsequently, the difference features of the reflection difference between the infrared image and the RGB image are calculated to better help the subsequent positioning of the thin film features.

[0018] 4. Since interference images only exist in areas where there is film on the packaging of the item, the areas where there is film features are marked as intact images, which facilitates further feature extraction of interference features in the intact image and the packaging image to obtain more accurate images to be detected.

[0019] 5. The present invention constructs a new loss function by adding packaging perception loss and image reconstruction loss to improve the accuracy of detail features in the acquired image to be detected and standard interference image and the clarity of the image. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings: Figure 1 It is a flow chart of the method for detecting article packaging defects of the present invention; Figure 2 The relationship between the intensity of the infrared light reflected in the infrared image of the present invention and the shooting angle; Figure 3 Schematic diagram of the reflection image on the film of the present invention. DETAILED DESCRIPTION

[0021] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific implementation methods, so that the implementation process of how the present application uses technical means to solve technical problems and achieve technical effects can be fully understood and implemented accordingly.

[0022] Those skilled in the art can understand that all or part of the steps in the following embodiments can be completed by instructing the relevant hardware through a program, so the present application can be in the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application can be in the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0023] In order to solve the problem that it is difficult to accurately identify whether the film is damaged and whether the pattern on the article packaging is abnormal when identifying packaging defects through image recognition, the present invention provides a method for detecting article packaging defects, which first detects whether there is a film at various locations on the article packaging surface and the location where the film should be present to determine whether the film is defective, then detects the surface of the article packaging where the film is present and removes the images of other articles reflected on the film surface, thereby restoring the article packaging, and finally further detects the article packaging defects through the existing image recognition algorithm, such as Figure 1 As shown, the detection method specifically includes the following steps: S1. Acquire several groups of infrared images and RGB images containing article packaging as input images. The infrared images and RGB images can be directly acquired by shooting with an RGB camera and an infrared camera. In order to reduce the influence of infrared reflected light on the infrared images, thereby facilitating the subsequent identification of the location of the film on the article packaging, a method for acquiring infrared images is provided. In step S1, the method specifically includes the following steps: S11, capturing an RGB image containing the item packaging through an RGB camera; S12, arranging a plurality of infrared cameras, and measuring the maximum angle between the line between the infrared camera lens and the surface of the item packaging and the surface of the item packaging, that is, the maximum angle between the shooting direction of the infrared camera lens and the surface of the item packaging; S13, setting an angle threshold, and using each infrared camera to shoot each area on the item packaging where the maximum angle is less than the angle threshold to obtain a plurality of sets of infrared images. After shooting a large number of infrared images and comparing them, it is found that when the film on the surface of the item packaging and the camera shooting direction, that is, the direction of the camera lens, form a certain angle, there are very few or even no reflection features on the infrared image, such as Figure 2 As shown in the figure, I_r is the relationship between the intensity of the infrared light reflected in the infrared image and the angle between the film on the surface of the item packaging and the camera shooting direction, and I_t is the relationship between the intensity of the infrared light passing through the film and the angle between the film on the surface of the item packaging and the camera shooting direction. In order to minimize the impact of the reflection of infrared light on the infrared image, as shown in the figure, when the maximum angle is greater than 70 degrees, the impact of the reflected infrared light on the infrared image begins to increase rapidly, so the angle threshold is set to 70 degrees.

[0024] S14. Mark each infrared image and the RGB image containing the corresponding position as a group of input images to obtain several groups of input images.

[0025] S2. Based on the difference features and fusion features of each area in the corresponding position of the packaging article surface in the input image, the film features are obtained, and the features in the infrared image and the RGB image are extracted, so as to realize the recognition of the film features through comparison. The structure of the neural network model is further explained below: S21. The encoder based on the neural network model extracts the one-dimensional features of the corresponding positions on the surface of the item packaging in each group of infrared images and RGB images at different scales respectively; the calculation formula for extracting the one-dimensional features of the corresponding positions on different scales of the infrared image and the RGB image is: ; In the above formula, represents the one-dimensional features of the infrared image or RGB image extracted at the i-th scale, The range is , Represents a pre-trained backbone network. In order to improve the extraction of complex reflective images on packaging containing thin films, the Swin Transformer v2 backbone network, which is good at extracting global features, can be used here. The moving window attention mechanism of the model can be used to simultaneously capture the local and global features of the input image. Represents the input infrared image or RGB image; S22, converting each group of one-dimensional features into two-dimensional features. The conversion method may be a Reshape function, etc., and then connecting the converted two-dimensional features of the infrared image and the RGB image through channels to obtain fused features, so as to mine deep features without losing shallow features. The calculation formula of the fused features is as follows: ; In the above formula, represents the one-dimensional feature of the extracted RGB image at the i-th scale, represents the one-dimensional feature of the extracted infrared image at the i-th scale, Indicates that the kernel is The convolutional layer of the convolutional neural network, represents the channel attention module, Indicates that the elements in the brackets are connected by channels. express function, and They represent the one-dimensional features of the RGB image and infrared image processed by the SE channel at the i-th scale, The one-dimensional features and fusion features of the RGB image and infrared image output by the channel attention module can determine the weight relationship of different channels through the correlation between feature channels, so as to obtain more effective feature information. At the same time, in order to avoid the loss of some feature information, the feature and The model is aggregated by skipping layers, so that it can continuously acquire deep features without losing gradient features. S23, calculating the difference between the one-dimensional features of the RGB image and the infrared image after passing through the SE channel at the i-th scale to obtain the difference feature. Since there may be a reflected image on the film area, there may be different patterns on the film area in the infrared image and the RGB image, so there is a difference feature. Therefore, by calculating the difference feature, it can better help to locate whether there is a film feature in each area on the item packaging in the subsequent step. The calculation formula of the difference feature is: ; In the above formula, Indicates differential characteristics; S24. The difference feature is converted into attention weight through activation function and then multiplied with fusion feature to obtain film features at different scales. The difference feature is converted into attention weight. Therefore, the greater the reflection difference, the greater the weight of the film feature. Finally, it is multiplied with the fusion feature, so as to achieve the purpose of enhancing film features and suppressing non-film features, so as to accurately identify film features. The calculation formula of film features at different scales is: ; In the above formula, represents the film characteristics at the i-th scale, Represents the activation function, which can be a Simoid function; S25, connect the film features at different scales according to the channels and send them to the convolution layer to obtain the film features; the calculation formula of the film features is as follows: ; In the above formula, represents the film detection result at the i-th scale, is the upsampling operation of the convolutional neural network, represents the final obtained film characteristics, The kernel of the convolutional neural network is represented by The convolutional layer; S3, judging whether each area on the input image belongs to a complete image according to the film characteristics; If yes, mark the area as a good image; If not, the area is marked as a defective area and output; S4, extract the interference features and intermediate features in the intact image, and multiply the intermediate features with the intermediate features after passing through the ECA channel to obtain the interference image. The full name of ECA is Efficient Channel Attention, which is an attention mechanism for deep learning models. It aims to efficiently extract the dependencies between channels through local cross-channel interactions without dimensionality reduction. By observing the reflection image of the film on the intact image and the pattern on the item packaging, it is found that the reflection image, i.e., the interference feature, and the item packaging pattern are unrelated, so the textures of the two are independent of each other. Therefore, the two patterns are extracted respectively by the two extraction methods in step S4 and step S5 to prevent the two patterns from interfering with each other. In addition, due to the relatively small amount of information contained in the display of the production line beat and the interference features, an extraction method with relatively less computational effort is provided to extract the interference image on the basis of ensuring the extraction effect. The structure of the convolutional neural network model of this step is introduced in detail below: S41. Extract interference features from the intact image based on the convolutional neural network model. The interference features are reflection images. The expression of the interference features is: ; In the above formula, The kernel of the convolutional neural network is represented by The convolutional layer, I represents the input intact image, represents the ReLU activation function, represents the batch normalization function, and Represent the interference features extracted at the i-th and i-1-th scales respectively; S42, the interference features at each scale are passed through several convolutional layers and activation functions to obtain intermediate features; the calculation formula of the intermediate features is: ; In the above formula, represents the intermediate features at the i-th scale, The range is , represents the ReLU activation function, The kernel of the convolutional neural network is represented by The convolutional layer; Indicated in scale Interference features extracted from S43, multiplying the intermediate feature by the intermediate feature after passing through the ECA channel and then adding the intermediate feature to the interference feature at the corresponding scale to obtain an intermediate interference image; the calculation formula of the intermediate interference image is as follows: ; In the above formula, represents the intermediate features at the i-th scale, represents the intermediate interference image at the i-th scale, represents an efficient channel attention layer for a convolutional neural network model, Represents the interference features extracted at the i-th scale; S44, setting the number of extractions, and counting whether the current number of times of step S42 is greater than the number of extractions; If yes, the intermediate interference image is marked as an interference image and the process proceeds to step S5, so that a more refined interference image can be finally obtained by extracting the intermediate interference image multiple times in a cycle; If not, the intermediate interference image is marked as an interference feature, and then the process returns to step S42.

[0026] After the interference features are extracted, through steps S42 to S44, the advantages of the ECA channel cross-channel interaction strategy can be retained, and more detailed interference images for subsequent detection can be further extracted from the interference features.

[0027] S5. Extract the packaging features of the intact image, and convert the packaging features into two-dimensional features after passing through several layers of RSTB modules to obtain the packaging image. RSTB is called Residual Swin Transformer Block, which is a key component in the Swin Transformer architecture and is mainly used for image restoration tasks. RSTB combines the advantages of convolutional neural networks (CNN) and Transformer, captures global information through the self-attention mechanism, and uses the local attention mechanism to process large-size images. Due to the larger amount of information in the packaging image and the limitation of the production line beat, in order to more accurately extract the pattern information on the packaging surface of the item without film, so as to obtain a clearer and more complete packaging image, the convolutional neural network model structure of this step is further explained below: S51. Extract the packaging features of the intact image by selecting the VGG-19 convolutional neural network model with better extraction performance; the expression of the packaging features is: ; In the above formula, , , and They represent the packaging features of the intact image at the 1st, 2nd, 3rd and 4th scales extracted by the VGG-19 convolutional neural network model, that is, the packaging features output by the 4 different convolutional layers of the VGG-19 convolutional neural network model. Indicates that a complete image is extracted through the VGG-19 convolutional neural network model The packaging features of the convolutional layer are small, so it is difficult to extract the packaging features through the shallow convolutional layer to obtain the packaging image. Therefore, it is imported into several layers of RSTB modules to extract deep features. S52, passing the packaging features through several layers of RSTB to obtain one-dimensional packaging features, and converting them into two-dimensional packaging features to obtain a packaging image; the calculation formula of the two-dimensional packaging features is as follows: ; In the above formula, Indicated in scale The two-dimensional packaging characteristics on The range is , Indicates that the intact image is extracted by the VGG-19 convolutional neural network model at scale The packaging features on Indicates that the packaging features are passed through the A layer RSTB, Indicates that the elements in brackets are converted into two-dimensional packaging features through the Patch Unembed function.

[0028] S6. Using the interference image to repair the packaged image to obtain the image to be detected, and identifying defects in the image to be detected, and restoring the pattern details of the image to be detected by a method combining the reverse attention mechanism and the interference image, specifically comprising the following steps: S61, input the interference image into several dilation rate hole convolution layers, and connect the hole convolution layers of different scales and fuse them through the convolution layer to obtain the interference fusion image, so as to capture the context information of the multi-scale interference features. The calculation formula of the interference fusion image is: ; In the above formula, represents the interference fusion image at the i-th scale, The range is , Indicates channel connection, represents the dilation rate of the dilated convolution, , and They are the values ​​of the three dilation rates of the hole convolution. represents the interference image at the i-th scale. In order to optimally extract the interference fusion image, it is found through experiments that the maximum scale of the interference fusion image is 4. At this time, the resolution of the input interference image is 1 / 8. In order to better detect tiny features, the optimal range of the optional dilation rate of the void convolution is , it was found through experiments that , and The best combination of values ​​is 1, 2 and 5, which can best achieve the extraction of subtle interference features to obtain an interference fusion image.

[0029] S62, connecting the interference fusion image with the interference features at the corresponding scale and then upsampling to obtain an upsampled interference image; the calculation formula of the upsampled interference image is: ; In the above formula, represents the upsampled interference image at the i-1th scale, represents upsampling, Represents the interference feature at the i-th scale; S63, after passing the packaging image through the convolution layer, removing the interference fusion image at the corresponding scale, connecting it with the packaging feature at the i-th scale, and upsampling it to obtain an upsampled packaging image; the calculation formula of the upsampled packaging image is: ; In the above formula, represents the upsampled packed image at the i-1th scale, Indicated in scale The packaging image on represents the interference fusion image at the i-th scale, represents the packaging feature of the intact image at the i-th scale, by subtracting the interference fusion image from the packaging image. is the activation function, which can be the Sigmoid activation function, that is, , to avoid retaining features that interfere with the fused image in the packaged image, and then By applying it to the interference fusion image, the attention map of the packaged image can be obtained. Then, the packaged image without interference fusion image is multiplied by the attention map and up-sampled to obtain the image to be detected with complete features. S64, acquiring an upsampled packed image and an upsampled interference image at scale 1 to obtain an image to be detected and a standard interference image; S65, setting a loss function for training a convolutional neural network model for acquiring an image to be detected. Conventional loss functions generally include packaging loss and interference loss. At this time, the output image is relatively blurred. In order to improve the clarity of the acquired image to be detected and avoid the loss of details in the image to be detected and the standard interference image, packaging perception loss and image reconstruction loss are added to construct a new loss function. Therefore, the expression of the loss function is: ; In the above formula, L represents the loss function of the convolutional neural network model training for acquiring the image to be detected, , , and They represent packaging loss, interference loss, packaging perception loss and image reconstruction loss respectively. , , and Respectively represent the first loss coefficient, the second loss coefficient, the third loss coefficient and the fourth loss coefficient, represents the image to be detected, This is an actual image of the product packaging without the film. represents the standard interference image on the film, represents the true image of the interference on the film, represents the input intact image, represents the reconstructed image obtained by adding the standard interference image and the image to be detected, represents a function for calculating the structural similarity between the image to be detected and the actual image of the product packaging without the film, is a hyperparameter that balances the mean squared error loss and the structural similarity loss, Represents the packaging features of images at several different scales extracted by the VGG-19 convolutional neural network model. The range is .

[0030] S66. Identify defects in the image to be detected through a defect detection algorithm and output the defects.

[0031] When the present invention is implemented, it first takes infrared images and RGB images, then calculates difference features and fusion features, and converts the difference features into attention weights through activation functions, and then multiplies and decodes the fusion features to obtain film features, so as to realize the recognition of whether there is a film in each area on the object packaging. Then, for the intact image with the film on the object, the interference image on the film and the packaging image on the object packaging are respectively extracted. Finally, the packaging image is finally repaired based on the interference image to obtain the image to be detected of the object packaging surface without the film. Finally, based on the current defect detection algorithm, the recognition of the image to be detected can be further realized.

[0032] Corresponding to the method for detecting article packaging defects provided in the above-mentioned embodiment, the present embodiment also provides a system for implementing the method for detecting article packaging defects. Since the system for detecting article packaging defects provided in the present embodiment corresponds to the method for detecting article packaging defects provided in the above-mentioned embodiment, the implementation method of the aforementioned method for detecting article packaging defects is also applicable to the system for detecting article packaging defects provided in the present embodiment, and will not be described in detail in the present embodiment.

[0033] The system comprises a processor and a memory, wherein the memory is used to store a computer program, and when the computer program is executed by the processor, a method for detecting defects in article packaging is implemented.

[0034] The above implementation methods have been described in detail. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A method for detecting article packaging defects, characterized in that: The detection method specifically comprises the following steps: S1, obtaining several groups of infrared images and RGB images containing object packaging as input images; S2, obtaining film features based on the difference features and fusion features of each area in the corresponding position of the surface of the packaging article in the input image; S3, judging whether each area on the input image belongs to a complete image according to the film characteristics; If yes, mark the area as a good image; If not, the area is marked as a defective area and output; S4, extracting interference features and intermediate features from the intact image, and multiplying the intermediate features with the intermediate features after passing through the ECA channel to obtain the interference image; S5, extracting packaging features of the intact image, and converting the packaging features into two-dimensional features after passing through several layers of RSTB modules to obtain a packaging image; S6. Use the interference image to repair the packaging image to obtain the image to be detected, and identify defects in the image to be detected.

2. The detection method according to claim 1, characterized in that: In step S1, the following steps are specifically included: S11, capturing an RGB image containing the item packaging through an RGB camera; S12, arranging a plurality of infrared cameras, and measuring a line connecting a lens of the infrared camera and a surface of the packaging of the item and a maximum angle between the surface of the packaging of the item; S13, setting an angle threshold, and using each infrared camera to photograph each area on the item packaging where the maximum angle is less than the angle threshold to obtain a plurality of groups of infrared images; S14. Mark each infrared image and the RGB image containing the corresponding position as a group of input images to obtain several groups of input images.

3. The detection method according to claim 2, characterized in that: In step S13, the angle threshold is 70 degrees.

4. The detection method according to claim 1, characterized in that: In step S2, the following steps are specifically included: S21, an encoder based on a neural network model extracts one-dimensional features at different scales of corresponding positions on the surface of the object packaging in each group of infrared images and RGB images; The calculation formula for extracting one-dimensional features of corresponding positions of infrared images and RGB images at different scales is: ； In the above formula, represents the one-dimensional features of the infrared image or RGB image extracted at the i-th scale, The range is , represents the pre-trained backbone network, Represents the input infrared image or RGB image; S22, converting each group of one-dimensional features into two-dimensional features, and then performing channel connection on the converted two-dimensional features of the infrared image and the RGB image to obtain fusion features; The calculation formula of fusion features is as follows: ； In the above formula, represents the one-dimensional feature of the extracted RGB image at the i-th scale, represents the one-dimensional feature of the extracted infrared image at the i-th scale, Indicates that the kernel is The convolutional layer of the convolutional neural network, represents the channel attention module, Indicates that the elements in the brackets are connected by channels. express function, and Respectively represent the one-dimensional features of the RGB image and infrared image processed by the SE channel at the i-th scale, Indicates fusion features; S23, calculating the difference between the one-dimensional features of the RGB image and the infrared image after passing through the SE channel at the i-th scale to obtain the difference feature; The calculation formula of the difference feature is: ； In the above formula, Indicates differential characteristics; S24. The difference feature is converted into attention weight through activation function and then multiplied with the fusion feature to obtain film features at different scales. The calculation formula of film features at different scales is: ； In the above formula, represents the film characteristics at the i-th scale, represents the activation function; S25, connecting the film features at different scales according to the channels and sending them to the convolution layer to obtain the film features; The film characteristics are calculated as follows: ； In the above formula, represents the film detection result at the i-th scale, is the upsampling operation of the convolutional neural network, represents the final obtained film characteristics, The kernel of the convolutional neural network is represented by The convolutional layer.

5. The detection method according to claim 1, characterized in that: In step S4, the following steps are specifically included: S41. Extract interference features from intact images based on a convolutional neural network model; the expression of interference features is: ； In the above formula, The kernel of the convolutional neural network is represented by The convolutional layer, I represents the input intact image, represents the ReLU activation function, represents the batch normalization function, and Represent the interference features extracted at the i-th and i-1-th scales respectively; S42, passing the interference features at each scale through several convolutional layers and activation functions to obtain intermediate features; The calculation formula of the intermediate feature is: ； In the above formula, represents the intermediate features at the i-th scale, The range is , represents the ReLU activation function, The kernel of the convolutional neural network is represented by The convolutional layer; Indicated in scale Interference features extracted from S43, multiplying the intermediate feature by the intermediate feature after passing through the ECA channel and then adding the intermediate feature to the interference feature at the corresponding scale to obtain an intermediate interference image; the calculation formula of the intermediate interference image is as follows: ； In the above formula, represents the intermediate features at the i-th scale, represents the intermediate interference image at the i-th scale, represents an efficient channel attention layer for a convolutional neural network model, Represents the interference features extracted at the i-th scale; S44, setting the number of extractions, and counting whether the current number of times of step S42 is greater than the number of extractions; If yes, the intermediate interference image is marked as an interference image and the process proceeds to step S5; If not, the intermediate interference image is marked as an interference feature, and then the process returns to step S42.

6. The detection method according to claim 1, characterized in that: In step S5, the following steps are specifically included: S51. Extract the packaging features of the intact image through the VGG-19 convolutional neural network model; the expression of the packaging features is: ； In the above formula, , , and They represent the packaging features of intact images at the 1st, 2nd, 3rd, and 4th scales extracted by the VGG-19 convolutional neural network model. Indicates that a complete image is extracted through the VGG-19 convolutional neural network model Packaging characteristics; S52, passing the packaging features through several layers of RSTB to obtain one-dimensional packaging features, and converting them into two-dimensional packaging features to obtain a packaging image; the calculation formula of the two-dimensional packaging features is as follows: ； In the above formula, Indicated in scale The two-dimensional packaging characteristics on The range is , Indicates that the intact image is extracted by the VGG-19 convolutional neural network model at scale The packaging features on Indicates that the packaging features are passed through the A layer RSTB, Indicates that the elements in brackets are converted into two-dimensional packaging features through the Patch Unembed function.

7. The detection method according to claim 1, characterized in that: In step S6, the following steps are specifically included: S61, input the interference image into several dilation rate hole convolution layers, and connect the hole convolution layers of different scales and fuse them through the convolution layer to obtain the interference fusion image; the calculation formula of the interference fusion image is: ； In the above formula, represents the interference fusion image at the i-th scale, The range is , Indicates channel connection, represents the dilation rate of the dilated convolution, , and They are the values ​​of the three dilation rates of the hole convolution. represents the interference image at the i-th scale; S62, connecting the interference fusion image with the interference features at the corresponding scale and then upsampling to obtain an upsampled interference image; the calculation formula of the upsampled interference image is: ； In the above formula, represents the upsampled interference image at the i-1th scale, represents upsampling, Represents the interference feature at the i-th scale; S63, after passing the packaging image through the convolution layer, removing the interference fusion image at the corresponding scale, connecting it with the packaging feature at the i-th scale, and upsampling it to obtain an upsampled packaging image; the calculation formula of the upsampled packaging image is: ； In the above formula, represents the upsampled packed image at the i-1th scale, Indicated in scale The packaging image on represents the interference fusion image at the i-th scale, Represents the packaging features of the intact image at the i-th scale; S64, acquiring an upsampled packed image and an upsampled interference image at scale 1 to obtain an image to be detected and a standard interference image; S65, setting a loss function for training a convolutional neural network model for acquiring an image to be detected; S66. Identify defects in the image to be detected through a defect detection algorithm and output the defects.

8. The detection method according to claim 1, characterized in that: In step S61, the , and They are 1, 2 and 5 respectively.

9. The detection method according to claim 1, characterized in that: In step S65, the loss function is expressed as: ； In the above formula, L represents the loss function of the convolutional neural network model training for acquiring the image to be detected, , , and They represent packaging loss, interference loss, packaging perception loss and image reconstruction loss respectively. , , and Respectively represent the first loss coefficient, the second loss coefficient, the third loss coefficient and the fourth loss coefficient, represents the image to be detected, This is an actual image of the product packaging without the film. represents the standard interference image on the film, represents the true image of the interference on the film, represents the input intact image, represents the reconstructed image obtained by adding the standard interference image and the image to be detected, represents a function for calculating the structural similarity between the image to be detected and the actual image of the product packaging without the film, is a hyperparameter that balances the mean squared error loss and the structural similarity loss, Represents the packaging features of images at several different scales extracted by the VGG-19 convolutional neural network model. The range is .

10. A system for implementing the method for detecting article packaging defects according to any one of claims 1 to 9, characterized in that: The method comprises a processor and a memory, wherein the memory is used to store a computer program, and when the computer program is executed by the processor, the method for detecting article packaging defects according to any one of claims 1 to 9 is implemented.

Images