Sealing Quality Evaluation System for Sealing Strips of Packaging Webs in a Packaging Machine

Abstract

An automatic heat seal quality control system (12) for automatically controlling the quality of heat seals at the longitudinal ends of a packaging web (3) of a sealing strip (6) in a heat seal station (7) of a packaging machine (1). A sealed package (5) containing a cast food product is continuously manufactured from a continuous longitudinal tube (2) filled with the cast food product. The automatic heat seal quality control system (12) is formed by folding the packaging web (3) longitudinally and overlapping its longitudinal ends, and heat sealing the overlapping longitudinal ends. This system (12) includes a visible light camera (19) designed to operate in the electromagnetic spectrum visible to the human eye in order to capture and output a visible light digital image, and captures and outputs a visible light digital image.

Description

Technical Field

[0001] The present invention generally relates to packages of cast foods produced by sequentially sealing in the transverse direction a packaging material tube with webs continuously filled with cast foods.

[0002] More specifically, the present invention relates to a method for evaluating the seal quality of a sealing strip heat-sealed to a packaging web in a packaging machine operable to produce sealed packages containing cast foods.

[0003] The present invention further relates to the calibration of a camera used to evaluate the seal quality of a sealing strip.

Background Art

[0004] As is well known, many cast foods such as fruit juice, vegetable juice, pasteurized or UHT (ultra-high temperature treated) milk, wine, etc. are sold in packages made of sterilized packaging materials.

[0005] A typical example of this type of package is the parallelepiped package for cast foods known as Tetra Brik Aseptic®, which is made by sealing and folding laminated strip-shaped packaging materials.

[0006] This packaging material has a multi-layer structure including one or more reinforcing and strengthening base layers typically made of substantially fibrous materials such as paper or cardboard or mineral-filled polypropylene materials, and both sides are covered with heat-sealing plastic material layers, such as layers of polyethylene film. In the case of aseptic packaging for long-term storable products such as UHT milk, the packaging material includes a layer of oxygen barrier material, such as aluminum foil or ethylene vinyl alcohol (EVOH) film, which layer overlaps with the layer of heat-sealing plastic material and is further covered with another layer of heat-sealing plastic material to finally form the inner surface of the package that contacts the food.

[0007] Packages of this kind are usually manufactured on a fully automatic packaging machine, also known as a filling machine, of the type shown in Figure 1 and generally referred to by reference numeral 1 as a whole. A continuous longitudinal vertical tube 2 is formed from the packaging web 3. This vertical tube 2 is sterilized by applying a chemical sterilant such as an aqueous hydrogen peroxide solution or a physical sterilization such as an electron beam. When the sterilization is complete, the residues left, especially by the chemical sterilant, are evaporated and removed from the surface of the packaging material, for example by heating. The packaging material web 3 is maintained in a sealed sterile environment, especially within an isolation chamber, and after being folded longitudinally, it is heat-sealed to form the vertical tube 2.

[0008] The vertical tube 2 is filled with a sterilized or aseptic cast food by means of a filling pipe extending into the vertical tube 2. The vertical tube 2 is advanced along the vertical direction to a forming station where it is gripped along a cross-section arranged at equal intervals by a jaw system including two or more pairs of jaws acting periodically and continuously on the vertical tube 2, and a continuous pillow pack 4 connected to each other by a lateral seal band is formed. The pillow packs 4 are then separated from each other by cutting the seal bands and conveyed to a folding station where they are mechanically folded to become finished packages 5 having, for example, a substantially parallelepiped shape.

[0009] In order to prevent the fiber material of the multilayer sheet structure of the packaging web 3 at the longitudinal edge or boundary of the packaging web 3, especially the part exposed to the external environment, from first coming into contact with and being absorbed by the poured food in the vertical tube 2 and the pillow packs 4, next, before folding the packaging web 3 to form the vertical tube 2 and heat-sealing it longitudinally, the longitudinal edge of the packaging web 3 is fluid-tightly isolated by heat-sealing a web supply sealing strip 6 of a heat-sealing plastic material to the packaging web 3 at a heat-sealing station 7.

[0010] In the embodiment shown in FIG. 2, the sealing strip 6 is flat. Its first longitudinal side portion is first heat-sealed to the inner surface of the longitudinal side portion of the packaging web 3, then folded so as to bring its opposing longitudinal edges into contact, and then the remaining (loose) longitudinal portion of the sealing strip 6 that protrudes from the longitudinal edge of the packaging web 3 and is not heat-sealed thereto is heat-sealed to the inner surface of the opposing longitudinal side portion of the packaging web 3. In this embodiment, by heat-sealing the sealing strip 6 to the longitudinal side portions on the opposite sides of the packaging web 3, the packaging web 3 is also heat-sealed longitudinally to form the longitudinal tube 2. Thereafter, the package 5 shown in FIG. 3 is formed as described above.

[0011] In another embodiment shown in FIG. 4, the sealing strip 6 is U-shaped or C-shaped, and the package 5 shown in FIG. 5 is formed.

[0012] FIG. 6 schematically shows the operations performed at the heat-sealing station 7 for heat-sealing the U-shaped sealing strip 6 to the packaging web 3.

[0013] As shown in FIG. 6, the packaging web 3 and the sealing strip 6 are first web-fed to the heat-sealing station 7, where they are heated by the first heating unit 8, and then sent to the first pressure roller, which pressure-seals the first longitudinal side portion of the sealing strip 6 to either the inner surface or the outer surface (or decorative surface) of the longitudinal (narrow-width) side portion of the packaging web 3 that remains inside the longitudinal tube 2 after the longitudinal tube 2 is formed, leaving the second (loose) longitudinal portion of the sealing strip 6 that protrudes from the longitudinal end of the packaging web 3.

[0014] The packaging material web 3 and the sealing strip 6 are then advanced to a second heating unit 10, where a second longitudinal portion of the sealing strip 6 that protrudes from a first longitudinal side of the packaging material 3 and is not heat-sealed to the packaging material web 3 is heated together with a surface of the longitudinal side of the packaging material web 3 opposite to the longitudinal side where the first longitudinal side of the sealing strip 6 is heat-sealed.

[0015] The packaging web 3 and the sealing strip 6 are then advanced to a U-shaped folding unit 11, where the second longitudinal portion of the sealing strip 6 is first U-shaped folded around the longitudinal end of the packaging web 3 and then pressure-sealed to a surface of the packaging web 3 opposite to the longitudinal side portion by a second pressure roller (not shown).

[0016] As a result, the first and second longitudinal portions of the sealing strip 6 are heat-sealed to the packaging web 3, but the longitudinal central portion of the sealing strip 6 is not heat-sealed to the packaging web 3. After the longitudinal tube 2 is formed, it unfolds around the longitudinal ends of the packaging web 3 that remain disposed within the longitudinal tube 2 to form loops that are fluidly separated. Summary of the Invention Problems to be Solved by the Invention

[0017] To ensure that the package 5 containing the cast food meets the predetermined sealing requirements, the sealing strip 6 is heat-sealed to the packaging web 3 in accordance with predetermined quality requirements in terms of geometric accuracy and adhesiveness.

[0018] To the best knowledge of the applicant, the quality of the heat-sealing of the sealing strip 6 to the packaging web 3 is currently roughly checked qualitatively and / or quantitatively by a visually trained operator by looking for characteristic low heat-sealing quality marks or patterns and / or mechanically, for example, by a tensile test.

[0019] Although the current heat seal quality check by a properly trained operator is satisfactory in many respects, the Applicant has found that there is room for improvement, and thus feels the need to develop a technology that can improve this quality check.

[0020] Accordingly, an object of the present invention is to develop a technology that enables automatic evaluation of the quality of heat sealing of a sealing strip to a packaging material, thereby reducing the involvement of human operators.

Means for Solving the Problem

[0021] This object is achieved by the present invention relating to an automatic seal quality management system and a packaging machine equipped with the same, as described in the appended claims.

Brief Description of the Drawings

[0022]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19A

Figure 19B

Figure 19C

Figure 20A

Figure 20B

Figure 21

Figure 22

Figure 23

Figure 24

Best Mode for Carrying Out the Invention

[0023] Hereinafter, the present invention will be described in detail with reference to the accompanying drawings so that those skilled in the art can realize and use it. Various modifications to the presented embodiments will be immediately apparent to those skilled in the art, and the general principles disclosed herein may be applied to other embodiments and applications, thereby not departing from the scope of protection of the present invention defined by the appended claims. Therefore, the present invention should not be limited to the described and shown embodiments, but should be given the broadest scope of protection in accordance with the described and claimed features.

[0024] Unless otherwise defined, all technical and scientific terms used herein shall have the same meaning as commonly used by those skilled in the art in the field related to the present invention. In case of conflict, the present specification including the provided definitions shall be binding. Furthermore, the examples are provided for illustrative purposes only and should not be considered limiting.

[0025] In particular, the block diagrams included in the accompanying figures and described below should not be understood as representing structural features or compositional limitations, but rather as functional features, characteristics, i.e., the essential characteristics of the device, defined by the resulting effects or functional limitations. Therefore, in order to protect its functionality (functional possibility), it can be implemented in different ways.

[0026] To facilitate the understanding of the embodiments described herein, some specific embodiments are referred to and specific terms are used to describe them. The terms used herein are for the purpose of describing only the specific embodiments and are not intended to limit the scope of the present invention.

[0027] Starting again from FIG. 6, the packaging machine 1 further comprises an automatic heat seal quality management system 12, which is designed to automatically detect, identify (recognize / classify), and evaluate the quality of the heat seal of the sealing strip 6 to the packaging web 3 based on the results, any defects that may have occurred in the longitudinal seal of the package 5 being liquid-tight or potentially becoming liquid-tight when the sealing strip 6 is heat-sealed to the packaging web 3.

[0028] The automatic heat seal quality management system 12 comprises the following components: - A sensor system 13 disposed at the heat seal station 7, the sensor system 13 being capable of outputting an output that enables the automatic detection and identification of any possible defects that may have occurred when the sealing strip 6 is heat-sealed to the packaging web 3, and thus enabling the automatic evaluation of the quality of the heat seal of the sealing strip 6 to the longitudinal end of the packaging web 3, if any, based on the identified defects; and - An electronic computing resource 14, the electronic computing resource 14 being designed to communicate with, in particular being electrically connected to, the sensor system 13, controlling its operation, receiving and processing its output, automatically detecting and identifying any possible defects that may have occurred when the sealing strip 6 is heat-sealed to the packaging web 3, and evaluating the quality of the heat seal of the sealing strip 6 to the packaging web 3 based on the identified defects if any.

[0029] The sensor system 13 comprises an industrial artificial vision system 15, also known as a computer vision system, operable to capture digital images of the packaging web 3 and the sealing strip 6. The artificial vision system 15 is an electronic device that performs machine vision functions and comprises one or more digital cameras with an integrated or external image acquisition and processing system, internal and / or external software of the camera, and a lighting system.

[0030] The sensor system 13 may optionally include a pyrometer 16 for remotely sensing the temperatures of the packaging material 3 and the sealing strip 6.

[0031] The artificial vision system 15 comprises either or both of the following configurations: - A first electronic digital image capture device 17 disposed near the first pressure roller 9 and outputting an output capable of automatically detecting and identifying possible defects occurring during the heat seal between the first longitudinal side of the sealing strip 6 and the longitudinal side of the packaging web 3; - A second electronic digital image capture device 18 disposed near the second pressure roller and outputting an output capable of automatically detecting and specifying possible defects occurring during the heat seal of the second longitudinal side of the sealing strip 6 with respect to the longitudinal side of the packaging material 3.

[0032] Each of the electronic digital image capture devices 17, 18 comprises a respective pair of digital image sensors in the form of commercially available digital cameras arranged to capture digital images of the packaging material 3 and the sealing strip 6 heat-sealed thereto.

[0033] In particular, each of the electronic digital image capture devices 17, 18 comprises the following configuration: - A first digital camera 19 (hereinafter simply referred to as a "visible light camera") designed to operate in the electromagnetic spectrum visible to the human eye, which captures and outputs a digital image (hereinafter simply referred to as a "visible light digital image"); - A second digital camera 20 (hereinafter simply referred to as an "infrared camera" (also called a thermograph, thermal imaging camera, IR camera or imager)) designed to operate in the electromagnetic spectrum invisible to the human eye, particularly in the infrared spectrum, which captures and outputs a digital image (hereinafter simply referred to as a "thermal digital image").

[0034] The visible light and infrared cameras 19, 20 of the first electronic digital image capture device 17 are arranged with respect to the packaging material web 3 and the sealing strip 6 heat-sealed thereto such that their fields of view (FOVs) are such that the visible light and infrared cameras 19, 20 capture visible light and infrared digital images of one and the same area of the surface of the packaging material 3 and of the heat-sealed surface of the sealing strip 6 where the first longitudinal side of the sealing strip 6 was first heat-sealed.

[0035] Similarly, the visible light and infrared cameras 19, 20 of the second electronic digital image capture device 18 are arranged with respect to the packaging material web 3 and the sealing strip 6 heat-sealed thereto such that their fields of view (FOVs) are such that they capture visible light and thermal digital images of one and the same area of the opposite surface of the packaging material 3 where the second longitudinal side of the sealing strip 6 is next heat-sealed.

[0036] The electronic computing resources 14 may be in the form of a centralized architecture, i.e., a single electronic processing unit to which both of the electronic digital image capture devices 17, 18 can be connected, or in the form of a distributed architecture, i.e., two or different communicating and cooperating electronic processing units to which the individual digital cameras of the electronic digital image capture devices 17, 18 can be connected, depending on the hardware and software architecture that the manufacturer deems appropriate for controlling the heat-sealing quality.

[0037] The electronic computing resource 14 is configured to operate each of the electronic digital image capture devices 17, 18 simultaneously or sequentially at subsequent temporal instants during the movement of the packaging material web 3 and the sealing strip 6 heat-sealed thereto, whereby each of the electronic digital image capture devices 17, 18 performs a series of digital image multi-capture operations, and during each operation, the visible light and infrared cameras 19, 20 of each of the electronic digital image capture devices 17, 18 operate simultaneously or sequentially to capture a pair of visible light digital images and thermal digital images. The pair of digital images consisting of the visible light digital image and the thermal digital image is a pair of digital images consisting of the visible light digital image and the thermal digital image, the pair of digital images consisting of the visible light digital image and the thermal digital image is a pair of digital images consisting of the visible light digital image and the thermal digital image, the pair of digital images consisting of the visible light digital image and the thermal digital image is a pair of digital images consisting of the visible light digital image and the thermal digital image, and the pair of digital images consisting of the visible light digital image and the thermal digital image is a pair of digital images consisting of the visible light digital image and the thermal digital image.

[0038] Figures 7 through 9 exemplarily show visible light images and thermal digital images taken by the visible light and infrared cameras 19, 20.

[0039] In particular, FIG. 7 illustratively shows a visible light digital image of the area inside the packaging material 3, which is captured by the visible light camera 19 of the first electronic digital image capture device 17 after the first longitudinal side portion of the sealing strip 6 has been heat-sealed. In the image, the narrow, thin gray vertical stripes 21 represent the entire sealing strip 6, including both the first longitudinal side portion heat-sealed to the longitudinal side portion of the packaging material 3 and the second (loose) longitudinal side portion protruding from the longitudinal end of the packaging material 3. The narrow, dark gray vertical stripe 22 to the left of the thin gray vertical stripes represents the background. The wide, dark gray vertical stripe to the right of the thin gray vertical stripes represents the packaging material 3, which consists of the narrow vertical stripe portion 23 on the left, is slightly darker and more uniform in gray than the wide vertical stripe portion on the right, and represents the area of the longitudinal side surface portion of the packaging material 3 that is adjacent to the sealing strip 6 and was heated during the heat-sealing of the first longitudinal side surface portion of the sealing strip 6, but is not covered by the sealing strip 6 and does not overlap. On the other hand, the wide vertical stripe portion 24 on the right is less dark and less uniform in gray and represents the remaining portion of the packaging material 3 that has not been heated.

[0040] FIG. 8 is similar to that shown in FIG. 7, but is an exemplary thermal digital image of the area on the inner surface side of the packaging material 3 captured by the infrared camera 20 of the first electronic digital image capture device 17 after the first longitudinal side portion of the sealing strip 6 has been heat-sealed. In the image, the same reference numerals as in FIG. 7 designate the same parts of the digital image. Similar to the image shown in FIG. 7, the narrow, light gray vertical stripes represent the entire sealing strip 6 of both the first longitudinal side portion heat-sealed to the longitudinal side portion of the packaging material 3 and the loose longitudinal side portion protruding from the longitudinal end portion of the packaging material 3. The wide, dark gray vertical stripe to the right of the light gray vertical stripe represents the packaging material 3, and the narrow, dark gray vertical stripe to the left of the light gray vertical stripe represents the background. Also, in this image, the wide, dark gray vertical stripe on the right consists of a narrow vertical stripe portion on the left that is not as dark as the wide vertical stripe portion on the right, and represents the heating pattern. The wide vertical stripe portion on the right that is slightly darker than the narrow vertical stripe portion on the left represents the remaining portion of the packaging material 3 that has not been heated.

[0041] FIG. 9 exemplarily shows a visible light digital image similar to that shown in FIG. 7, but captured from the outside or decorative side of the packaging material 3 by the visible light camera 19 of the second electronic digital image capture device 18 after the first longitudinal side portion of the sealing strip 6 has been heat-sealed. In the image, the same reference numerals as in FIGS. 7 and 8 designate the same parts of the digital image. Similar to the images shown in FIGS. 7 and 8, the narrow, light gray vertical stripes represent the loose longitudinal side portions protruding from the longitudinal end portions of the packaging material 3. The narrow, dark gray vertical stripe to the left of the light gray vertical stripe represents the background, and the wide, light gray vertical stripe to the right of the light gray vertical stripe represents the packaging material 3.

[0042] In one embodiment, the electronic computing resource 14 is further configured to process the captured digital image to automatically detect and identify possible defects that may have occurred when heat-sealing the sealing strip 6 to the packaging material 3 based on the captured digital image.

[0043] In particular, FIG. 10 shows a block diagram of image processing executed by the electronic computing resource 14 for visible light digital images and corresponding thermal digital images of the same regions of the inner side of the packaging web 3 and the sealing strip 6, as shown in FIGS. 7 and 8.

[0044] In particular, the electronic computing resource 14 is programmed to receive a set of visible light digital images and thermal digital images, and for each set of visible light digital images and thermal digital images, execute both the image processing of the received visible light digital image and the image processing of the set of visible light digital images and thermal digital images.

[0045] Regarding the image processing of the visible light digital image, the electronic computing resource 14 is programmed as follows: - Receive a pair of visible light digital image and thermal digital image (block 25); - Process the received visible light digital image and search for the following: · Marks or patterns (block 26) representing typical rough, major or macro defects or anomalies such as scratches or peeling, exemplified in FIGS. 11a and 11b, occurring in the heating pattern when heat-sealing the sealing strip 6 to the packaging web 3, and · Marks or patterns (block 27) representing typical fine, minor or micro defects or anomalies such as specific wrinkles exemplified in the visible light digital image shown in FIGS. 11c - 11f, resulting from the heating pattern during heat-sealing of the sealing strip 6 to the packaging web 3.

[0046] Output data representing the identified marks or patterns representing typical macro and micro defects or anomalies is calculated and input to the inner classifier (block 28). If there are identified marks or patterns representing typical macro and micro defects or anomalies, the heat-sealing of the sealing strip 6 to the packaging web 3 is evaluated based thereon.

[0047] To search for marks and patterns representing typical micro defects and anomalies, the captured visible light digital image is processed to search for the following: - In the region of the captured visible light digital image corresponding to the heating pattern, diagonal wrinkles (block 29) as shown in the captured visible light digital image shown in FIG. 11c, particularly at the longitudinal ends of the sealing strip 6, particularly at the longitudinal ends of the longitudinal side portions of the sealing strip 6 heat-sealed to the packaging web 3, inclined at approximately 45° with respect to the longitudinal direction. This diagonal wrinkle is caused by overheating of the heating pattern when heat-sealing the sealing strip 6 to the packaging web 3; and - Substantially orthogonal wrinkles (block 30), hereinafter referred to as horizontal wrinkles, as shown in the digital image shown in FIG. 11e, particularly at the longitudinal ends of the sealing strip 6, particularly at the longitudinal ends of the longitudinal side portions of the sealing strip 6 heat-sealed to the packaging web 3, inclined at approximately 90° with respect to the longitudinal direction, in either one or both of the region of the captured visible light digital image corresponding to the sealing strip 6 and the region corresponding to the heating pattern as shown in the visible light digital image shown in FIG. 11d. This diagonal wrinkle is due to process defects generated when heat-sealing the sealing strip 6 to the packaging web 3.

[0048] Output data representing the identified diagonal and horizontal wrinkles is calculated and input to the inner classifier (block 28).

[0049] As shown in FIG. 10, the visible light digital image is further processed as follows (block 31): - Save or receive data indicating a width tolerance range indicating the expected heating pattern width; - Calculate the actual heating pattern width measured in a direction orthogonal to the longitudinal end of the sealing strip 6; - Check the actual heating pattern width against the width tolerance range to determine whether the heating pattern width is within or outside the width tolerance range. - Perform a calculation to indicate whether the heating pattern width is within the width tolerance range or outside the width tolerance range. - Input the calculated data indicating whether the heating pattern width is within the width tolerance range or outside the width tolerance range into the inner classifier, and the heat seal of the sealing strip 6 to the packaging web 3 is evaluated based on whether the heating pattern width is within the width tolerance range or outside the width tolerance range.

[0050] In another embodiment, the calculated heating pattern width may be provided to the inner classifier, and the heat seal of the sealing strip 6 to the packaging web 3 is evaluated based also on the calculated heating pattern width.

[0051] The visible light digital image is processed as follows (block 32): - Save or receive data indicating an overlap tolerance range showing the range where the sealing strip 6 overlaps the longitudinal end of the packaging web 3; - Calculate the actual overlap of the sealing strip by measuring in a direction perpendicular to the longitudinal end of the sealing strip 6 and showing the range where the sealing strip 6 actually overlaps the longitudinal end of the packaging web 3; - Compare the overlapping part of the actual sealing strip with the overlapping part tolerance range to determine whether the overlapping part of the actual sealing strip is within or outside the overlapping part tolerance range; - Calculate data indicating whether the overlapping part of the actual sealing strip is within the overlapping part tolerance range or outside the overlapping part tolerance range; - Input the calculated data indicating whether the overlapping part of the actual sealing strip is within the overlapping part tolerance range or outside the overlapping part tolerance range into the inner classifier, and when heat-sealing the sealing strip 6 to the packaging web 3, it is evaluated based on whether the calculated overlapping part of the actual sealing strip is within the overlapping part tolerance range or outside the overlapping part tolerance range.

[0052] In another embodiment, the calculated actual overlap of the sealing strip may be provided to the inner classifier, and the heat seal of the sealing strip 6 to the packaging web 3 is evaluated based on the calculated actual overlap of the sealing strip.

[0053] The actual heating pattern width and the overlap of the sealing strip are calculated by determining the number of pixels in the direction orthogonal to the longitudinal ends of the sealing strip 6 in the region corresponding to the heating pattern and the sealing strip of the processed visible light digital image, and multiplying the number of pixels by a predetermined conversion coefficient that converts the number of pixels to the heating pattern width and the number of pixels of the overlap of the sealing strip expressed in millimeters.

[0054] Advantageously, but not necessarily, the processing of the captured visible light digital image to search for marks or patterns representing typical microscopic defects or anomalies may first be performed by segmenting the visible light digital image into a plurality of segments or regions (sets of pixels, also called image objects) corresponding to the sealing strip 6 (block 33), thereby outputting a segmented visible light digital image consisting only of the regions corresponding to the sealing strip 6 and the heating pattern as shown in FIG. 12, and then processing the segmented visible light digital image to search for the above-mentioned microscopic defects or anomalies.

[0055] In particular, the region corresponding to the sealing strip 6 is identified by detecting a significant variation in pixel luminance in the direction orthogonal to the longitudinal end of the sealing strip 6 as compared to adjacent regions, and the region of the heating pattern is identified by evaluating the color uniformity as compared to adjacent regions corresponding to the remaining part of the packaging web 3 with high color dispersion and average gray level.

[0056] The processing of visible - light digital images to search for marks or patterns representing typical macro defects or anomalies is advantageously performed based on semi - supervised learning and generative modeling, particularly based on an artificial neural network architecture known as a variational auto - encoder (VAE), which is learned based on a dataset containing only visible - light digital images without macro defects or anomalies, as shown particularly in FIG. 13. The visible - light digital images depicted on the right side of FIG. 13 show a non - abnormal scenario with a small reconstruction error and a high reconstruction probability, i.e., the output of the VAE when no typical macro defects or anomalies are depicted in the processed visible - light digital image, and an abnormal scenario with a large reconstruction error and a low reconstruction probability, as shown in the upper left of FIG. 13, i.e., the VAE output when typical macro defects or anomalies are depicted in the processed visible - light digital image, as shown in the lower left of FIG. 13.

[0057] The processing of visible - light digital images to search for horizontal wrinkles in the heating pattern is based on a convolutional neural network (CNN) having the architecture of UNet, exemplarily shown in FIG. 14, and is advantageously learned based on a labeled dataset created and schematically shown in FIG. 15.

[0058] The processing of captured visible - light digital images for searching diagonal wrinkles is advantageously based on a convolutional neural network (CNN) architecture trained based on a dataset containing digital images with diagonal wrinkles and digital images without diagonal wrinkles, exemplarily shown in FIG. 16.

[0059] Referring again to the block diagram shown in FIG. 10, with respect to the image processing of visible - light digital images and thermal digital images, the electronic computing resource 14 is programmed as follows: - Process the received visible light digital image (block 34) to calculate data including information indicating the geometric shape of the heating pattern of the sealing strip 6 and the packaging web 3, hereinafter simply referred to as "geometric information", and in particular, appropriate points of the processed visible light digital image, for example, in a reference frame with the origin of the coordinate system, the longitudinal edges of the sealing strip 6, the longitudinal edges of the packaging web 3, and the positions of its heating pattern. The appropriate points are, for example, either the longitudinal edge of the loose longitudinal portion of the sealing strip 6 protruding from the packaging web 3 or the longitudinal edge of the packaging web 3; - Process the received thermal digital image (block 34) to calculate data including information indicating the temperature profile of the packaging web 3 and the sealing strip 6 heat-sealed thereto in a direction orthogonal to the longitudinal ends of the packaging web 3 and the sealing strip 6 heat-sealed thereto, hereinafter simply referred to as "thermal information", in a reference frame having an origin, conveniently in the form of a coordinate system, and information indicating the temperature profile of the packaging web 3 and the sealing strip 6 heat-sealed thereto in a reference frame having an origin at either the longitudinal end of the sealing strip 6 not heat-sealed to the packaging web 3 or the longitudinal end of the packaging web 3, hereinafter simply referred to as "thermal information". The appropriate one is, for example, either the longitudinal end of the sealing strip 6 not heat-sealed to the packaging web 3 or the longitudinal end of the packaging web 3; - Fuse (data fusion), i.e., combine or merge (block 34), the output data of the visible light digital image analysis and the output data of the thermal digital image analysis to calculate fusion data representing the combination of the geometric information obtained from the visible light digital image analysis and the thermal information obtained from the thermal digital image analysis, based on which the cold seal area or overheating area generated when heat-sealing the sealing strip 6 to the packaging web 3 can be identified; and - Process the fused data to identify potential underheating or cold seal (block 35) or overheating (block 36) that may have occurred when heat-sealing the sealing strip 6 to the packaging web 3, calculate output data representing the identified underheating or overheating, and input it to the inner classifier (block 28).

[0060] FIG. 17 is a graphical display of the fusion of output data from visible light digital image analysis and output data from thermal digital image analysis.

[0061] As can be understood, the thermal information from the thermal digital image analysis in the form of a temperature profile in a direction orthogonal to the longitudinal edges of the packaging web 3 and the sealing strip 6 heat-sealed thereto is superimposed on the graphic representation of the geometric information obtained from the visible light digital image analysis as the geometric shape of the sealing strip 6 and the shape of the heating pattern of the packaging web 3, thereby enabling the identification of underheating (cold seal) or overheating regions that occurred when heat-sealing the sealing strip 6 to the packaging web 3.

[0062] Next, in the inner classifier (block 28), the quality of the heat seal of the sealing strip 6 to the inner surface of the packaging web 3 is evaluated based on a unique criterion developed to take into account the heating pattern and macro and micro defects (if any) identified in the sealing strip 6, the calculated heating pattern width and strip overlap, and the identified underheating (cold seal) or overheating (if any), as well as the severity of the above-indicated amounts.

[0063] Regarding the calculation of the temperature profiles of the packaging web 3 and the sealing strip 6, with exemplary reference to the thermal digital image shown in FIG. 8, the temperature profiles of the packaging web 3 and the sealing strip 6 are first determined by identifying the left longitudinal boundary of the sealing strip 6 in the thermal digital image, and then, using a custom algorithm, proceeding from left to right in a direction orthogonal to the sealing strip 6, the temperature values are calculated based on the lightness values of the primary colors of the pixels of the thermal digital image from the identified left longitudinal boundary of the sealing strip 6 to the right longitudinal boundary of the sealing strip 6 and across the entire heating pattern.

[0064] In order for the calculated geometric information and thermal information to be correctly fused, the calculated geometric information and thermal information need to be first aligned in the spatial domain, particularly in both the direction parallel to and the direction orthogonal to the sealing strip 6.

[0065] When the calculated geometric information and thermal information are aligned, the visible light digital image and the thermal digital image can be conceptually superimposed, that is, the packaging web 3 and the sealing strip 6 depicted in the visible light digital image can be conceptually superimposed with the packaging web 3 and the sealing strip 6 depicted in the thermal digital image.

[0066] FIG. 18 exemplarily shows a composite visible light and thermal digital image obtained by superimposing a visible light digital image and a corresponding thermal digital image.

[0067] For that purpose, the reference frames of the visible light digital image and the thermal digital image, to which the geometric information and the thermal information are referred, need to be spatially / geometrically related to each other via a so-called spatial transformation matrix that represents the mutual rotational transformation of the two reference frames so that the sealing strip 6 and the packaging web 3 in the visible light digital image can be related to the sealing strip 6 and the packaging web 3 in the thermal digital image.

[0068] Calibration of the visible light and infrared cameras 19, 20 of each of the electronic digital image capture devices 17, 18 is performed to calculate a transformation matrix used for alignment of each set of visible light digital images and thermal digital images during execution of heat seal quality control.

[0069] Calibration of the visible light and infrared cameras is performed using a calibration marker 21 provided with or designed to show a calibration pattern 22, the calibration marker 21 being stably disposed within the fields of view (FOV) of the visible light and infrared cameras 19, 20 of each of the electronic digital image capture devices 17, 18, such that when the visible light and infrared cameras 19, 20 are operated simultaneously or sequentially, the calibration pattern 22 is imaged, i.e., visualized, in both the visible light digital image and the thermal digital image captured by the visible light and infrared cameras 19, 20 respectively, whereby a spatial transformation matrix can be calculated based on the position and orientation of the pattern imaged in the visible digital image and the thermal digital image.

[0070] In the embodiment shown in FIGS. 19A - 19C, the calibration marker 21 includes a foreground member 23 disposed in the foreground with respect to the visible light / infrared cameras 19, 20 and carrying the calibration pattern 22 in the form of a through pattern, and a background member 24 disposed behind the foreground member 23 with respect to the visible light / infrared cameras 19, 20 and exposed, i.e., visible, through the through pattern 22.

[0071] In the embodiment shown in FIGS. 19A - 19C, the background member 24 is in the form of a foil, e.g., an aluminum foil, and the foreground member 23 includes a plate 25 in which the calibration pattern 22 is formed, and a support and coupling structure 26 designed to be integrally with the plate 25 and to easily and removably couple / hold the background member 24 behind the plate 25 and to stably dispose the plate 25 with the background member 24 coupled thereto within the fields of view of the visible and infrared cameras 19, 20.

[0072] In the embodiment shown in FIGS. 19A to 19C, the calibration pattern 22 is in the form of a grid of through holes in offset columns, i.e., a grid of through holes where the through holes in a column are offset with respect to the through holes in adjacent columns.

[0073] The foreground member 23 and the background member 24 are made of materials having different emissivities. In particular, the foreground member 23 is made of a material having an emissivity lower than (higher) emissivity of the material of the background member 24, for example, aluminum, for example, PLA plastic. Thus, when the background member 24 is appropriately heated, the heated area of the background member 24 exposed through the through pattern 22 is imaged in the thermal digital image to form a corresponding calibration pattern, while the remaining portion of the background member 24 is shielded by the foreground member 23 and thus not imaged in the thermal digital image.

[0074] In this way, the calibration pattern 22 formed on the foreground member 23 can be imaged in the visible light digital image, and after the background member 24 is appropriately heated, the corresponding pattern formed by the heated area of the background member 24 exposed through the calibration pattern 22 formed on the foreground member 23 can also be imaged in the thermal digital image.

[0075] FIG. 20A shows the calibration pattern 22 formed on the foreground member 23 and imaged in the visible light digital image, and FIG. 20B shows the corresponding pattern formed by the heated area of the background member 24 exposed through the calibration pattern 22 formed on the foreground member 23 and imaged in the thermal digital image.

[0076] Therefore, calibration of the output data of the visible and infrared cameras 19, 20 of each electronic digital image capture device 17, 18 is required as follows: - Heating the background member 24 of the calibration marker 21; - Placing the calibration marker 21 within the fields of view of the visible light camera 19 and the infrared camera 20; - Operating the visible light camera 19 and the infrared camera 20 to capture a visible light digital image and a thermal digital image; - Process the captured visible light digital image and thermal digital image to identify the calibration pattern contained therein; - Calculate the position and orientation of the calibration pattern displayed in the visible light image and the digital image; - Calculate a spatial transformation matrix based on the positions and orientations of the calibration patterns in the visible light image and the digital image.

[0077] For this purpose, the electronic computing resource 14 is further programmed as follows: - Actuate the visible light camera 19 and the infrared camera 20 simultaneously or sequentially to capture visible light digital images and thermal digital images; - Process the captured visible light digital image and thermal digital image to identify the calibration pattern contained therein and calculate its position and orientation. - Calculate a spatial transformation matrix based on the positions and orientations of the calibration patterns included in the visible light image and the digital image.

[0078] FIG. 21 is a block diagram of image processing executed by the electronic computing resource 14, which captures visible light digital images and thermal digital images of the outer surface or decorative surface of the packaging web 3 and the same area of the sealing strip 6, and the visible light digital image is shown in FIG. 9.

[0079] For the sake of convenience, in the block diagram shown in FIG. 21, the blocks related to the processes similar or corresponding to the processes described above with reference to the block diagram shown in FIG. 10 are labeled with the same reference numerals as those in the block diagram shown in FIG. 10.

[0080] As shown in FIG. 21, it is determined whether the visible light digital image is processed and whether the sealing strip 6 is U-folded around the longitudinal end of the packaging web 3 (block 37), and whether one or more marks similar to the "fishbone" as shown in FIG. 21 are formed between the U-folded and heat-sealed sealing strip 6 on the packaging web 3 (block 38).

[0081] If it is determined that the sealing strip 6 is U-shaped folded, as described above with reference to block 32 shown in FIG. 10, the actual overlap of the sealing strip measured in a direction orthogonal to the longitudinal end of the sealing strip 6 is calculated (block 39).

[0082] Instead, a thermal digital image is identified with an area of the thermal digital image having predefined characteristics, in particular an area having a predefined luminance pattern such as the area surrounded by the red rectangle shown in the thermal digital image reproduced in FIG. 21 (block 41), and the identified area of the thermal digital image is processed to search for insufficient heating (cold seal) that may have occurred when heat-sealing the sealing strip 6 to the packaging web 3, as previously explained with reference to block 35 shown in FIG. 10 (block 42).

[0083] Next, the quality of the heat seal of the sealing strip 6 to the packaging web 3 is evaluated in an outer classifier (block 40) similar to the inner classifier shown in FIG. 10 with reference to block 28, based on the aforementioned features and its own criteria considering the significance of the aforementioned features.

[0084] To detect the presence and number of "fishbones" in the visible light digital image, a deep learning algorithm based on the UNet network exemplarily shown in FIG. 22 is used and trained based on the dataset created and labeled as schematically shown in FIG. 23.

[0085] Finally, FIG. 24 shows a block diagram of the image processing executed by the electronic computing resource 14 for the captured visible light digital image and thermal digital image of the longitudinal seal area of the longitudinal tube 2 after the packaging web 3 is folded and its longitudinal ends are heat-sealed.

[0086] As shown in FIG. 24, the visible light digital image is processed (block 41) to identify an area of the thermal digital image having predefined characteristics, in particular an area having a predefined luminance pattern, such as the area enclosed by the red rectangle shown in the visible light digital image reproduced in FIG. 21. The identified area of the visible light digital image is then processed to determine whether the vertical tube 2 is twisted in the vertical seal (block 42) and to determine whether the longitudinal ends of the packaging web 3 are incorrectly overlapped (block 43).

[0087] Output data representing whether the vertical tube 2 is twisted in the vertical seal or the longitudinal ends of the packaging web 3 are incorrectly overlapped is calculated and input to a vertical seal classifier (block 44) similar to the inner and outer classifiers shown in FIGS. 10 and 21.

[0088] Instead, the thermal digital image is processed to search for marks indicating insufficient heating (cold seal) that occurred when heat-sealing the sealing strip 6 to the packaging web 3, as previously described with reference to block 35 shown in FIG. 10.

[0089] Output data representing the identified marks indicating insufficient heating (cold seal) is calculated and input to the vertical seal classifier (block 44), and the quality of the vertical seal of the vertical tube 2 is evaluated based on a unique criterion considering the severity of the identified features.

[0090] The advantages achieved by the present invention will be readily understood by those skilled in the art.

[0091] In particular, the present invention enables the quality of the heat-sealing of the sealing strip to the packaging web to be automatically checked by a simple, robust and reliable optical image technique, which is significantly improved compared to the heat-sealing quality check performed by a properly trained operator.

Claims

1. An automatic heat seal quality control system (12) for automatically controlling the quality of heat sealing of a sealing strip (6) to a longitudinal edge of a packaging web (3) in a heat seal station (7) of a packaging machine (1), wherein a sealed package (5) containing a cast food is continuously produced from a continuous vertical tube (2) filled with a cast food, and the packaging web (3) is bent in the longitudinal direction to overlap its longitudinal edges, and formed by heat-sealing the overlapped longitudinal edges, The system (12) is, A sensor system (13) disposed in the heat seal station (7) and outputting an output capable of evaluating the quality of heat sealing of the sealing strip (6) to the longitudinal end of the packaging web (3), An electronic computing resource (14) that communicates with the sensor system (13), receives and processes the output, and automatically evaluates the quality of heat sealing of the sealing strip (6) to the longitudinal end of the packaging web (3), Comprising, The sensor system (13) comprises an artificial vision system (15) for capturing a digital image of the packaging web (3) and the sealing strip (6), The artificial vision system (15) is, A visible light camera (19) designed to operate in the electromagnetic spectrum visible to the human eye to capture and output a visible light digital image, Comprising, System (12).

2. The electronic computing resource (14) is, Operating the visible light camera (19) to capture a visible light digital image of the packaging web (3) and the sealing strip (6), Receiving the captured visible light digital image, Processing the received visible light digital image to, A mark or pattern (26) representing macro defects or anomalies such as scratches or delamination that may be formed during heat sealing of the sealing strip (6) to the packaging web (3), and A mark or pattern (27) representing micro defects or anomalies such as wrinkles that may be formed during heat sealing of the sealing strip (6) to the packaging web (3), Searching for at least one of, Evaluating the quality of heat sealing of the sealing strip (6) to the packaging web (3) based on the identified mark or pattern, The system (12) according to claim 1.

3. The electronic computing resource (14) further processes the received visible light digital image to search for marks or patterns representing macro and micro defects or anomalies in the heating pattern of the packaging web (3), which is defined as the area of the packaging web (3) that is adjacent to the sealing strip (6) and is heated during heat-sealing of the sealing strip (6) to the packaging web (3), but is not covered by the sealing strip (6). The system (12) according to claim 1 or 2. Claim 4 The electronic computing resource (14) processes the received visible light digital image to search for marks or patterns representing the micro defects or anomalies. Diagonal wrinkles (29) in the heating pattern of the packaging web (3); Horizontal wrinkles (30) in one or both of the heating pattern of the sealing strip (6) and the packaging web (3). Search for The system (12) according to claim 3. Claim 5 The electronic computing resource (14) processes the received visible light digital image to (31), calculates the actual heating pattern width measured in a direction orthogonal to the sealing strip (6) of the heating pattern of the area of the packaging web (3) that is adjacent to the sealing strip (6) and is heated during heat-sealing of the sealing strip (6) to the packaging web (3), but is not covered by the sealing strip (6), and evaluates the quality of the heat-sealing of the sealing strip (6) to the packaging web (3) based on the calculated heating pattern width. The electronic computing resource (14) processes the received visible light digital image to (31), calculates the actual heating pattern width measured in a direction orthogonal to the sealing strip (6) of the heating pattern of the area of the packaging web (3) that is adjacent to the sealing strip (6) and is heated during heat-sealing of the sealing strip (6) to the packaging web (3), but is not covered by the sealing strip (6). Based on the calculated heating pattern width, evaluate the quality of the heat-sealing of the sealing strip (6) to the packaging web (3). The system (12) according to any one of claims 1 to 4. Claim 6 The electronic computing resource (14) processes the received visible light digital image to (32), calculates the actual overlap of the sealing strip, which is measured in a direction orthogonal to the sealing strip (6) and indicates the range where the sealing strip (6) actually overlaps the packaging web (3), and evaluates the quality of the heat-sealing of the sealing strip (6) to the packaging web (3) based on the calculated actual overlap of the sealing strip. The electronic computing resource (14) processes the received visible light digital image to (32), calculates the actual overlap of the sealing strip, which is measured in a direction orthogonal to the sealing strip (6) and indicates the range where the sealing strip (6) actually overlaps the packaging web (3). Evaluate the quality of the heat-sealing of the sealing strip (6) to the packaging web (3) based on the calculated actual overlap of the sealing strip. The system (12) according to any one of claims 1 to 5. Claim 7 The electronic computing resource (14) processes the received visible light digital image to (37). Determine whether a fishbone-like mark is formed on the sealing strip (6) while the sealing strip (6) is U-folded around the longitudinal end of the packaging web (3) and the sealing strip (6) is U-folded and heat-sealed to the packaging web (3). If it is determined that the sealing strip (6) is U-shaped folded, calculate the actual overlapping portion of the sealing strip, which is measured in a direction perpendicular to the sealing strip (6) and indicates the range where the sealing strip (6) actually overlaps the packaging web (3). Evaluate the quality of the heat seal of the sealing strip (6) to the packaging web (3) based on the determination of whether the sealing strip (6) is U-folded around the longitudinal end of the packaging web (3), the determination of whether a fishbone-like mark is formed on the sealing strip (6) during the U-folding of the sealing strip (6) and the heat sealing to the packaging web (3), and the calculated actual overlapping portion of the sealing strip. The system (12) according to any one of claims 1 to 6.

8. The artificial vision system (15) Further includes an infrared camera (20) designed to operate in the electromagnetic spectrum invisible to the human eye, capture, and output a thermal digital image. And further comprises The electronic computing resource (14) Operates the infrared camera (20), captures thermal digital images of the packaging web (3) and the sealing strip (6). Receives the captured thermal digital images. Processes (41) the received thermal digital images. Identifies from the thermal digital images regions with pre-defined characteristics, particularly regions with pre-defined luminance patterns. Processes the identified regions of the thermal digital images to identify the heat signature of the heat seal at the longitudinal ends of the sealing strip (6), particularly the possible underheating or cold seal that may have occurred during the heat sealing of the sealing strip (6) to the packaging web (3). Evaluates the quality of the heat seal of the sealing strip (6) to the packaging web (3) based on the identified underheating or cold seal. Comprising The system (12) according to any one of claims 1 to 7.

9. The visible light camera (19) and the infrared camera (20) are arranged with respect to the packaging web (3) and the sealing strip (6) to capture visible light and thermal digital images of the same region on the same surface of the packaging web (3) and the sealing strip (6), The electronic computing resource (14) is, operate the visible light / infrared cameras (19, 20) to capture a pair of visible light / thermal digital images of the packaging web (3) and the sealing strip (6), receive the pair of captured visible light digital image and infrared digital image, perform visible light digital image analysis (34) on the received visible light digital image, and calculate and output data including geometric information representing the geometric shape of the sealing strip (6) and the heating pattern of the packaging web (3) in a reference frame with an appropriate point of the processed visible light digital image as the origin, perform thermal digital image analysis (34) on the received thermal digital image, and calculate and output data including thermal information representing the temperature profiles of the packaging web (3) and the sealing strip (6) in a direction orthogonal to the packaging web (3) and the sealing strip (6) in a reference frame having an origin at an appropriate position of the processed thermal digital image, fuse the geometric information obtained from the visible light digital image analysis and the thermal information obtained from the thermal digital image analysis, and calculate fusion data representing the fusion information, process the fused data to identify possible underheating or cold seal (35) or overheating (36) that occurred during the heat sealing of the sealing strip (6) to the packaging web (3), evaluate the quality of the heat seal of the sealing strip (6) to the packaging web (3) based on the identified underheating or overheating, The system (12) according to claim 8.

10. The electronic computing resource (14) aligns the geometric information with the thermal information in a spatial region in both a direction parallel to the sealing strip (6) and a direction orthogonal thereto such that the calculated thermal information is related to the same spatial point of the packaging web (3) and the sealing strip (6) to which the calculated visible light information is related, In order to align the geometric information with the thermal information in the spatial region, the electronic computing resource (14) calculates a spatial transformation matrix that correlates the reference frames in which the geometric information and the thermal information are referenced in the visible light digital image and the thermal digital image, whereby the images of the sealing strip (6) and the packaging web (3) in the visible light digital image are associated with the sealing strip (6) and the packaging web (3) captured in the thermal digital image, In order to calculate the spatial transformation matrix, the system (12) comprises a calibration marker (21) showing a calibration pattern (22) that can be imaged in both the visible light digital image and the thermal digital image, The electronic computing resource (14) calculates the spatial transformation matrix based on the positions and orientations of the calibration patterns (22) captured in both the visible light digital image and the thermal digital image, The system (12) according to claim 8 or 9.

11. The calibration marker (21) is arranged in the foreground with respect to the visible light / infrared cameras (19, 20) and comprises a foreground member (23) intended to carry the calibration pattern (22) in the form of a through pattern, and a background member (24) intended to be arranged behind the foreground member (23) with respect to the visible light / infrared cameras (19, 20) so as to be exposed in the foreground through the calibration pattern (22), The foreground member (23) and the background member (24) are formed of materials having different emissivities. The calibration pattern (22) formed on the foreground member (23) is captured in the visible light digital image. When the background member (24) is heated, the heated area of the background member (24) exposed in the foreground through the calibration pattern (22) formed on the foreground member (23) captured in the visible light digital image is captured in the thermal digital image and can form a corresponding calibration pattern. The remaining portion of the background member (24) is shielded by the foreground member (23) and thus not captured in the thermal digital image, The system (12) according to claim 10.

12. An operating packaging machine (1) capable of continuously manufacturing a sealed package (5) containing a cast food from a continuous longitudinal tube (2) formed by filling a cast food, bending a packaging web (3) in the longitudinal direction, overlapping its longitudinal edges, and then heat-sealing the overlapped longitudinal edges, The packaging machine (1) comprises an automatic heat-sealing quality management system (12) according to any one of claims 1 to 11, Packaging machine (1).

Images