Evaluation of the sealing quality of a sealing strip sealed to a web by a packaging machine

Abstract

An automatic heat seal quality control system (12) for automatically controlling the quality of heat seals at the longitudinal ends of a packaging web (3) of a sealing strip (6) in a heat seal station (7) of a packaging machine (1). A sealed package (5) containing a cast food product is continuously manufactured from a continuous longitudinal tube (2) filled with the cast food product. The automatic heat seal quality control system (12) is formed by folding the packaging web (3) longitudinally and overlapping its longitudinal ends, and heat-sealing the overlapping longitudinal ends. This system (12) includes a visible light camera (19) designed to operate in the electromagnetic spectrum visible to the human eye for capturing and outputting a visible light digital image, and an infrared camera (20) designed to operate in the electromagnetic spectrum invisible to the human eye.

Description

Technical Field

[0001] The present invention generally relates to packages of cast foods produced by sequentially sealing in the transverse direction a tube of packaging material with webs filled continuously with cast foods.

[0002] More specifically, the present invention relates to a method for evaluating the sealing quality of a sealing strip heat-sealed to a packaging web in a packaging machine operable to produce sealed packages containing cast foods.

[0003] The present invention further relates to the calibration of a camera used to evaluate the sealing quality of a sealing strip.

Background Art

[0004] As is well known, many cast foods such as fruit juice, vegetable juice, pasteurized or UHT (ultra-high temperature treated) milk, wine, etc. are sold in packages made of sterilized packaging materials.

[0005] A typical example of this type of package is the parallelepiped package for cast foods known as Tetra Brik Aseptic®, which is made by sealing and folding laminated strip-shaped packaging materials.

[0006] This packaging material has a multi-layer structure including one or more reinforcing and strengthening base layers typically made of substantially fibrous materials such as paper or cardboard or mineral-filled polypropylene materials, and both sides are covered with heat-sealing plastic material layers, such as layers of polyethylene film. In the case of aseptic packaging for long-term storable products such as UHT milk, the packaging material includes a layer of oxygen barrier material, such as aluminum foil or ethylene vinyl alcohol (EVOH) film, which layer overlaps with the layer of heat-sealing plastic material and is further covered with another layer of heat-sealing plastic material to finally form the inner surface of the package in contact with the food.

[0007] Packages of this kind are usually manufactured on a fully automatic packaging machine, also known as a filling machine, of the type shown in Figure 1 and generally referred to by the reference numeral 1 as a whole. A continuous vertical tube 2 is formed from the packaging web 3, and this vertical tube 2 is sterilized by applying a chemical sterilant such as an aqueous hydrogen peroxide solution or a physical sterilization such as an electron beam. When the sterilization is complete, the residues left, especially by the chemical sterilant, are evaporated and removed from the surface of the packaging material, for example by heating. The packaging material web 3 is maintained in a sealed sterile environment, especially within an isolation chamber, and after being folded longitudinally, it is heat-sealed to form the vertical tube 2.

[0008] The vertical tube 2 is filled with a sterilized or aseptic cast foodstuff by means of a filling pipe extending into the vertical tube 2. The vertical tube 2 is advanced vertically along to a forming station where, by means of a jaw system comprising two or more pairs of jaws acting periodically and continuously on the vertical tube 2, it is gripped along a cross-section arranged at equal intervals and a continuous pillow pack 4 is formed which is connected to each other by a transverse seal band. The pillow packs 4 are then separated from each other by cutting the seal bands and conveyed to a folding station where they are mechanically folded to form a finished package 5, for example substantially in the shape of a parallelepiped.

[0009] In order to prevent the fiber material of the multilayer sheet structure of the packaging web 3 at the longitudinal edge or boundary of the packaging web 3, especially the part exposed to the external environment, from first coming into contact with and being absorbed by the poured-out foodstuff within the vertical tube 2 and the pillow packs 4, next, before folding the packaging web 3 to form the vertical tube 2 and heat-sealing it longitudinally, the longitudinal edges of the packaging web 3 are fluid-tightly isolated by heat-sealing a web supply sealing strip 6 of heat-sealing plastic material to the packaging web 3 at a heat-sealing station 7.

[0010] In the embodiment shown in FIG. 2, the sealing strip 6 is flat. First, its first longitudinal side portion is heat-sealed to the inner surface of the longitudinal side portion of the packaging web 3, then it is folded so as to bring its opposing longitudinal edges into contact, and then the remaining (loose) longitudinal portion of the sealing strip 6 that protrudes from the longitudinal edge of the packaging web 3 and is not heat-sealed thereto is heat-sealed to the inner surface of the opposing longitudinal side portion of the packaging web 3. In this embodiment, by heat-sealing the sealing strip 6 to the longitudinal side portions on the opposite sides of the packaging web 3, the packaging web 3 is also heat-sealed longitudinally to form the longitudinal tube 2. Thereafter, the package 5 shown in FIG. 3 is formed as described above.

[0011] In another embodiment shown in FIG. 4, the sealing strip 6 is U-shaped or C-shaped, and the package 5 shown in FIG. 5 is formed.

[0012] FIG. 6 schematically shows the operations performed at the heat-sealing station 7 for heat-sealing the U-shaped sealing strip 6 to the packaging web 3.

[0013] As shown in FIG. 6, the packaging web 3 and the sealing strip 6 are first web-fed to the heat-sealing station 7, where they are heated by the first heating unit 8, and then sent to the first pressure roller, which pressure-seals the first longitudinal side portion of the sealing strip 6 to either the inner surface or the outer surface (or decorative surface) of the longitudinal (narrow-width) side portion of the packaging web 3 that remains inside the longitudinal tube 2 after the longitudinal tube 2 is formed, leaving the second (loose) longitudinal portion of the sealing strip 6 that protrudes from the longitudinal end of the packaging web 3.

[0014] The packaging material web 3 and the sealing strip 6 are then advanced to a second heating unit 10, where a second longitudinal portion of the sealing strip 6 that protrudes from a first longitudinal side of the packaging material 3 and is not heat-sealed to the packaging material web 3 is heated together with a surface of the packaging material web 3 opposite the longitudinal side to which the first longitudinal side of the sealing strip 6 is heat-sealed.

[0015] The packaging web 3 and the sealing strip 6 are then advanced to a U-shaped folding unit 11, where the second longitudinal portion of the sealing strip 6 is first U-shaped folded around the longitudinal end of the packaging web 3 and then pressure-sealed to a surface of the packaging web 3 opposite the longitudinal side portion by a second pressure roller (not shown).

[0016] As a result, the first and second longitudinal portions of the sealing strip 6 are heat-sealed to the packaging web 3, but the longitudinal central portion of the sealing strip 6 is not heat-sealed to the packaging web 3. After the longitudinal tube 2 is formed, it unfolds around the longitudinal ends of the packaging web 3 that remain disposed within the longitudinal tube 2 to form loops that are fluidly separated. Summary of the Invention Problems to be Solved by the Invention

[0017] To ensure that the package 5 containing the cast food meets the predetermined sealing requirements, the sealing strip 6 is heat-sealed to the packaging web 3 in accordance with predetermined quality requirements in terms of geometric accuracy and adhesiveness.

[0018] To the best knowledge of the applicant, the quality of the heat-sealing of the sealing strip 6 to the packaging web 3 is currently roughly checked qualitatively and / or quantitatively by a visually trained operator by looking for characteristic low heat-sealing quality marks or patterns and / or mechanically, for example, by tensile testing.

[0019] Although the current heat-seal quality check by a properly trained operator is satisfactory in many respects, the Applicant has found that there is room for improvement, and thus feels the need to develop a technology that can improve this quality check.

[0020] Therefore, an object of the present invention is to develop a technology that enables automatic evaluation of the quality of heat-sealing of a sealing strip to a packaging material, thereby reducing the involvement of human operators.

Means for Solving the Problems

[0021] This object is achieved by the present invention relating to an automatic seal quality management system and a packaging machine equipped with the same, as described in the appended claims.

Brief Description of the Drawings

[0022]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11a

Figure 11b

Figure 11c

Figure 11d

Figure 11e

Figure 11f

Figure 11g

Figure 11h

Figure 11i

Figure 11j

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19A

Figure 19B

Figure 19C

Figure 20A

Figure 20B

Figure 21

Figure 22

Figure 23

Figure 24

Embodiments for Carrying Out the Invention

[0023] Hereinafter, the present invention will be described in detail with reference to the accompanying drawings so that those skilled in the art can implement and use it. Various modifications to the presented embodiments will be immediately apparent to those skilled in the art, and the general principles disclosed herein may be applied to other embodiments and applications without departing from the scope of protection of the present invention defined by the appended claims. Therefore, the present invention should not be limited to the described and shown embodiments, but should be given the broadest scope of protection in accordance with the described and claimed features.

[0024] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly used by those skilled in the art in the field related to the present invention. In case of conflict, this specification including the provided definitions shall prevail. Furthermore, the examples are provided for illustrative purposes only and should not be considered limiting.

[0025] In particular, the block diagrams included in the accompanying figures and described below should not be understood as representing structural features or configurational limitations, but rather as functional features, characteristics, i.e., the essential characteristics of the device, defined by the resulting effects or as functional limitations, and thus can be implemented in different ways to protect their functionality (functional possibility).

[0026] To facilitate the understanding of the embodiments described in this specification, some specific embodiments are referred to and specific terms are used to describe them. The terms used in this specification are for the purpose of describing only specific embodiments and are not intended to limit the scope of the present invention.

[0027] Starting again from FIG. 6, the packaging machine 1 further includes an automatic heat-sealing quality management system 12. This automatic heat-sealing quality management system 12 automatically detects and identifies defects that may have occurred when heat-sealing the sealing strip 6 to the packaging web 3, such as longitudinal seals of the package 5 being leak-tight defective or having the potential to become leak-tight defective, and if there are identified defects, it is designed to evaluate the quality of the heat-sealing of the sealing strip 6 to the packaging web 3 based on the results.

[0028] The automatic heat-sealing quality management system 12 comprises the following configuration: - A sensor system 13 disposed at the heat-sealing station 7, which outputs an output enabling it to automatically detect and identify possible defects that may have occurred when heat-sealing the sealing strip 6 to the packaging web 3, and thus, if there are defects, enables the quality of the heat-sealing of the sealing strip 6 to the longitudinal end of the packaging web 3 to be automatically evaluated based on the identified defects; and - An electronic computing resource 14, which is designed to communicate with the sensor system 13, particularly electrically connected, control its operation, receive and process its output, automatically detect and identify possible defects that may have occurred when heat-sealing the sealing strip 6 to the packaging web 3, and if there are identified defects, evaluate the quality of the heat-sealing of the sealing strip 6 to the packaging web 3 based on the identified defects.

[0029] The sensor system 13 includes an industrial artificial vision system 15, also known as a computer vision system, operable to capture digital images of the packaging web 3 and the sealing strip 6. The artificial vision system 15 is an electronic device that performs machine vision functions and includes one or more digital cameras with an integrated or external image acquisition and processing system, internal and / or external software of the camera, and a lighting system.

[0030] Optionally, the sensor system 13 may include a pyrometer 16 for remotely sensing the temperature of the packaging material 3 and the sealing strip 6.

[0031] The artificial vision system 15 includes either or both of the following configurations: - A first electronic digital image capture device 17 disposed near the first pressure roller 9 and outputting an output capable of automatically detecting and identifying possible defects occurring during the heat seal between the first longitudinal side of the sealing strip 6 and the longitudinal side of the packaging web 3. - A second electronic digital image capture device 18 disposed near the second pressure roller and outputting an output capable of automatically detecting and specifying possible malfunctions occurring during the heat seal of the second longitudinal side of the sealing strip 6 to the longitudinal side of the packaging material 3.

[0032] Each electronic digital image capture device 17, 18 includes a respective pair of digital image sensors in the form of commercially available digital cameras arranged to capture digital images of the packaging material 3 and the sealing strip 6 heat-sealed thereto.

[0033] In particular, each electronic digital image capture device 17, 18 includes the following configuration: - A first digital camera 19 (hereinafter simply referred to as a "visible light camera") designed to operate in the electromagnetic spectrum visible to the human eye, which captures and outputs a digital image (hereinafter simply referred to as a "visible light digital image"). - A second digital camera 20 designed to operate in the electromagnetic spectrum invisible to the human eye, particularly in the infrared spectrum (hereinafter simply referred to as the "infrared camera" (also called a thermograph, thermal imaging camera, IR camera or imager)), which captures and outputs a digital image (hereinafter simply referred to as a "thermal digital image").

[0034] The visible light and infrared cameras 19, 20 of the first electronic digital image capture device 17 are arranged with respect to the packaging material web 3 and the sealing strip 6 heat-sealed thereto such that their fields of view (FOV) are such that the visible light and infrared cameras 19, 20 capture visible light and infrared digital images of the same region of one of the heat-sealed surfaces of the surface of the packaging material 3 and the first longitudinal side of the sealing strip 6 where the first heat seal is made first.

[0035] Similarly, the visible light and infrared cameras 19, 20 of the second electronic digital image capture device 18 are arranged with respect to the packaging material web 3 and the sealing strip 6 heat-sealed thereto such that their fields of view (FOV) are such that they capture visible light and thermal digital images of the same region of one of the opposite surfaces of the packaging material 3 where the second longitudinal side of the sealing strip 6 is heat-sealed next.

[0036] The electronic computing resources 14 may be in the form of a centralized architecture, i.e., a single electronic processing unit to which both of the electronic digital image capture devices 17, 18 can be connected, or a distributed architecture, i.e., two or different communicating and cooperating electronic processing units to which the individual digital cameras of the electronic digital image capture devices 17, 18 can be connected, depending on the hardware and software architecture that the manufacturer deems appropriate for controlling the heat seal quality.

[0037] The electronic computing resource 14 is configured to operate each of the electronic digital image capture devices 17, 18 simultaneously or sequentially at subsequent time instants during the movement of the packaging material web 3 and the sealing strip 6 heat-sealed thereto, whereby each of the electronic digital image capture devices 17, 18 performs a series of digital image multi-capture operations, and during each operation, the visible light and infrared cameras 19, 20 of each of the electronic digital image capture devices 17, 18 operate simultaneously or sequentially to capture a pair of visible light digital images and thermal digital images. The pair of digital images consisting of the visible light digital image and the thermal digital image is a pair of digital images consisting of the visible light digital image and the thermal digital image, the pair of digital images consisting of the visible light digital image and the thermal digital image is a pair of digital images consisting of the visible light digital image and the thermal digital image, the pair of digital images consisting of the visible light digital image and the thermal digital image is a pair of digital images consisting of the visible light digital image and the thermal digital image, and the pair of digital images consisting of the visible light digital image and the thermal digital image is a pair of digital images consisting of the visible light digital image and the thermal digital image.

[0038] Figures 7 through 9 exemplarily show visible light images and thermal digital images taken by the visible light and infrared cameras 19, 20.

[0039] In particular, FIG. 7 illustratively shows a visible light digital image of the inner region of the packaging material 3, which is captured by the visible light camera 19 of the first electronic digital image capture device 17 after the first longitudinal side portion of the sealing strip 6 has been heat-sealed. In the image, the narrow, thin gray vertical stripes 21 represent the entire sealing strip 6, including both the first longitudinal side portion heat-sealed to the longitudinal side portion of the packaging material 3 and the second (loose) longitudinal side portion protruding from the longitudinal end of the packaging material 3. The narrow, dark gray vertical stripe 22 to the left of the thin gray vertical stripes represents the background. The wide, dark gray vertical stripe to the right of the thin gray vertical stripes represents the packaging material 3, which consists of the narrow vertical stripe portion 23 on the left, is slightly darker and more uniform in gray than the wide vertical stripe portion on the right, and represents the region of the longitudinal side surface portion of the packaging material 3 that is adjacent to the sealing strip 6 and was heated during the heat-sealing of the first longitudinal side surface portion of the sealing strip 6 but is not covered by the sealing strip 6 and does not overlap. On the other hand, the wide vertical stripe portion 24 on the right is less dark and less uniform in gray and represents the remaining portion of the packaging material 3 that has not been heated.

[0040] FIG. 8 is similar to that shown in FIG. 7, but illustrates a thermal digital image of a region on the inner surface side of the packaging material 3 captured by the infrared camera 20 of the first electronic digital image capture device 17 after the first longitudinal side portion of the sealing strip 6 has been heat-sealed. In the image, the same reference numerals as in FIG. 7 designate the same portions of the digital image. Similar to the image shown in FIG. 7, the narrow, thin gray vertical stripes represent the entire sealing strip 6 on both the first longitudinal side portion heat-sealed to the longitudinal side portion of the packaging material 3 and the loose longitudinal side portion protruding from the longitudinal end of the packaging material 3. The wide, dark gray vertical stripe to the right of the narrow, thin gray vertical stripe represents the packaging material 3, and the narrow, dark gray vertical stripe to the left of the narrow, thin gray vertical stripe represents the background. Also, in this image, the wide, dark gray vertical stripe on the right consists of a narrow vertical stripe portion on the left that is not as dark as the wide vertical stripe portion on the right, representing the heating pattern, and the wide vertical stripe portion on the right that is slightly darker than the narrow vertical stripe portion on the left represents the remaining portion of the unheated packaging material 3.

[0041] FIG. 9 illustratively shows a visible light digital image similar to that shown in FIG. 7, but captured from the outside or decorative side of the packaging material 3 by the visible light camera 19 of the second electronic digital image capture device 18 after the first longitudinal side portion of the sealing strip 6 has been heat-sealed. In the image, the same reference numerals as in FIGS. 7 and 8 designate the same portions of the digital image. Similar to the images shown in FIGS. 7 and 8, the narrow, thin gray vertical stripes represent the loose longitudinal side portions protruding from the longitudinal ends of the packaging material 3. The narrow, dark gray vertical stripe to the left of the narrow, thin gray vertical stripe represents the background, and the wide, light gray vertical stripe to the right of the narrow, thin gray vertical stripe represents the packaging material 3.

[0042] In one embodiment, the electronic computing resource 14 is further configured to process the captured digital image to automatically detect and identify possible defects that may have occurred when heat-sealing the sealing strip 6 to the packaging material 3 based on the captured digital image.

[0043] In particular, FIG. 10 shows a block diagram of image processing executed by the electronic computing resource 14 for visible light digital images and corresponding thermal digital images of the same region of the inner side of the packaging web 3 and the sealing strip 6, as shown in FIGS. 7 and 8.

[0044] In particular, the electronic computing resource 14 is programmed to receive a set of visible light digital images and thermal digital images, and for each set of visible light digital images and thermal digital images, to perform both image processing of the received visible light digital image and image processing of the set of visible light digital images and thermal digital images.

[0045] With regard to the image processing of the visible light digital image, the electronic computing resource 14 is programmed as follows: - Receive a pair of visible light digital image and thermal digital image (block 25); - Process the received visible light digital image and search for the following: · Marks or patterns (block 26) representing typical rough, major or macro defects or anomalies such as scratches or peeling, as exemplified in FIGS. 11a and 11b, occurring in the heating pattern when the sealing strip 6 is heat-sealed to the packaging web 3, and · Marks or patterns (block 27) representing typical fine, minor or micro defects or anomalies such as specific wrinkles, as exemplified in the visible light digital images shown in FIGS. 11c to 11f, resulting from the heating pattern during heat-sealing of the sealing strip 6 to the packaging web 3.

[0046] Output data representing the identified marks or patterns indicating typical macro and micro defects or anomalies is calculated and input to the inner classifier (block 28). If there are identified marks or patterns indicating typical macro and micro defects or anomalies, the heat-sealing of the sealing strip 6 to the packaging web 3 is evaluated based thereon.

[0047] To search for marks and patterns representing typical micro defects and anomalies, the captured visible light digital image is processed to search for the following: - In the region of the captured visible light digital image corresponding to the heating pattern, diagonal wrinkles (block 29) as shown in the captured visible light digital image shown in FIG. 11c, particularly at the longitudinal ends of the sealing strip 6, particularly at the longitudinal ends of the longitudinal side portions of the sealing strip 6 heat-sealed to the packaging web 3, inclined at approximately 45° with respect to the longitudinal direction. This diagonal wrinkle is caused by overheating of the heating pattern when heat-sealing the sealing strip 6 to the packaging web 3; and - Substantially orthogonal wrinkles (block 30), hereinafter referred to as horizontal wrinkles, in either or both of the region of the captured visible light digital image corresponding to the sealing strip 6 and the region corresponding to the heating pattern, as shown in the digital image shown in FIG. 11e, particularly at the longitudinal ends of the sealing strip 6, particularly at the longitudinal ends of the longitudinal side portions of the sealing strip 6 heat-sealed to the packaging web 3, inclined at approximately 90° with respect to the longitudinal direction. This diagonal wrinkle is due to process defects generated when heat-sealing the sealing strip 6 to the packaging web 3.

[0048] Output data representing the identified diagonal and horizontal wrinkles is calculated and input to the inner classifier (block 28).

[0049] As shown in FIG. 10, the visible light digital image is further processed as follows (block 31): - Save or receive data indicating a width tolerance range indicating the expected heating pattern width; - Calculate the actual heating pattern width measured in a direction orthogonal to the longitudinal end of the sealing strip 6; - Check the actual heating pattern width against the width tolerance range to determine whether the heating pattern width is within or outside the width tolerance range. - Perform a calculation to indicate whether the heating pattern width is within the width tolerance range or outside the width tolerance range. - Input the calculated data indicating whether the heating pattern width is within the width tolerance range or outside the width tolerance range into the inner classifier, and the heat seal of the sealing strip 6 to the packaging web 3 is evaluated based on whether the heating pattern width is within the width tolerance range or outside the width tolerance range.

[0050] In another embodiment, the calculated heating pattern width may be provided to the inner classifier, and the heat seal of the sealing strip 6 to the packaging web 3 is evaluated based on the calculated heating pattern width.

[0051] The visible light digital image is processed as follows (block 32): - Save or receive data indicating an overlap tolerance range indicating the range where the sealing strip 6 overlaps the longitudinal end of the packaging web 3; - Measure in a direction orthogonal to the longitudinal end of the sealing strip 6 and calculate the actual overlap of the sealing strip, which indicates the range where the sealing strip 6 actually overlaps the longitudinal end of the packaging web 3; - Compare the overlapping portion of the actual sealing strip with the overlapping portion tolerance range to determine whether the overlapping portion of the actual sealing strip is within or outside the overlapping portion tolerance range; - Calculate data indicating whether the overlapping portion of the actual sealing strip is within the overlapping portion tolerance range or outside the overlapping portion tolerance range; - Input the calculated data indicating whether the overlapping portion of the actual sealing strip is within the overlapping portion tolerance range or outside the overlapping portion tolerance range into the inner classifier, and when heat-sealing the sealing strip 6 to the packaging web 3, it is evaluated based on whether the calculated overlapping portion of the actual sealing strip is within the overlapping portion tolerance range or outside the overlapping portion tolerance range.

[0052] In another embodiment, the calculated actual overlap of the sealing strip may be provided to the inner classifier, and the heat seal of the sealing strip 6 to the packaging web 3 is evaluated based on the calculated actual overlap of the sealing strip.

[0053] The actual heating pattern width and the overlap of the sealing strip are calculated by determining the number of pixels in the direction orthogonal to the longitudinal ends of the sealing strip 6 in the regions corresponding to the heating pattern and the sealing strip in the processed visible light digital image, and multiplying by a predetermined conversion coefficient that converts the number of pixels into the heating pattern width and the number of pixels of the overlap of the sealing strip expressed in millimeters.

[0054] Advantageously, but not necessarily, the processing of the captured visible light digital image to search for marks or patterns representing typical microscopic defects or anomalies may first be performed by segmenting the visible light digital image into a plurality of segments or regions (sets of pixels, also called image objects) corresponding to the sealing strip 6 (block 33), thereby outputting a segmented visible light digital image consisting only of the regions corresponding to the sealing strip 6 and the heating pattern as shown in FIG. 12, and then processing the segmented visible light digital image to search for the above-mentioned microscopic defects or anomalies.

[0055] In particular, the region corresponding to the sealing strip 6 is identified by detecting a significant variation in pixel luminance in the direction orthogonal to the longitudinal end of the sealing strip 6 compared to adjacent regions, and the region of the heating pattern is identified by evaluating the color uniformity compared to adjacent regions corresponding to the rest of the packaging web 3 with high color dispersion and average gray level.

[0056] Processing of visible light digital images to search for marks or patterns representing typical macro defects or anomalies is advantageously performed based on semi-supervised learning and generative modeling, in particular, based on an artificial neural network architecture known as a variational autoencoder (VAE) as shown in FIG. 13, which is trained on a dataset containing only visible light digital images without macro defects or anomalies. The visible light digital images depicted on the right side of FIG. 13 show a non-anomalous scenario with a small reconstruction error and a high reconstruction probability, i.e., the output of the VAE when no typical macro defects or anomalies are depicted in the processed visible light digital image, and an anomalous scenario with a large reconstruction error and a low reconstruction probability as shown in the upper left of FIG. 13, i.e., the VAE output when typical macro defects or anomalies are depicted in the processed visible light digital image as shown in the lower left of FIG. 13.

[0057] Processing of visible light digital images to search for horizontal wrinkles in the heating pattern is based on a convolutional neural network (CNN) having the architecture of a UNet, exemplarily shown in FIG. 14, and is advantageously learned based on a labeled dataset created and schematically shown in FIG. 15.

[0058] Processing of captured visible light digital images to search for diagonal wrinkles is advantageously based on a convolutional neural network (CNN) architecture trained on a dataset containing digital images with diagonal wrinkles and digital images without diagonal wrinkles, exemplarily shown in FIG. 16.

[0059] Referring again to the block diagram shown in FIG. 10, with respect to the image processing of visible light digital images and thermal digital images, the electronic computing resource 14 is programmed as follows: - Process the received visible light digital image (block 34) and calculate data including information indicating the geometric shape of the heating pattern of the sealing strip 6 and the packaging web 3, hereinafter simply referred to as "geometric information", and in particular, appropriate points of the processed visible light digital image, for example, the longitudinal edges of the sealing strip 6, the longitudinal edges of the packaging web 3, and the positions of their heating patterns in a reference frame with the origin of the coordinate system. The appropriate points are, for example, either the longitudinal edges of the loose longitudinal portion of the sealing strip 6 protruding from the packaging web 3 or the longitudinal edges of the packaging web 3; - Process the received thermal digital image (block 34) and calculate data including information indicating the temperature profile of the packaging web 3 and the sealing strip 6 heat-sealed thereto in a direction orthogonal to the longitudinal ends of the packaging web 3 and the sealing strip 6 heat-sealed thereto in a reference frame having an origin, conveniently in the form of a coordinate system, hereinafter simply referred to as "thermal information", and information indicating the temperature profile of the packaging web 3 and the sealing strip 6 heat-sealed thereto in a reference frame having an origin at either the longitudinal end of the sealing strip 6 not heat-sealed to the packaging web 3 or the longitudinal end of the packaging web 3, hereinafter simply referred to as "thermal information". The appropriate one is, for example, either the longitudinal end of the sealing strip 6 not heat-sealed to the packaging web 3 or the longitudinal end of the packaging web 3; - Fuse (data fusion), i.e., combine or merge (block 34), the output data of the visible light digital image analysis and the output data of the thermal digital image analysis to calculate fusion data representing the combination of the geometric information obtained from the visible light digital image analysis and the thermal information obtained from the thermal digital image analysis, based on which it is possible to identify the underheating (cold seal) region or overheating region that occurred when heat-sealing the sealing strip 6 to the packaging web 3; and - Process the fused data to identify possible underheating or cold seal (block 35) or overheating (block 36) that may have occurred when heat-sealing the sealing strip 6 to the packaging web 3, calculate output data representing the identified underheating or overheating, and input it to the inner classifier (block 28).

[0060] Figure 17 is a graphical display of the fusion of output data from visible light digital image analysis and output data from thermal digital image analysis.

[0061] As can be understood, the thermal information from the thermal digital image analysis in the form of a temperature profile in a direction orthogonal to the longitudinal edges of the packaging web 3 and the sealing strip 6 heat-sealed thereto is overlaid on the graphic representation of the geometric information obtained from the visible light digital image analysis as the geometric shape of the sealing strip 6 and the shape of the heating pattern of the packaging web 3, whereby the underheating (cold seal) or overheating regions that occurred when heat-sealing the sealing strip 6 to the packaging web 3 can be identified.

[0062] Next, in the inner classifier (block 28), the quality of the heat-sealing of the sealing strip 6 to the inner surface of the packaging web 3 is evaluated based on a unique criterion developed to take into account the heating pattern and the macro and micro defects (if any) identified in the sealing strip 6, the calculated heating pattern width and strip overlap, and the identified underheating (cold seal) or overheating (if any), as well as the severity of the above-mentioned indicated amounts.

[0063] Regarding the calculation of the temperature profiles of the packaging web 3 and the sealing strip 6, referring illustratively to the thermal digital image shown in FIG. 8, the temperature profiles of the packaging web 3 and the sealing strip 6 are first determined by identifying the left longitudinal boundary of the sealing strip 6 in the thermal digital image, and then using a custom algorithm to progress from left to right in a direction orthogonal to the sealing strip 6. The temperature values are calculated based on the brightness values of the primary colors of the pixels of the thermal digital image from the identified left vertical boundary of the sealing strip 6 to the right vertical boundary of the sealing strip 6 and across the entire heating pattern.

[0064] In order for the calculated geometric information and thermal information to be correctly fused, the calculated geometric information and thermal information need to be aligned first in the spatial domain, particularly in both the direction parallel to and the direction orthogonal to the sealing strip 6.

[0065] When the calculated geometric information and thermal information are aligned, the visible light digital image and the thermal digital image can be conceptually superimposed, that is, the packaging web 3 and the sealing strip 6 depicted in the visible light digital image can be conceptually superimposed with the packaging web 3 and the sealing strip 6 depicted in the thermal digital image.

[0066] FIG. 18 illustratively shows a composite visible light and thermal digital image obtained by superimposing a visible light digital image and a corresponding thermal digital image.

[0067] For this purpose, the reference frames of the visible light digital image and the thermal digital image, to which the geometric information and the thermal information are referred, need to be spatially / geometrically related to each other via a so-called spatial transformation matrix that represents the mutual rotational transformation of the two reference frames so that the sealing strip 6 and the packaging web 3 in the visible light digital image can be related to the sealing strip 6 and the packaging web 3 in the thermal digital image.

[0068] The calibration of the visible light and infrared cameras 19, 20 of each of the electronic digital image capture devices 17, 18 is performed to calculate a transformation matrix that is used for the alignment of each set of visible light digital images and thermal digital images during the execution of heat seal quality control.

[0069] The calibration of the visible light and infrared cameras is performed using a calibration marker 21 that comprises or is designed to show a calibration pattern 22. The calibration marker 21 is stably arranged within the fields of view (FOV) of the visible light and infrared cameras 19, 20 of each of the electronic digital image capture devices 17, 18. When the visible light and infrared cameras 19, 20 are operated simultaneously or sequentially, the calibration pattern 22 becomes imaged, i.e., visualized, in both the visible light digital image and the thermal digital image captured by the visible light and infrared cameras 19, 20 respectively. Thereby, a spatial transformation matrix can be calculated based on the position and orientation of the patterns imaged in the visible digital image and the thermal digital image.

[0070] In the embodiment shown in FIGS. 19A - 19C, the calibration marker 21 comprises a foreground member 23 that is arranged in front of the visible light / infrared cameras 19, 20 and carries the calibration pattern 22 in the form of a through pattern, and a background member 24 that is arranged behind the foreground member 23 with respect to the visible light / infrared cameras 19, 20 and is exposed, i.e., visible, through the through pattern 22 to the foreground.

[0071] In the embodiment shown in FIGS. 19A - 19C, the background member 24 is in the form of a foil, for example, an aluminum foil. The foreground member 23 comprises a plate 25 in which the calibration pattern 22 is formed, and a support and coupling structure 26 that is integral with the plate 25 and is designed to easily and removably couple / hold the background member 24 behind the plate 25 and is designed such that the plate 25 with the background member 24 coupled thereto can be stably arranged within the fields of view of the visible and infrared cameras 19, 20.

[0072] In the embodiment shown in FIGS. 19A to 19C, the calibration pattern 22 is in the form of a grid of through holes in offset columns, i.e., a grid of through holes where the through holes in a column are offset with respect to the through holes in an adjacent column.

[0073] The foreground member 23 and the background member 24 are made of materials having different emissivities. In particular, the foreground member 23 is made of a material having an emissivity lower than that of the material of the background member 24, for example, aluminum (higher) emissivity, for example, PLA plastic. Thus, when the background member 24 is properly heated, the heated area of the background member 24 exposed through the through pattern 22 is imaged in the thermal digital image to form a corresponding calibration pattern, while the remaining portion of the background member 24 is shielded by the foreground member 23 and thus not imaged in the thermal digital image.

[0074] In this way, the calibration pattern 22 formed on the foreground member 23 can be imaged in the visible light digital image, and after the background member 24 is properly heated, the corresponding pattern formed by the heated area of the background member 24 exposed through the calibration pattern 22 formed on the foreground member 23 can also be imaged in the thermal digital image.

[0075] FIG. 20A shows the calibration pattern 22 formed on the foreground member 23 and imaged in the visible light digital image, and FIG. 20B shows the corresponding pattern formed by the heated area of the background member 24 exposed through the calibration pattern 22 formed on the foreground member 23 and imaged in the thermal digital image.

[0076] Therefore, calibration of the output data of the visible and infrared cameras 19, 20 of each electronic digital image capture device 17, 18 is required as follows: - Heating the background member 24 of the calibration marker 21; - Placing the calibration marker 21 within the fields of view of the visible light camera 19 and the infrared camera 20; - Operating the visible light camera 19 and the infrared camera 20 to capture the visible light digital image and the thermal digital image; - Process the captured visible light digital image and thermal digital image to identify the calibration pattern contained therein; - Calculate the positions and orientations of the calibration patterns displayed in the visible light image and the digital image; - Calculate a spatial transformation matrix based on the positions and orientations of the calibration patterns in the visible light image and the digital image.

[0077] For this purpose, the electronic computing resource 14 is further programmed as follows: - Actuate the visible light camera 19 and the infrared camera 20 simultaneously or sequentially to capture the visible light digital image and the thermal digital image; - Process the captured visible light digital image and thermal digital image to identify the calibration pattern contained therein and calculate its position and orientation. - Calculate a spatial transformation matrix based on the positions and orientations of the calibration patterns included in the visible light image and the digital image.

[0078] FIG. 21 is a block diagram of image processing executed by the electronic computing resource 14, capturing visible light digital images and thermal digital images of the outer or decorative surface of the packaging web 3 and the same region of the sealing strip 6, and the visible light digital image is the one shown in FIG. 9.

[0079] For the sake of convenience, in the block diagram shown in FIG. 21, the blocks related to the processes similar or corresponding to the processes described above with reference to the block diagram shown in FIG. 10 are labeled with the same reference numerals as the block diagram shown in FIG. 10.

[0080] As shown in FIG. 21, it is determined whether the visible light digital image is processed and the sealing strip 6 is U-folded around the longitudinal end of the packaging web 3 (block 37), and whether one or more marks similar to the "fishbone" as shown in FIG. 21 are formed between the U-folded and heat-sealed sealing strip 6 on the packaging web 3 (block 38).

[0081] When it is determined that the sealing strip 6 is U-shaped folded, as described above with reference to block 32 shown in FIG. 10, the actual overlap of the sealing strip measured in a direction orthogonal to the longitudinal ends of the sealing strip 6 is calculated (block 39).

[0082] Instead, a region of the thermal digital image having predefined characteristics, particularly a region having a predefined luminance pattern such as the region surrounded by the red rectangle shown in the thermal digital image reproduced in FIG. 21, is identified (block 41), and the identified region of the thermal digital image is processed to search for insufficient heating (cold seal) that may have occurred when heat-sealing the sealing strip 6 to the packaging web 3, as previously described with reference to block 35 shown in FIG. 10 (block 42).

[0083] Next, the quality of the heat-sealing of the sealing strip 6 to the packaging web 3 is evaluated in an outer classifier (block 40) similar to the inner classifier shown in FIG. 10 with reference to block 28, based on the aforementioned features and its own criteria considering the significance of the aforementioned features.

[0084] To detect the presence and number of "fishbones" in the visible light digital image, a deep learning algorithm based on the UNet network exemplified in FIG. 22 is used and trained based on the dataset created and labeled as schematically shown in FIG. 23.

[0085] Finally, FIG. 24 shows a block diagram of the image processing executed by the electronic computing resource 14 for the captured visible light digital image and thermal digital image of the longitudinal seal region of the longitudinal tube 2 after the packaging web 3 is folded and its longitudinal ends are heat-sealed.

[0086] As shown in FIG. 24, the visible light digital image is processed (block 41) to identify regions of the thermal digital image having predefined characteristics, particularly regions having a predefined luminance pattern, such as the region enclosed by the red rectangle shown in the visible light digital image reproduced in FIG. 21. The identified regions of the visible light digital image are then processed to determine whether the vertical tube 2 is twisted in the vertical seal (block 42) and to determine whether the longitudinal ends of the packaging web 3 are incorrectly overlapping (block 43).

[0087] Output data representing whether the vertical tube 2 is twisted in the vertical seal or whether the longitudinal ends of the packaging web 3 are incorrectly overlapping is calculated and input to a vertical seal classifier (block 44) similar to the inner and outer classifiers shown in FIGS. 10 and 21.

[0088] Instead, the thermal digital image is processed to search for marks indicating insufficient heating (cold seal) that occurred when the sealing strip 6 was heat-sealed to the packaging web 3, as previously described with reference to block 35 shown in FIG. 10.

[0089] Output data representing the identified marks indicating insufficient heating (cold seal) is calculated and input to the vertical seal classifier (block 44), and the quality of the vertical seal of the vertical tube 2 is evaluated based on a unique criterion considering the severity of the identified features.

[0090] The advantages achieved by the present invention will be readily understood by those skilled in the art.

[0091] In particular, the present invention enables the quality of the heat seal of the sealing strip to the packaging web to be automatically checked by a simple, robust and reliable optical image technique, which is significantly improved compared to the heat seal quality check performed by a properly trained operator.

Claims

1. An automatic heat seal quality control system (12) in the heat seal station (7) of a packaging machine (1) automatically controls the quality of heat sealing of a sealing strip (6) to the longitudinal edge of a packaging web (3), wherein sealed packages (5) containing cast food are continuously manufactured from a continuous vertical tube (2) filled with cast food, and the packaging web (3) is formed by folding it longitudinally, overlapping its longitudinal edges, and heat sealing the overlapped longitudinal edges. The system (12) is A sensor system (13) is located in the heat sealing station (7) and outputs an output that can evaluate the quality of the heat seal of the sealing strip (6) to the longitudinal end of the packaging web (3), An electronic computing resource (14) communicates with the sensor system (13), receives and processes the output, and automatically evaluates the quality of the heat seal of the sealing strip (6) to the longitudinal end of the packaging web (3), Equipped with, The sensor system (13) includes an artificial vision system (15) for capturing digital images of the packaging web (3) and the sealing strip (6). The artificial vision system (15) is A visible light camera (19) designed to operate in the electromagnetic spectrum visible to the human eye in order to capture and output a visible light digital image, An infrared camera (20) designed to operate in the electromagnetic spectrum invisible to the human eye, and to capture and output thermal digital images, Equipped with, System (12).

2. The visible light camera (19) and the infrared camera (20) are positioned relative to the packaging web (3) and the sealing strip (6) to capture visible light and thermal digital images of the same area on the same surface of the packaging web (3) and the sealing strip (6). The system (12) according to claim 1.

3. The aforementioned electronic computing resource (14) is The visible light / infrared cameras (19, 20) are activated to capture a pair of visible light / thermal digital images of the packaging web (3) and the sealing strip (6). The pair of the captured visible light digital image and infrared digital image is received. The received visible light digital image is subjected to visible light digital image analysis (34), and data including geometric information representing the geometric shape of the sealing strip (6) and the heating pattern of the packaging web (3) is calculated and output in a reference frame with an appropriate point in the processed visible light digital image as the origin. The received thermal digital image is subjected to thermal digital image analysis (34), and data including thermal information representing the temperature profile of the packaging web (3) and the sealing strip (6) in a direction perpendicular to the packaging web (3) and the sealing strip (6) is calculated and output in a reference frame having the origin at an appropriate position in the processed thermal digital image. The geometric information obtained from the visible light digital image analysis and the thermal information obtained from the thermal digital image analysis are fused, and fused data representing the fused information is calculated. The fused data is processed to identify any possible underheating, cold sealing (35), or overheating (36) that may have occurred during the heat sealing of the sealing strip (6) to the packaging web (3). Based on the identified underheating or overheating, the quality of the heat seal of the sealing strip (6) to the packaging web (3) is evaluated. The system (12) according to claim 1.

4. The aforementioned electronic computing resource (14) is The visible light imaging camera (19) is activated to capture a visible light digital image of the packaging web (3) and the sealing strip (6). The captured visible light digital image is received, The received visible light digital image is processed, A mark or pattern (26) representing a macro defect or abnormality such as scratches or delamination that may be formed during the heat sealing of the sealing strip (6) to the packaging web (3), Marks or patterns (27) representing minor defects or abnormalities such as wrinkles that may be formed during the heat sealing of the sealing strip (6) to the packaging web (3). Search, The quality of the heat seal of the sealing strip (6) to the packaging web (3) is evaluated based on the identified mark or pattern. The system (12) according to claim 1.

5. The electronic computing resource (14) further processes the received visible light digital image to search for marks or patterns representing macro and micro defects or anomalies in the heating pattern of the packaging web (3), which is defined as an area of ​​the packaging web (3) adjacent to the sealing strip (6) and heated during the heat sealing of the sealing strip (6) to the packaging web (3), but not covered by the sealing strip (6). The system (12) according to claim 4.

6. The electronic computing resource (14) processes the received visible light digital image in order to search for marks or patterns representing the microscopic defects or anomalies. The diagonal wrinkles (29) in the heating pattern of the packaging web (3), Transverse wrinkles (30) are provided in the heating patterns of the sealing strip (6) and the packaging web (3), Search The system (12) according to claim 5.

7. The electronic computing resource (14) processes the received visible light digital image in (31), The actual heating pattern width is calculated, measured in a direction perpendicular to the sealing strip (6), of the heating pattern representing the area of ​​the packaging web (3) adjacent to the sealing strip (6) that was heated during the heat sealing of the sealing strip (6) to the packaging web (3) but not covered by the sealing strip (6). Based on the calculated heating pattern width, the quality of the heat seal of the sealing strip (6) to the packaging web (3) is evaluated. The system (12) according to claim 4.

8. The electronic computing resource (14) processes the received visible light digital image in (32), The actual overlap of the sealing strip is calculated, which is measured in a direction perpendicular to the sealing strip (6) and indicates the extent to which the sealing strip (6) actually overlaps the packaging web (3). The quality of the heat seal of the sealing strip (6) to the packaging web (3) is evaluated based on the calculated actual overlap of the sealing strip. The system (12) according to claim 4.

9. The electronic computing resource (14) processes the received visible light digital image in (37), The sealing strip (6) is U-folded around the longitudinal end of the packaging web (3), and while the sealing strip (6) is U-folded and heat-sealed to the packaging web (3), it is determined whether a mark resembling a fishbone is formed on the sealing strip (6). If it is determined that the sealing strip (6) is folded in a U-shape, the actual overlap portion of the sealing strip, which indicates the area where the sealing strip (6) actually overlaps the packaging web (3), is calculated, and the sealing strip (6) is measured in a direction perpendicular to the sealing strip (6). The quality of the heat seal of the sealing strip (6) to the packaging web (3) is evaluated based on whether the sealing strip (6) is U-folded around the longitudinal end of the packaging web (3), whether a fishbone-like mark is formed on the sealing strip (6) during the U-folding and heat sealing to the packaging web (3), and the calculated actual overlap of the sealing strip. The system (12) according to claim 4.

10. The electronic computing resource (14) further processes the received thermal digital image (41), From the aforementioned thermal digital image, regions having predefined characteristics, particularly a predefined brightness pattern, are identified. To identify the thermal signature of the heat seal to the longitudinal end of the sealing strip (6), in particular, to identify any underheating or cold seal that may have occurred during the heat seal of the sealing strip (6) to the packaging web (3), the identified region of the thermal digital image is processed. Based on the identified underheating or cold seal, the quality of the heat seal of the sealing strip (6) to the packaging web (3) is evaluated. The system (12) according to claim 9.

11. The electronic computing resource (14) aligns the geometric information with the thermal information in a spatial domain, both in a direction parallel to and perpendicular to the sealing strip (6), such that the calculated thermal information relates to the same spatial point on the packaging web (3) and the sealing strip (6) to which the calculated visible light information relates. In order to reconcile the geometric information with the thermal information in the spatial domain, the electronic computing resource (14) calculates a spatial transformation matrix that correlates the reference frames in the visible light digital image and the thermal digital image that reference the geometric information and the thermal information, thereby associating the images of the sealing strip (6) and the packaging web (3) in the visible light digital image with the images of the sealing strip (6) and the packaging web (3) captured in the thermal digital image. To calculate the spatial transformation matrix, the system (12) includes a calibration marker (21) that shows a calibration pattern (22) that can be imaged in both the visible light digital image and the thermal digital image. The electronic computing resource (14) calculates the spatial transformation matrix based on the position and orientation of the calibration pattern (22) captured in both the visible light digital image and the thermal digital image. The system (12) according to claim 1.

12. The calibration marker (21) comprises a foreground member (23) positioned in front of the visible light / infrared cameras (19, 20) and intended to carry a calibration pattern (22) in the form of a through-pattern, and a background member (24) positioned behind the foreground member (23) in front of the visible light / infrared cameras (19, 20) so as to be exposed to the foreground through the calibration pattern (22), The foreground member (23) and the background member (24) are formed of materials having different emissivity, and a calibration pattern (22) formed on the foreground member (23) is captured in the visible light digital image, and when the background member (24) is heated, the heated area of ​​the background member (24) exposed to the foreground through the calibration pattern (22) formed on the foreground member (23) that is captured in the visible light digital image is captured in the thermal digital image and can form a corresponding calibration pattern, while the rest of the background member (24) is shielded by the foreground member (23) and is therefore not captured in the thermal digital image. The system (12) according to claim 10.

13. A packaging machine (1) capable of continuously producing sealed packages (5) containing cast food from a continuous vertical tube (2) formed by filling a cast food, folding a packaging web (3) in the longitudinal direction and overlapping its longitudinal edges, and then heat-sealing the overlapped longitudinal edges, The packaging machine (1) is equipped with the automatic heat seal quality control system (12) described in claim 1. Packaging machine (1).

Images