System for evaluating sealing quality of sealing strip to packaging web in packaging machine

Abstract

An automatic heat seal quality control system (12) for automatically controlling the heat seal quality of a sealing strip (6) to a longitudinal edge of a packaging web (3) at a heat seal station (7) in a packaging machine (1) wherein sealed packages (5) containing pourable food products are continuously produced from a continuous vertical tube (2) filled with pourable food products, and the sealed package is formed by longitudinally folding the packaging web (3) to overlap its longitudinal edges and then heat-sealing the overlapped longitudinal edges. The system (12) comprises a visible light imaging camera (19) designed to operate in an electromagnetic spectrum visible to the human eye to capture and output visible light digital images.

Description

Technical Field

[0001] The present invention generally relates to packaging pourable food products by sequentially transversely sealing a web-fed tube of packaging material continuously filled with the pourable food product.

[0002] More particularly, the present invention relates to evaluating the seal quality of a sealing bar heat-sealed to a packaging web in a packaging machine for producing sealed packages containing pourable food products.

[0003] The invention also relates to calibrating a camera for evaluating the sealing quality of a sealing strip. Background Art

[0004] As is known, many pourable foods, such as fruit or vegetable juices, pasteurized or UHT (ultra-high temperature treated) milk, wine etc., are sold in packages made of sterilized packaging material.

[0005] A typical example of this type of packaging is the parallelepiped-shaped package for pourable foods called Tetra Brik It is made by folding and sealing laminated strip packaging material.

[0006] The packaging material has a multi-layer sheet structure, which mainly includes one or more hardened and reinforced base layers, which are usually made of fibrous materials (such as paper or paperboard) or mineral-filled polypropylene materials, and are covered on both sides with one or more heat-sealable plastic material layers (such as polyethylene film). For aseptic packaging for long-term storage products (such as UHT milk), the packaging material also includes a gas barrier and light barrier material layer (such as aluminum foil or ethyl vinyl alcohol (EVOH) film), which is superimposed on the heat-sealable plastic material layer and in turn covered with another heat-sealable plastic material layer to form the inner surface of the package that finally contacts the food.

[0007] This type of packaging is usually Figure 1 The packaging material is produced on a fully automatic packaging machine (also called a filling machine) of the type shown, indicated as a whole by the reference numeral 1, in which a continuous vertical tube 2 is formed from a packaging web 3 which is sterilized by applying, for example, a chemical sterilizing agent (e.g. a hydrogen peroxide solution) or a physical sterilization (e.g. by an electron beam). Once the sterilization is complete, any residues (in particular those left by the chemical sterilizing agent) are removed from the surface of the packaging material (e.g. by evaporation by heating). The packaging web 3 is kept in a closed sterile environment, in particular in an isolator, and is folded and then longitudinally heat-sealed to form the vertical tube 2.

[0008] The sterile or sterile-treated pourable food is filled into the vertical tube 2 through a filling tube extending inside the vertical tube 2. The vertical tube 2 advances in a vertical advancement direction to a forming station, where the vertical tube 2 is clamped along equidistant transverse sections by a jaw system comprising two or more pairs of jaws, which cyclically and continuously act on the vertical tube 2 to form a continuous sequence of pillow-shaped packages 4 interconnected by transverse sealing strips. The pillow-shaped packages 4 are then separated from each other by cutting the sealing strips and conveyed to a folding station, where they are mechanically folded into a finished product (e.g., a substantially parallelepiped-shaped package 5).

[0009] In order to prevent the longitudinal edges or borders of the packaging web 3, in particular the fibrous material in the multilayer sheet structure of the packaging web 3 exposed to the external environment, from first contacting and absorbing the pourable food in the vertical tube 2 and then contacting and absorbing the pourable food in the pillow-shaped package 4, the longitudinal edges of the packaging web 3 are fluidly isolated by heat sealing a sealing strip 6 of a web feed of heat-seal plastic material to the packaging web 3 at a heat sealing station 7 before the packaging web 3 is folded and longitudinally sealed to form the vertical tube 2.

[0010] exist Figure 2 In the embodiment shown, the sealing strip 6 is flat, and its first longitudinal side portion is first heat-sealed to the inner surface of the longitudinal side portion of the packaging web 3, then folded so that its opposite longitudinal edges are in contact, and then the remaining (loose) longitudinal portion of the sealing strip 6 that extends from the longitudinal edge of the packaging web 3 and is not heat-sealed thereto is heat-sealed to the inner surface of the opposite longitudinal side portion of the packaging web 3. In this embodiment, the heat sealing of the sealing strip 6 to the opposite longitudinal side portion of the packaging web 3 causes the packaging web 3 to also be heat-sealed longitudinally to form a vertical tube 2. Then a vertical tube 2 is formed. Figure 3 The package 5 shown in , as described above.

[0011] exist Figure 4 In the different embodiments shown, the sealing strip 6 is U-shaped or C-shaped and forms Figure 5 Packaging 5 shown.

[0012] Figure 6 The operation of heat sealing the U-shaped sealing strip 6 to the packaging web 3 performed at the heat sealing station 7 is schematically shown.

[0013] like Figure 6As shown, the packaging web 3 and the sealing strip 6 are first fed by the web to a heat sealing station 7, where they are heated by a first heating unit 8, and then advance to a first pressure roller 9, which pressure roller pressure-seals a first longitudinal side portion of the sealing strip 6 to the inner or outer surface (or decorative surface) of the longitudinal (narrow) side portion of the packaging web 3. After the vertical tube 2 is formed, the side portion remains arranged in the vertical tube 2, while the remaining second (loose) longitudinal portion of the sealing strip 6 extends from the longitudinal edge of the packaging web 3.

[0014] The packaging roll 3 and the sealing strip 6 are then advanced to a second heating unit 10, where a second longitudinal portion of the sealing strip 6 extending from the first longitudinal side of the packaging material 3 and not heat-sealed to the packaging roll 3 and an opposite side of the longitudinal side of the packaging roll 3 that has been heat-sealed to the first longitudinal side of the sealing strip 6 are heated.

[0015] The packaging web 3 and the sealing strip 6 then advance to the U-folding unit 11, where the second longitudinal portion of the sealing strip 6 is first U-folded around the longitudinal edge of the packaging web 3 and then pressure-sealed to the opposite surface of the longitudinal side of the packaging web 3 by a second pressure roller (not shown).

[0016] Therefore, although the first and second longitudinal portions of the sealing strip 6 are heat-sealed with the packaging web 3, the longitudinal center portion of the sealing strip 6 is not heat-sealed with the packaging web 3 and forms a loop, which is formed around the longitudinal edges of the packaging web 3 and fluid-isolates the longitudinal edges of the packaging web 3, and after the vertical tube 2 is formed, the loop is still arranged in the vertical tube 2. Summary of the invention

[0017] In order to ensure that the package 5 filled with the pourable food meets the specified sealing requirements, the sealing strip 6 is heat-sealed to the packaging coil 3 according to the specified geometric accuracy and adhesive quality requirements.

[0018] To the best of the applicant's knowledge, currently, the heat seal quality of the sealing strip 6 to the packaging web 3 is roughly checked qualitatively and / or quantitatively by visual inspection looking for characteristic low heat seal quality marks or patterns and / or by mechanical tests (e.g. tensile tests) performed by appropriately trained operators.

[0019] The applicant has found that, although satisfactory in many respects, the current quality inspection of heat seals by suitably trained operators leaves room for improvement and the applicant has therefore seen a need to develop a technique which could improve such quality inspection.

[0020] The object of the present invention is therefore to develop a technique which allows automatic assessment of the quality of the heat seal of a sealing strip to a packaging material, thereby reducing the involvement of human operators.

[0021] This object is achieved by the present invention, which relates to an automatic sealing quality control system and a packaging machine comprising the same, as claimed in the accompanying claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 There is shown a perspective view of a packaging machine, with some parts removed for clarity, which can be used to produce sealed packages containing pourable food products from a tube of packaging material.

[0023] Figure 2 and Figure 3 The heat sealing of an adhesive tape sealing strip to a packaging web and the resulting sealed package are respectively schematically shown.

[0024] Figure 4 and Figure 5 The heat sealing of a U-shaped sealing strip to a packaging web and the resulting sealed package are respectively schematically shown.

[0025] Figure 6 A heat sealing station is schematically shown, where a sealing bar is heat sealed to a packaging web and the quality of the heat seal is automatically checked by an automatic heat seal quality control system.

[0026] Figures 7 to 9 Schematic illustration of visible light and thermal digital images captured by visible light and thermal imaging cameras of an automated heat seal quality control system.

[0027] Fig.10 A visible light digital image and a corresponding thermal digital image (eg Figure 7 and Figure 8 Block diagram for image processing (the image shown in ).

[0028] Fig.11 a-11f show visible light digital images in which the marks or patterns represent typical macro and micro defects or anomalies produced on the packaging web or sealing strip during the heat sealing process of the sealing strip and the packaging web.

[0029] Fig.12 The segmentation of a visible light digital image is schematically shown.

[0030] Fig.13 Schematically illustrated is a Variation Auto-Encoder (VAE) artificial neural network structure for identifying macroscopic defects or anomalies originating from a packaging web during heat sealing of a sealing strip and the packaging web.

[0031] Fig.14 Schematically illustrated is a convolutional neural network (CNN) with UNet architecture for identifying microscopic defects or anomalies originating from a packaging web during heat sealing of a sealing strip and the packaging web.

[0032] Fig.15 Schematic diagram showing the creation and labeling of Fig.14 The process of the UNet dataset is shown.

[0033] Fig.16 A CNN architecture is shown for identifying microscopic defects or anomalies originating from the packaging web and / or the sealing strip during heat sealing of the sealing strip and the packaging web.

[0034] Fig.17 Shown is a graphical representation obtained by fusing the output data of visible light and thermal digital image analysis of a pair of simultaneously captured visible light and thermal digital images.

[0035] Fig.18 The alignment and overlay of a pair of visible light and thermal digital images in the space domain is schematically shown.

[0036] 19a, 19b and 19c show calibration markers used to calibrate visible light and thermal imaging cameras.

[0037] Figures 20a and 20b show the calibration patterns presented by the calibration marks shown in Figures 19a-19c and imaged in visible light digital images and thermal digital images.

[0038] Fig.21 A visible light digital image and a corresponding thermal digital image (eg Figure 7 and Figure 8 Block diagram for image processing (the image shown in ).

[0039] Fig. 22 Schematic illustration of a convolutional neural network (CNN) with UNet architecture used to identify fishbones in a sealing strip heat sealed to a packaging web.

[0040] Fig.23 Schematic diagram showing the creation and labeling of Fig. 22 The process of the UNet dataset is shown.

[0041] Fig.24 A visible light digital image and a corresponding thermal digital image (eg, Figure 7 and Figure 8Block diagram for image processing (the image shown in ). DETAILED DESCRIPTION

[0042] The invention will now be described in detail with reference to the accompanying drawings so that a person skilled in the art can make and use it. It will be immediately clear to a person skilled in the art that various modifications may be made to the embodiments presented, and that the general principles disclosed herein may be applied to other embodiments and applications without thereby departing from the scope of protection of the invention as defined in the appended claims. Therefore, the invention should not be considered limited to the embodiments described and shown, but should be given the broadest scope of protection according to the features described and claimed.

[0043] Unless otherwise defined, all technical and scientific terms used herein have the same meaning commonly used by one of ordinary skill in the art to which the invention belongs. In the event of a conflict, the present specification, including the definitions provided, shall prevail. In addition, the examples provided are for illustrative purposes only and therefore should not be considered as limiting.

[0044] In particular, the block diagrams contained in the drawings and described below should not be understood as representations of structural features or construction limitations, but rather as representations of functional properties, attributes, i.e., intrinsic properties of the device and defined by the effects obtained or functional limitations that can be implemented in different ways, thereby protecting its functionality (functional possibilities).

[0045] To facilitate understanding of the embodiments described herein, reference will be made to certain specific embodiments and specific language will be used to describe them. The terminology used herein is for describing particular embodiments only and is not intended to limit the scope of the invention.

[0046] Again from Figure 6 Initially, the packaging machine 1 is also equipped with an automatic heat seal quality control system 12, which is designed to automatically detect and identify (distinguish / classify) any faults that may occur during the heat sealing of the sealing strip 6 to the packaging web 3, which may cause the longitudinal seal of the package 5 to be or become fluid-tight defective, and evaluate the heat seal quality of the sealing strip 6 to the packaging web 3 based on the results of the identified faults (if any).

[0047] The automatic heat seal quality control system 12 includes: a sensor system 13, arranged at the heat sealing station 7, for delivering an output allowing the automatic detection and identification of any faults that may have occurred during the heat sealing of the sealing strip 6 to the packaging web 3, thereby allowing the automatic evaluation of the quality of the heat sealing of the sealing strip 6 to the longitudinal edge of the packaging web 3 based on the identified faults, if any; and - an electronic computing resource 14 for communicating with the sensing system 13 (in particular being electrically connected to the sensing system 13) in order to control its operation and to receive and process its output in order to automatically detect and identify any faults that may occur during the heat sealing of the sealing strip 6 to the packaging web 3 and to evaluate the quality of the heat sealing of the sealing strip 6 and the packaging web 3 based on the identified faults (if any).

[0048] The sensing system 13 includes an industrial artificial vision system 15, also known as a computer vision system, operable to capture digital images of the packaging web 3 and the sealing strip 6. The artificial vision system 15 is an electronic device that performs machine vision functions and has one or more digital cameras equipped with an integrated or external image acquisition and processing system, camera internal and / or external software, and a lighting system.

[0049] The sensing system 13 may optionally include a pyrometer 16 to remotely sense the temperature of the packaging material 3 and the sealing strip 6 .

[0050] The artificial vision system 15 includes one or both of the following: a first electronic digital image capture device 17 arranged near the first pressure roller 9 to output an output allowing automatic detection and identification of any faults that may occur during heat sealing of the first longitudinal side of the sealing strip 6 to the longitudinal side of the packaging web 3; and a second electronic digital image capture device 18 , arranged near the second pressure roller, to deliver an output allowing automatic detection and identification of any faults that may have occurred during heat sealing of the second longitudinal side of the sealing strip 6 to the longitudinal sides of the packaging web 3 .

[0051] Each electronic digital image capture device 17, 18 comprises a pair of respective digital image sensors in the form of commercially available digital cameras arranged to capture digital images of the packaging material 3 and the sealing strip 6 heat sealed thereto.

[0052] Specifically, each electronic digital image capture device 17, 18 comprises: a first digital camera 19 (hereinafter referred to as a visible light imaging camera), which is designed to operate in the electromagnetic spectrum visible to the human eye, for capturing and outputting digital images (hereinafter referred to as visible light digital images); and - A second digital camera 20 (hereinafter referred to as a thermal imaging camera (also known as a thermal imaging, thermal imaging or infrared camera or imager)), which is designed to operate in the electromagnetic spectrum invisible to the human eye, in particular in the infrared spectrum, to capture and output digital images (hereinafter referred to as thermal digital images).

[0053] The visible light and thermal imaging cameras 19, 20 of the first electronic digital image capture device 17 are arranged relative to the packaging web 3 and the sealing strip 6 heat-sealed thereon so that their field of view (FOV) enables the visible light and thermal imaging cameras 19, 20 to capture visible light and thermal digital images of the same area of ​​the surface of the packaging material 3 and the sealing strip 6 heat-sealed thereon, wherein the first longitudinal side of the sealing strip 6 is heat-sealed first.

[0054] Similarly, the visible light and thermal imaging cameras 19, 20 of the second electronic digital image capture device 18 are arranged relative to the packaging web 3 and the sealing strip 6 heat-sealed thereon so that their field of view (FOV) enables the visible light and thermal imaging cameras 19, 20 to capture visible light and thermal digital images of the same area of ​​the opposite side of the packaging material 3, wherein the second longitudinal side of the sealing strip 6 is subsequently heat-sealed in this area.

[0055] The electronic computing resource 14 can have a centralized architecture (i.e., it can be in the form of a single electronic processing unit, to which both electronic digital image capture devices 17 and 18 can be connected), or it can have a distributed architecture (i.e., it can be in the form of two or different communicating and cooperating electronic processing units, to which each digital camera of the electronic digital image capture device 17 and 18 can be connected, depending on the hardware and software architecture that the manufacturer considers suitable for controlling the quality of the heat seal.

[0056] The electronic computing resource 14 is configured to operate each electronic digital image capture device 17, 18 simultaneously or successively at subsequent moments during the movement of the packaging roll 3 and the sealing strip 6 heat-sealed thereon, whereby each electronic digital image capture device 17, 18 performs a series of digital image multiple capture operations, during each operation, the visible light and thermal imaging camera 19, 20 of each electronic digital image capture device 17, 18 operates simultaneously or continuously to capture a pair of visible light and thermal digital images, that is, a pair of digital images of the same area of ​​the packaging material 3 and the sealing strip 6 heat-sealed thereon within the field of view of the visible light and thermal imaging cameras 19, 20, which includes a visible light digital image and a thermal digital image.

[0057] Figures 7 to 9 Visible light and thermal digital images captured by the visible light and thermal imaging cameras 19 , 20 are exemplarily shown.

[0058] Specifically, Figure 7The visible light digital image of the inner area of ​​the packaging material 3 captured by the visible light imaging camera 19 of the first electronic digital image capture device 17 after the first longitudinal side of the sealing strip 6 is heat sealed is shown by way of example. In the image, the narrower light grey vertical stripe 21 represents the entire sealing strip 6, including the first longitudinal side portion heat-sealed with the longitudinal side portion of the packaging material 3 and the second (loose) longitudinal side portion extending from the longitudinal edge of the packaging material 3. The narrower dark grey stripe 22 to the left of the light grey vertical stripe represents the background. The wider dark gray vertical stripes to the right of the light gray vertical stripes represent the packaging material 3, which includes a narrower vertical stripe portion 23 on the left, which is slightly darker and more uniformly gray than the wider vertical stripe portion on the right, and represents an area on the longitudinal side of the packaging material 3, which is adjacent to the sealing strip 6 and is heated during the heat sealing of the first longitudinal side of the sealing strip 6, but is not covered or overlapped by the sealing strip 6, while the wider vertical stripe portion 24 on the right is slightly darker and less uniformly gray, representing the remaining part of the packaging material 3 that is not heated.

[0059] Figure 8 A thermal digital image of the inner area of ​​the packaging material 3 is shown as an example, which is similar to Figure 7 , but captured by the thermal imaging camera 20 of the first electronic digital image capture device 17 after the first longitudinal side of the sealing strip 6 has been heat sealed. Figure 7 The same reference numerals in the figures represent the same parts of the digital images. Figure 7 As shown, the narrower light gray vertical stripe represents the entire sealing strip 6, including the first longitudinal side portion heat-sealed to the longitudinal side portion of the packaging material 3 and the loose longitudinal side portion extending from the longitudinal edge of the packaging material 3; the wider dark gray vertical stripe on the right side of the light gray vertical stripe represents the packaging material 3, and the narrower dark gray stripe on the left side of the light gray vertical stripe represents the background. Also in this image, the wider dark gray vertical stripe on the right side includes the narrower vertical stripe portion on the left side, the narrower vertical stripe portion on the left side is slightly darker than the wider vertical stripe portion on the right side, representing the heating pattern, and the wider vertical stripe portion on the right side is slightly darker than the narrower vertical stripe portion on the left side, representing the remaining portion of the packaging material 3 that is not heated.

[0060] Fig. 9 The example shows Figure 7 , but captured by the visible light imaging camera 19 of the second electronic digital image capture device 18 from the outside or decorative side of the packaging material 3 after the first longitudinal side portion of the sealing strip 6 has been heat sealed. Figure 7 and Figure 8 The same reference numerals in the figures represent the same parts of the digital images. Figure 7 and Figure 8 In the image shown, the narrower light grey vertical stripes represent loose longitudinal side portions extending from the longitudinal edges of the packaging material 3 ; the narrower dark grey stripes to the left of the light grey vertical stripes represent the background; and the wider light grey vertical stripes to the right of the light grey vertical stripes represent the packaging material 3 .

[0061] In one embodiment, the electronic computing resource 14 is further configured to process the captured digital images to automatically detect and identify any failures that may have occurred during heat sealing of the sealing strip 6 to the packaging material 3 based on the captured digital images.

[0062] Specifically, Fig.10 The electronic computing resource 14 is shown processing the visible light digital image and the corresponding thermal digital image (eg Figure 7 and Figure 8 Block diagram of image processing of images (those shown in FIG. 1 ), these images are of the same area of ​​the inside of the packaging web 3 and the sealing strip 6.

[0063] Specifically, electronic computing resource 14 is programmed to receive a sequence of visible light and thermal digital image pairs and, for each pair of visible light and thermal digital images, perform image processing of the received visible light digital image and image processing of the visible light and thermal digital image pair.

[0064] For image processing of visible light digital images, the electronic computing resource 14 is programmed to: - receiving a pair of visible light and thermal digital images (block 25); -Process the received visible light digital images to find: ·Marks or patterns that indicate typical coarse, large or macroscopic defects or anomalies, such as Fig.11 The scratches or delaminations shown by way of example in a and 11b, these defects or anomalies originate from the heat pattern during the heat sealing of the sealing strip 6 and the packaging web 3 (box 26), and ·Marks or patterns that indicate typical finer, smaller or microscopic defects or anomalies, e.g. Fig.11 Certain wrinkles are exemplarily shown in the visible light digital images in c to 11f and originate from the heat pattern during heat sealing of the sealing strip 6 and the packaging web 3 (frame 27).

[0065] Output data representing the identified marks or patterns representing typical macro and micro defects or anomalies are calculated and input to the inner classifier (box 28), wherein the heat seal of the sealing strip 6 to the packaging web 3 is evaluated based on the identified marks or patterns representing typical macro and micro defects or anomalies, if any.

[0066] To look for markings or patterns that represent typical microscopic defects or anomalies, the captured visible light digital images are processed to look for: -Diagonal wrinkles (box 29), such as Fig.11 The wrinkles shown in the captured visible light digital image shown in c are, in the area of ​​the captured visible light digital image corresponding to the heat pattern, particularly wrinkles inclined at about 45° relative to the longitudinal edges of the sealing strip 6 (particularly the longitudinal edges of the longitudinal sides of the sealing strip 6 heat-sealed to the packaging web 3); the diagonal wrinkles are caused by overheating of the heat pattern during heat sealing of the sealing strip 6 to the packaging web 3; and - substantially orthogonal wrinkles (hereinafter referred to as transverse wrinkles) (block 30), in the region corresponding to the sealing strip 6 in the captured visible light digital image (e.g. Fig.11 d) and the area corresponding to the thermal pattern (e.g. Fig.11 e), in particular wrinkles inclined at about 90° relative to the longitudinal edges of the sealing strip 6 (in particular the longitudinal edges of the longitudinal sides of the sealing strip 6 heat-sealed to the packaging web 3); the diagonal wrinkles are caused by process defects and occur during the heat sealing of the sealing strip 6 to the packaging web 3.

[0067] Output data representing any identified diagonal and transverse wrinkles is calculated and input to the medial classifier (box 28).

[0068] like Fig.10 As shown, the visible light digital image is further processed (box 31): - storing or receiving data indicating a width tolerance range for an expected thermal pattern width; - Calculation of the actual heat pattern width, measured in a direction orthogonal to the longitudinal edge of the sealing strip 6; - Check the actual thermal pattern width against the width tolerance range to determine whether the thermal pattern is within or outside the width tolerance range; and - calculations indicating whether the thermal pattern width is within or outside the width tolerance; and - the calculated data indicating whether the heat pattern width is within or outside the width tolerance is input to an inside classifier, wherein the heat seal of the sealing strip 6 to the packaging web 3 is also evaluated based on whether the heat pattern width is within or outside the width tolerance.

[0069] In a different embodiment, the calculated heat pattern width may be provided to an internal classifier, wherein the heat seal of the sealing strip 6 to the packaging web 3 is also evaluated based on the calculated heat pattern width.

[0070] The visible light digital image is also processed (block 32): - storing or receiving data indicative of an overlap tolerance range indicating the extent to which the sealing strip 6 is expected to overlap the longitudinal edge of the packaging web 3; - Calculate the actual sealing strip overlap, which is measured in a direction orthogonal to the longitudinal edge of the sealing strip 6, indicating the extent to which the sealing strip 6 actually overlaps the longitudinal edge of the packaging web 3; - Check the actual seal strip overlap with the overlap tolerance range to determine whether the actual seal strip overlap is within or outside the overlap tolerance range; - calculating data indicating whether the actual seal overlap is within or outside the overlap tolerance range; and - the calculated data indicating whether the actual sealing strip overlap is within or outside the overlap tolerance range is input to the inner classifier, wherein the heat seal of the sealing strip 6 to the packaging web 3 is also evaluated based on whether the calculated actual sealing strip overlap is within or outside the overlap tolerance range.

[0071] In a different embodiment, the calculated actual sealing strip overlap may be provided to an internal classifier, wherein the heat seal of the sealing strip 6 to the packaging web 3 is evaluated based on the calculated actual sealing strip overlap.

[0072] The actual thermal pattern width and the amount of sealing strip overlap are calculated by determining the number of pixels in the area corresponding to the thermal pattern and the sealing strip in the processed visible light digital image in a direction orthogonal to the longitudinal edge of the sealing strip 6, and multiplying it by a predetermined conversion factor, which converts the number of pixels of the thermal pattern width and the amount of sealing strip overlap into a numerical value in millimeters.

[0073] Conveniently (but not necessarily), the captured visible light digital image may be processed by first segmenting (block 33) the visible light digital image into a plurality of segments or regions (sets of pixels, also referred to as image objects) corresponding to the sealing strip 6, the heat pattern and the unheated remainder of the packaging web 3 to look for any marks or patterns representing typical microscopic defects or anomalies, such as Fig.12 As shown, a segmented visible light digital image including only the area corresponding to the sealing strip 6 and the thermal pattern is output, and then the segmented visible light digital image is processed to find the above-mentioned microscopic defects or abnormalities, as described above.

[0074] Specifically, areas corresponding to the sealing strip 6 are identified by detecting significant changes in pixel brightness in a direction orthogonal to the longitudinal edge of the sealing strip 6 compared to adjacent areas, while areas of the thermal pattern are identified by evaluating the color uniformity compared to adjacent areas corresponding to the remainder of the packaging web 3, where color changes and average gray levels are high.

[0075] Processing of visible light digital images to find any marks or patterns representing typical macroscopic defects or anomalies can be conveniently based on semi-supervised learning and generative models, in particular based on Fig.13 An artificial neural network structure called a variational autoencoder (VAE) is shown, which is trained on a dataset consisting only of visible-light digital images without macroscopic defects or anomalies. Fig.13 The visible light digital images shown on the right show the output of VAE in non-anomalous scenes with low reconstruction error and high reconstruction probability, that is, when the processed visible light digital images do not show typical macroscopic defects or anomalies, such as Fig.13 The upper left part shows, as well as the output in an abnormal scene with high reconstruction error and low reconstruction probability, i.e., when the processed visible light digital image exhibits typical macroscopic defects or anomalies, e.g. Fig.13 Shown in the lower left portion.

[0076] Processing visible light digital images to look for transverse wrinkles in thermal patterns can be conveniently based on a convolutional neural network (CNN) with a UNet architecture, which is Fig.14 exemplarily shown in FIG. Fig.15 The dataset created and labeled as shown is used for training.

[0077] Processing the captured visible light digital images to look for diagonal wrinkles can be conveniently based on Fig.16 The convolutional neural network (CNN) architecture shown in Figure 1 was trained on a dataset of digital images with and without diagonal wrinkles.

[0078] Reference again Fig.10 The block diagram shown, and with respect to image processing of visible light and thermal digital images, the electronic computing resource 14 is programmed to: - processing the received visible light digital image (hereinafter also referred to as visible light digital image analysis (block 34)) to calculate data containing information (hereinafter also referred to as geometric information) indicating the geometry of the sealing strip 6 and the thermal pattern of the packaging web 3, in particular the positions of the longitudinal edges of the sealing strip 6, the longitudinal edges of the packaging web 3 and their thermal patterns in a reference system, which conveniently takes the form of a coordinate system, the origin of which is located at a suitable point of the processed visible light digital image, for example a longitudinal edge of a loose longitudinal portion of the sealing strip 6 extending from the packaging web 3 or a longitudinal edge of the packaging web 3; - processing the received thermal digital image (hereinafter also referred to as thermal digital image analysis (block 34)) to calculate data containing information (hereinafter referred to as thermal information) indicating the temperature profile of the packaging web 3 and of the sealing strip 6 heat-sealed thereto in a direction orthogonal to the longitudinal edges of the packaging web 3, and the temperature profile of the sealing strip 6 in a reference system, the reference system conveniently taking the form of a coordinate system, the origin of which is located at a suitable position of the processed thermal digital image, for example at a longitudinal edge of a longitudinal portion of the sealing strip 6 not heat-sealed to the packaging web 3, or at a longitudinal edge of the packaging web 3; - fusing (data fusion), i.e. combining or merging the output data of the visible light digital image analysis and the thermal digital image analysis (block 34) to calculate fused data representing a combination of geometric information obtained from the visible light digital image analysis and thermal information obtained from the thermal digital image analysis and based on which any underheating (cold seal) or overheating areas generated during the heat sealing of the sealing strip 6 to the packaging web 3 can be identified; and - Processing the fusion data to identify any underheating or cold sealing (box 35) or overheating (box 36) that may have occurred during the heat sealing of the sealing bar 6 to the packaging web 3, and calculating output data representing the identified underheating or overheating and inputting it into the inside classifier (box 28).

[0079] Fig.17 A graphical representation of the fusion of the output data of the visible light digital image analysis with the output data of the thermal digital image analysis is shown.

[0080] It will be appreciated that the thermal information from the thermal digital image analysis in the form of the temperature distribution of the packaging web 3 and the sealing strip 6 heat-sealed thereto in a direction orthogonal to the longitudinal edge of the sealing strip 6 is graphically superimposed on the graphical representation of the geometric information obtained from the visible light digital image analysis in the form of the geometric shape of the sealing strip 6 and the thermal pattern of the packaging web 3, thereby allowing the identification of any underheating (cold seal) or overheating areas produced during the heat sealing of the sealing strip 6 to the packaging web 3.

[0081] In the inside classifier (box 28), the quality of the heat seal of the sealing strip 6 to the inside of the packaging web 3 is evaluated based on the macro and micro defects (if any) identified in the heat pattern and sealing strip 6, the calculated heat pattern width and strip overlap, the identified underheating (cold seal) or overheating (if any), and a proprietary standard developed taking into account the severity of the above quantities.

[0082] Regarding the calculation of the temperature distribution of the packaging coil 3 and the sealing strip 6, and exemplarily referring to Figure 8 The thermal digital image shown can conveniently calculate the temperature distribution of the packaging coil 3 and the sealing strip 6. First, the left longitudinal boundary of the sealing strip 6 is identified in the thermal digital image. Then, a custom algorithm is used and the calculation is performed from left to right along a direction orthogonal to the sealing strip 6, starting from the identified left longitudinal boundary of the sealing strip 6, to the right longitudinal boundary of the sealing strip 6 and beyond the boundary, and throughout the entire thermal pattern. The temperature value is calculated based on the brightness value of the pixel primary color of the thermal digital image.

[0083] In order to correctly merge the calculated geometric information and thermal information, the calculated geometric information and thermal information first need to be aligned in the spatial domain, especially in the direction parallel to the sealing strip 6 and the direction orthogonal to the sealing strip 6, so that the calculated thermal information is related to the same spatial point of the packaging roll 3 and the sealing strip 6 involved in the calculated geometric information.

[0084] The calculated alignment of geometric and thermal information results in the visible light and thermal digital images being conceptually superimposable, that is, the packaging web 3 and sealing strip 6 depicted in the visible light digital image can be conceptually superimposed onto the packaging web 3 and sealing strip 6 depicted in the thermal digital image.

[0085] Fig.18 A combined visible light and thermal digital image obtained by superimposing a visible light digital image and a corresponding thermal digital image is exemplarily shown.

[0086] To this end, the reference frames of the visible light and thermal digital images and the geometric and thermal information referenced therein need to be spatially / geometrically related to each other through a so-called spatial transformation matrix, which expresses the mutual rotation and translation of the two reference frames so that the sealing strip 6 and packaging web 3 in the visible light digital image can be related to the sealing strip 6 and packaging web 3 in the thermal digital image.

[0087] Therefore, the visible light and thermal imaging cameras 19, 20 of each electronic digital image capture device 17, 18 are calibrated to calculate a transformation matrix that will be used to align each pair of visible light and thermal digital images during a heat seal quality control run.

[0088] Calibration of the visible light and thermal imaging cameras is performed using a calibration mark 21, which is designed to include or display a calibration pattern 22, so that when the calibration mark 21 is stably arranged in the field of view (FOV) of the visible light and thermal imaging cameras 19, 20 of each electronic digital image capture device 17, 18 and the visible light and thermal imaging cameras 19, 20 are operated simultaneously or continuously, the calibration pattern 22 is imaged (i.e., visible) in the visible light digital image and thermal digital image captured by the visible light and thermal imaging cameras 19, 20, respectively, thereby allowing the spatial transformation matrix to be calculated based on the position and orientation of the pattern imaged in the visible light and thermal digital images.

[0089] In the embodiment shown in Figures 19a to 19c, the calibration mark 21 includes a foreground member 23 and a background member 24, wherein the foreground member 23 is arranged in the foreground relative to the visible light and thermal imaging cameras 19, 20 and carries the calibration pattern 22 in the form of a through-passing pattern, and the background member 24 is arranged behind the foreground member 23 relative to the visible light and thermal imaging cameras 19, 20 so as to be exposed in the foreground (i.e., visible) through the through-passing pattern 22.

[0090] In the embodiment shown in Figures 19a to 19c, the background member 24 is in the form of a foil (e.g., aluminum foil), and the foreground member 23 includes a plate 25 having a calibration pattern 22 formed therethrough, and a support and coupling structure 26 integrated with the plate 25, the support and coupling structure 26 being designed to allow the background member 24 to be easily and detachably coupled / retained behind the plate 25, and the plate 25 coupled with the background member 24 can be stably placed in the field of view of the visible light and thermal imaging cameras 19, 20.

[0091] In the embodiment shown in FIGS. 19 a to 19 c , the calibration pattern 22 is in the form of a grid of staggered rows of through holes, ie a grid of through holes in which one row of through holes is staggered relative to an adjacent row of through holes.

[0092] The foreground member 23 and the background member 24 are made of materials having different emissivities, and in particular, the foreground member 23 is made of a material having a lower emissivity (e.g., PLA plastic) than the (higher) emissivity of the material of the background member 24 (e.g., aluminum), whereby, when the background member 24 is properly heated, the heated area of ​​the background member 24 exposed by the through pattern 22 can be imaged in the thermal digital image and form a corresponding calibration pattern, while the rest of the background member 24 may not be imaged in the thermal digital image due to being blocked by the foreground member 23.

[0093] In this way, the calibration pattern 22 formed in the foreground member 23 can be imaged in the visible light digital image, and after the background member 24 is appropriately heated, the corresponding pattern formed by the heated area of ​​the background member 24 exposed by the calibration pattern 22 formed in the foreground member 23 can also be imaged in the thermal digital image.

[0094] Figure 20a shows calibration pattern 22 formed in foreground member 23 and imaged in the visible light digital image, while Figure 20b shows the corresponding pattern formed by heated areas of background member 24 exposed by calibration pattern 22 formed in foreground member 23 and imaged in the thermal digital image.

[0095] Therefore, calibrating the output data of the visible light and thermal imaging cameras 19, 20 of each electronic digital image capture device 17, 18 requires: - a background member 24 for heating the calibration mark 21; - Arrange the calibration mark 21 within the field of view of the visible light and thermal imaging cameras 19, 20; - operating the visible light and thermal imaging cameras 19, 20 to capture visible light digital images and thermal digital images with respective calibration patterns; - Processing the captured visible light and thermal digital images to identify calibration patterns therein; - Calculate the position and orientation of the calibration pattern in visible light and digital images; and - Compute the spatial transformation matrix based on the position and orientation of the calibration pattern in the visible light and digital image.

[0096] To this end, the electronic computing resource 14 is further programmed to: - operating the visible light and thermal imaging cameras 19, 20 simultaneously or continuously to capture visible light digital images and thermal digital images with respective calibration patterns; - Processing the captured visible light and thermal digital images to identify the calibration pattern therein and calculate its position and orientation; - Compute a spatial transformation matrix based on the calculated positions and orientations of the calibration patterns in the visible light and digital images.

[0097] Fig.21 FIG. 1 is a block diagram showing image processing by the electronic computing resource 14 on the captured visible light digital image and thermal digital image of the same area of ​​the outer side or decorative side of the packaging web 3 and the sealing strip 6, wherein the visible light digital image is as shown in FIG. Fig. 9 shown.

[0098] For convenience, in the block diagram shown in FIG. 20, the Fig.10 The blocks described in the block diagrams are similar or corresponding to the processes described in the block diagrams. Fig.10The same reference numerals are used to refer to the same figures in FIG.

[0099] like Fig.21 As shown, by processing the visible light digital image, it is determined whether the sealing strip 6 is folded in a U shape around the longitudinal edge of the packaging web 3 (frame 37), and whether one or more "fishbone"-like marks (e.g., Fig.21 ) (box 38).

[0100] If it is determined that the sealing strip 6 is U-folded, the actual sealing strip overlap measured in a direction orthogonal to the longitudinal edge of the sealing strip 6 is calculated (block 39), as previously described with reference to Fig.10 As described in block 32 shown in FIG.

[0101] Instead, the thermal digital image is processed to identify thermal digital image regions having predefined characteristics, in particular regions having predefined brightness patterns, e.g. Fig.21 The area within the red rectangle shown in the reproduced thermal digital image in FIG. 4 is then processed to find underheating (cold sealing) that may have occurred during the heat sealing of the sealing strip 6 to the packaging web 3 (block 42), as previously described with reference to FIG. Fig.10 As described in box 35 shown in .

[0102] The heat seal quality of the sealing strip 6 to the packaging web 3 is evaluated in an outer classifier (block 40), which is similar to Fig.10 The inside classifier shown in , reference box 28, is also based on the above quantities and proprietary criteria (wherein the severity of the above quantities is taken into account).

[0103] In order to detect the presence and number of "fishbones" in visible light digital images, a Fig. 22 The deep learning algorithm of the UNet network is shown in the example, and is based on Fig.23 Schematically illustrated is the creation and labeling of a dataset for training.

[0104] at last, Fig.24 A block diagram is shown of the image processing by the electronic computing resource 14 of the captured visible light and thermal digital images of the longitudinal seal area of ​​the vertical tube 2 after the packaging web 3 has been folded and its longitudinal edges have been heat sealed.

[0105] like Fig.24As shown, the visible light digital image is processed to identify areas of the thermal digital image having predefined characteristics, in particular areas having predefined brightness patterns, such as the area within the red rectangle shown in the visible light digital image reproduced in Figure 210 (box 41), and the identified areas of the visible light digital image are then processed to determine whether the vertical tube 2 is twisted during the vertical sealing process (box 42) and whether the longitudinal edges of the packaging roll 3 are incorrectly overlapped (box 43).

[0106] Output data indicating whether the vertical tube 2 is twisted during the vertical sealing process and whether the longitudinal edges of the packaging web 3 are incorrectly overlapped is calculated and input into a similar Fig.10 and 21 The longitudinal sealing sorter (box 44) of the inner and outer sorters are shown.

[0107] Alternatively, the thermal digital image is processed to look for signs indicating any underheating (cold sealing) that occurred during the heat sealing of the sealing strip 6 to the packaging web 3, as previously described with reference to Fig.10 As described in box 35 shown in .

[0108] Output data representing any identified signatures of underheating (cold seals) is calculated and input to a longitudinal seal classifier (box 44) and the quality of the longitudinal seal of the vertical tube 2 is assessed based on a proprietary criterion that takes into account the severity of the identified features.

[0109] Those skilled in the art can easily understand the advantages that can be achieved by the present invention.

[0110] In particular, the present invention allows for automated inspection of the heat seal quality of a sealing bar to a packaging web by a simple yet powerful and reliable optical imaging technique and represents a significant improvement over heat seal quality inspection by a suitably trained operator.

Claims

1. An automatic heat seal quality control system (12) for automatically controlling the heat seal quality of a sealing strip (6) to a longitudinal edge of a packaging web (3) at a heat seal station (7) in a packaging machine (1), wherein: Continuously producing sealed packages (5) containing the pourable food product from a continuous vertical tube (2) filled with the pourable food product and formed by longitudinally folding the packaging web (3) so as to overlap its longitudinal edges and then heat-sealing the overlapped longitudinal edges; The system (12) comprises: a sensor system (13) arranged at the heat sealing station (7) for delivering an output allowing the quality of the heat seal of the sealing strip (6) to the longitudinal edge of the packaging web (3) to be assessed; and - an electronic computing resource (14) for communicating with the sensor system (13) to receive and process the output thereof to automatically assess the quality of the heat seal of the sealing strip (6) to the longitudinal edge of the packaging web (3); The sensing system (13) comprises an artificial vision system (15) for capturing digital images of the packaging web (3) and the sealing strip (6); The artificial vision system (15) comprises: - A visible light imaging camera (19) designed to operate in the electromagnetic spectrum visible to the human eye to capture and output visible light digital images.

2. The system (12) of claim 1, wherein the electronic computing resource (14) is configured to: - operating the visible light imaging camera (19) to capture visible light digital images of the packaging material (3) and the sealing strip (6); - receiving said captured visible light digital image; - processing the received visible light digital image to find at least one of: markings or patterns (26) representing macroscopic defects or anomalies, such as scratches or delaminations, which may have formed during the heat sealing of the sealing strip (6) to the packaging web (3), and markings or patterns (27) representing microscopic defects or anomalies, such as wrinkles, which may form during the heat sealing of the sealing strip (6) to the packaging web (3); and - evaluating the quality of the heat seal of the sealing strip (6) to the packaging web (3) based on the identified marking or pattern.

3. The system (12) of claim 1 or claim 2, wherein: The electronic computing resource (14) is also configured to process the received visible light digital image to look for marks or patterns indicative of macroscopic and microscopic defects or anomalies in the thermal pattern of the packaging web (3), the thermal pattern being defined as areas of the packaging web (3) adjacent to the sealing strip (6) and which are heated during heat sealing of the sealing strip (6) to the packaging web (3) but not covered by the sealing strip (6).

4. The system (12) of claim 3, wherein the electronic computing resource (14) is further configured to process the received visible light digital image to look for markings or patterns indicative of microscopic defects or anomalies, comprising: - diagonal wrinkles (29) in the thermal pattern of the packaging web (3); as well as - one or all of the transverse folds (30) in the sealing strip (6) and / or in the thermal pattern in the packaging web (3).

5. The system (12) of any one of the preceding claims, wherein the electronic computing resource (14) is further configured to process the received visible light digital image to (31): - calculating the actual heat pattern width, measured in a direction orthogonal to the sealing strip (6), the heat pattern representing the area of ​​the packaging web (3) adjacent to the sealing strip (6) and which is heated during heat sealing of the sealing strip (6) to the packaging web (3) but not covered by the sealing strip (6); and - The quality of heat sealing the sealing strip (6) to the packaging web (3) is also evaluated based on the calculated heat pattern width.

6. The system (12) of any one of the preceding claims, wherein the electronic computing resource (14) is further configured to process the received visible light digital image to (32): - calculating the actual sealing strip overlap measured in a direction orthogonal to the sealing strip (6), which overlap indicates the extent to which the sealing strip (6) to the packaging web (3) actually overlaps; and - also evaluating the heat seal quality of the sealing strip (6) to the packaging web (3) based on the calculated actual sealing strip overlap.

7. The system (12) of any one of the preceding claims, wherein the electronic computing resource (14) is further configured to process the received visible light digital image to (37): - determining whether the sealing strip (6) is folded in a U-shape around the longitudinal edge of the packaging web (3) and whether a herringbone-like mark is formed in the sealing strip (6) during the U-folding and during the heat sealing of the sealing strip (6) to the packaging web (3); - if it is determined that the sealing strip (6) is U-folded, calculating the actual sealing strip overlap measured in a direction orthogonal to the sealing strip (6), the overlap indicating the degree to which the sealing strip (6) actually overlaps the packaging web (3); as well as - also based on determining whether the sealing strip (6) is folded in a U-shape around the longitudinal edge of the packaging web (3), and whether a fishbone-like mark is formed in the sealing strip (6) during the U-shape folding and during the heat sealing of the sealing strip (6) to the packaging web (3), and the calculated actual sealing strip overlap, the heat sealing quality of the sealing strip (6) to the packaging web (3) is evaluated.

8. The system (12) according to any one of the preceding claims, wherein the artificial vision system (15) further comprises: - a thermal imaging camera (20) designed to operate in the electromagnetic spectrum invisible to the human eye to capture and output thermal digital images; The electronic computing resource (14) is configured to: - operating the thermal imaging camera (19) to capture thermal digital images of the packaging material (3) and the sealing strip (6); - receiving said captured thermal digital image; - Processing the received thermal digital image to (41): identifying areas of said thermal digital image having predefined characteristics, in particular a predefined brightness pattern; processing said identified areas of said thermal digital image, if any, to identify underheating or cold sealing which may have occurred during said heat sealing of said sealing strip (6) to said packaging web (3); and - also evaluating the quality of the heat seal of the sealing strip (6) to the packaging web (3) based on the identified underheated or cold seals, if any.

9. The system (12) according to claim 8, wherein the visible light and thermal imaging cameras (19, 20) are arranged relative to the packaging web (3) and the sealing strip (6) to capture visible light and thermal digital images of the same area of ​​the same side of the packaging material (3) and the sealing strip (6); The electronic computing resource (14) is further configured to: - operating the visible light and thermal imaging cameras (19, 20) to capture a pair of visible light and thermal digital images of the packaging material (3) and the sealing strip (6); - receiving said captured pair of visible light and thermal digital images; - performing a visible light digital image analysis (34) on the received visible light digital image to calculate and output data containing geometric information representing the geometric shape of the sealing strip (6) and the thermal pattern of the packaging web (3) in a reference system, the origin of the reference system being located at an appropriate point of the processed visible light digital image; - performing a thermal digital image analysis (34) on the received thermal digital image to calculate and output data containing thermal information representing the temperature distribution of the packaging web (3) and the sealing strip (6) in a reference system in a direction orthogonal to the packaging web (3) and the sealing strip (6), the origin of the reference system being located at an appropriate point of the processed thermal digital image; - fusing the geometric information obtained from the visible light digital image analysis with the thermal information obtained from the thermal digital image analysis to calculate fused data representing the fused information; - processing said fusion data to identify instances of underheating or cold sealing (35) or overheating (36), if any, which may have occurred during said heat sealing of said sealing bar (6) to said packaging web (3); and - Based on the identified underheating or overheating condition, evaluating the quality of the heat seal of the sealing strip (6) to the packaging web (3).

10. The system (12) according to claim 8 or 9, wherein: The electronic computing resource (14) is further configured to align the geometric information with the thermal information in the spatial domain and in a direction parallel to and orthogonal to the sealing strip (6) so that the calculated thermal information is associated with the same spatial point of the packaging web (3) and the sealing strip (6) to which the calculated visible light information relates; In order to align the geometric information with the thermal information in the spatial domain, the electronic computing resource (14) is further configured to calculate a spatial transformation matrix that relates the reference frames of the geometric information and the thermal information referenced in the visible light digital image and the thermal digital image to each other, so that the image of the sealing strip (6) and the packaging web (3) in the visible light digital image can be associated with the image of the sealing strip (6) and the packaging web (3) in the thermal digital image; In order to calculate the spatial transformation matrix, the system (12) comprises a calibration mark (21) designed to exhibit a calibration pattern (22) that can be imaged in a visible light digital image and a thermal digital image; And the electronic computing resource (14) is also configured to calculate the spatial transformation matrix based on the position and orientation of the calibration pattern (22) imaged in the visible light digital image and the thermal digital image.

11. The system (12) according to claim 10, wherein the calibration mark (21) comprises a foreground member (23) for being arranged in the foreground relative to the visible light and thermal imaging cameras (19, 20) and carrying the calibration pattern (22) in the form of a through pattern; and a background member (24) for being arranged behind the foreground member (23) relative to the visible light and thermal imaging cameras (19, 20) so as to be exposed in the foreground by the calibration pattern (22); The foreground component (23) and the background component (24) are made of materials with different emissivity, so that the calibration pattern (22) formed in the foreground component (23) can be imaged in a visible light digital image, and when the background component (24) is heated, the heated area of ​​the background component (24) exposed to the foreground through the calibration pattern (22) formed in the foreground component (23) can be imaged in a thermal digital image and form a corresponding calibration pattern, while the rest of the background component (24) is not imaged in the thermal digital image because it is blocked by the foreground component (23).

12. A packaging machine (1) operable to continuously produce sealed packages (5) containing a pourable food product from a continuous vertical tube (2) filled with said pourable food product, said continuous vertical tube (2) being formed by longitudinally folding a packaging web (3) so as to overlap its longitudinal edges and then heat sealing said overlapping longitudinal edges; The packaging machine (1) comprises an automatic heat seal quality control system (12) as claimed in any one of the preceding claims.

Images