A computer-implemented method for quality control of packages produced on roll-fed packaging machines using machine learning

Abstract

The present invention relates to a method (100) for quality control of packages produced on a roll-fed packaging machine. The method (100) includes a series of iterative steps, including acquiring image data depicting at least a portion of a package from an image data capture device (S102); determining a quality measurement of the package by inputting the image data into a machine learning model, such as a neural network, in response to the roll-fed packaging machine being in an autonomous mode (S106); determining an action to be taken based on the determined quality measurement (S108); communicating the action to a control system of the roll-fed packaging machine (S110); determining a performance index of the machine learning model based on the quality measurement of the package and previously evaluated quality measurements of the package (S112); assigning the mode of the roll-fed packaging machine to a re-learning mode if the performance index is outside a performance index interval (S114); and, in response to the roll-fed packaging machine being in the re-learning mode, collecting image data as training data for the machine learning model (S116), re-training the machine learning model using the training data (S118), and assigning the mode of the roll-fed packaging machine to the autonomous mode (S120).

Description

[Technical Field]

[0001] The present invention relates to the field of packaging technology, and more particularly to a method and apparatus for quality control of packages produced on a roll-fed packaging machine, and a food production system for a roll-fed packaging machine. [Background technology]

[0002] Food packaging technology plays a vital role in today's food industry. Food packaging has several important functions. In addition to branding products and providing information to customers, food packaging also plays a vital role in ensuring food safety. Packaging materials used in food packaging can be designed to provide strength and stability to prevent damage during transportation. Furthermore, packaging materials can create a protective environment for food, protecting it from factors such as bacteria, germs, oxygen, and sunlight, thereby extending its shelf life. However, packaging materials are not the only important aspect of a package. A package must also be properly sealed. Damage to the packaging material or improper sealing can compromise this protective environment. For example, in roll-fed packaging systems, packages are created by forming a longitudinal seal along the overlap between the two ends of the packaging material to form a tube. A package can then be formed from the tube by forming transverse seals at the top and bottom.

[0003] As such, food packaging has many important aspects. Ensuring the quality of food packaging produced by a packaging machine is no easy task. Today, the common method is to manually evaluate each batch of finished products. This involves not only visually inspecting the appearance of the package, but also opening the package and evaluating the seal formed inside the package. However, this is a tedious task and leads to product waste. Furthermore, to ensure quality, a large number of packages must be evaluated. Therefore, there is a need to improve the quality control of packaging produced by a packaging machine. Summary of the Invention [Problem to be solved by the invention]

[0004] The present disclosure seeks to at least partially mitigate, alleviate or eliminate one or more of the above-mentioned deficiencies and drawbacks of the prior art, and in particular to provide a method and apparatus for quality control of packages produced on roll-fed packaging machines.

[0005] The inventors of the present invention have realized a new and improved method for assessing packaging quality using machine learning. The present invention also provides for continuous monitoring of packaging to achieve greater confidence in packaging quality compared to today's methods of manually assessing samples. Furthermore, early action can be taken, leading to less wasted product. [Means for solving the problem]

[0006] Various aspects of the disclosed embodiments of the invention are defined in the accompanying independent and dependent claims below.

[0007] According to a first aspect, there is provided a method for quality control of packages produced by a roll-fed packaging machine, the method including a series of iterative steps including acquiring image data depicting at least a portion of a package from an image data capture device, determining a quality measurement of the package by inputting the image data into a machine learning model such as a neural network in response to the roll-fed packaging machine being in an autonomous mode, determining an action to be taken based on the determined quality measurement, and communicating the action to a control system of the roll-fed packaging machine, determining a performance index of the machine learning model based on the package quality measurement and previously evaluated package quality measurements, and assigning the mode of the roll-fed packaging machine to a re-learning mode if the performance index is outside a performance index interval, and in response to the roll-fed packaging machine being in the re-learning mode, collecting image data as training data for the machine learning model, re-training the machine learning model using the training data, and assigning the mode of the roll-fed packaging machine to the autonomous mode.

[0008] The machine learning model may be a deep learning model. A deep learning model should be understood to be a model based on deep learning. Deep learning should be understood to be machine learning in which hypotheses take the form of complex algebraic circuits with adjustable connection strengths. The term "deep" refers to the fact that circuits are typically organized into many layers, meaning that the computational path from input to output has many steps.

[0009] The phrase "quality control" refers to the evaluation of one or more quality aspects of a package. A quality aspect may be related to food safety, such as whether the package or packaging materials are intact, or whether there are any imperfections or irregularities in the formation of the package (e.g., the sealing or folding of the package). Another quality aspect may be related to the aesthetics of the package, for example, looking for deformations in the package (which may also be related to food safety) or errors / misalignments in the packaging decoration.

[0010] The phrase "manufactured," as used herein, such as "manufactured on a roll-fed packaging machine," means that the package has been manufactured or is being manufactured on a roll-fed packaging machine.

[0011] The method can be repeated (e.g., by performing a series of iterative steps multiple times) to continually perform quality control on subsequent packages. It should be noted, therefore, that the image data obtained in each iteration can depict a different portion of the same package. Alternatively, the image data obtained can depict a different package than during a previous iteration. In other words, the method 100 can be performed on multiple packages subsequently manufactured. Furthermore, the package can be depicted in the image data at any state during the manufacturing process. For example, the image data can depict the package in an unformed state (e.g., a web of packaging material), a partially formed state (e.g., formed as a tube), or a fully formed state (e.g., a finished package), or any state in between.

[0012] The term "autonomous mode" refers to a mode in which quality control in a roll-fed packaging machine is performed automatically using a machine learning model, independent of operator input. As long as the machine learning model meets certain requirements (i.e., based on performance indicators), the roll-fed packaging machine can be in autonomous mode. Over time, and as new scenarios (e.g., new types of defects) emerge, the performance of the machine learning model may degrade. To mitigate this, a retraining mode for the roll-fed packaging machine can be utilized. The expression "retraining mode" is herein understood as a mode in which the machine learning model is retrained with new scenarios. In other words, the machine model is monitored to detect whether it begins to deviate from expected behavior. In that case, the collection of data necessary to retrain the machine learning model is automatically triggered. After retraining, the machine learning model can be deployed again, and the roll-fed packaging machine can be set in autonomous mode again. Therefore, a related benefit is potentially the possibility of continuous inspection of packages, and the continuous learning system of the machine learning model can identify new or previously unseen defects in the packages. In other words, the machine learning model may be allowed to run as long as it performs as intended. If it does degrade, it may stop running and re-learn instead. Additionally, by continuously monitoring the quality measurements of the packages, it may be possible to detect if any component of the packaging machine begins to degrade (i.e., if the quality measurement begins to degrade). This is advantageous in that preventative action, such as part replacement, can be taken before the part fails completely.

[0013] Additionally, when machine learning models perform poorly or new scenarios emerge, they can be retrained to handle new packaging materials or foods that may lead to changes in how quality assessments are performed. As an example, if the packaging material on a packaging machine is changed, for example to a thicker material to produce a different type of food packaging, potential defects in the packaging may appear differently compared to if a different packaging material were used. Thus, the machine learning model can self-adjust as needed to accommodate a wider range of scenarios.

[0014] As a more specific example, the proposed method allows a "generic" trained machine learning model to be deployed to different types of packaging machines, packaging materials, or food products. The machine learning model can then be trained (i.e., retrained) for the specific machine on which it is deployed to reach the desired performance. This is advantageous in that it simplifies the training process because the machine learning model does not need to be trained for its specific application. Instead, the "generic" machine learning model is trained once using the same training data and then deployed to the packaging machine, where it can be retrained for the specific case.

[0015] The package may comprise a carton layer and at least one plastic layer.

[0016] The image data may depict a longitudinal seal of the package, and the determined quality measure may be a quality measure of the longitudinal seal.

[0017] The longitudinal seal may be formed by forming a tube of the web of packaging material such that a first longitudinal edge of the web overlaps a second longitudinal edge of the web, attaching the first and second longitudinal edges, and applying a protective strip inside the tube to the overlapping portions of the first and second longitudinal edges. Image data of the longitudinal seal may be captured from inside the tube by an image data capture device.

[0018] The actions can be (i) reject or accept the package, (ii) send a warning signal to the packaging machine operator, (iii) stop the packaging machine, or (iv) adjust the packaging machine settings. Having the reject or accept actions allows for automatic discarding of defective packages based on quality measurements. A warning signal may also be sent to the operator, such as when it is detected that the quality of a package has started to deteriorate. This may be an early sign that something is wrong within the packaging machine, for example, that a part within the packaging machine needs to be replaced soon. Alerting the operator allows the packaging machine to be inspected to detect potential problems early. Stopping the packaging machine may also be performed if a more serious defect is detected in (or some) packages. Alternatively, if the defect is less serious, the packaging machine settings may be adjusted to correct the error.

[0019] The action may be to accept or reject the package. Communicating the action may include communicating the action to a package rejection unit located downstream of the packaging machine.

[0020] The action may be to accept or reject the package. When the roll-fed packaging machine is in relearn mode, the method may further include instructing a secondary rejection system to perform a quality assessment of the package. The secondary rejection system may be, for example, a manual quality control performed by a user or other quality control system based on which the package is either accepted or rejected.

[0021] The performance index may be determined based on the number of false positives and false negatives of previously evaluated packages. Thus, the performance index may be determined by looking at multiple subsequent packages that have been evaluated.

[0022] The method may further include adjusting settings of the roll-fed packaging machine based on the determined quality measurements. Preferably, the settings may be adjusted by increasing or decreasing the heating temperature and / or increasing or decreasing the amount of power during formation of the longitudinal seal.

[0023] The image data may be associated with identification data of the package. Communicating the action to the control system may further include communicating the identification data to the control system. The identification data may be used by the control system to know which package the action is associated with.

[0024] If the performance index decreases with the number of consecutive packages evaluated, the mode of the roll-fed packaging machine may be assigned to a re-learn mode. A consecutive decrease in the performance index may indicate that something in the machine learning model is not working properly or that the image data in the package depicts something that the machine learning model was not trained to evaluate.

[0025] The performance index may include multiple sub-indicators. In response to a sub-indicator being outside a corresponding sub-indicator interval, the method generates a task associated with the sub-indicator. The task includes image data and settings of the roll-fed packaging machine. The method may further include sending the task to a user group associated with the sub-indicator, receiving updated settings of the roll-fed packaging machine in response to the task from the user group, and adjusting the settings of the roll-fed packaging machine according to the updated settings. By utilizing the sub-groups, the method can utilize a user group best suited to evaluating the image data, and more specifically, package defects present in the image data.

[0026] According to a second aspect, there is provided a quality control device for quality control of packages produced by a roll-fed packaging machine. The quality control device includes a circuit configured to execute an acquisition function configured to acquire image data depicting at least a portion of a package from an image data capture device. In response to the roll-fed packaging machine being in an autonomous mode, the circuit is further configured to execute: a first decision function configured to determine a quality measurement of the package by inputting the image data into a machine learning model, such as a neural network; a second decision function configured to determine an action to be taken based on the quality measurement; a communication function configured to communicate the action to a control system of the roll-fed packaging machine; a third decision function configured to determine a performance index of the machine learning model based on the quality measurement of the package and previously evaluated quality measurements of the package; and a first assignment function configured to assign the mode of the roll-fed packaging machine to a relearning mode if the performance index is outside a performance index interval. In response to the roll-fed packaging machine being in the retrain mode, the circuitry is further configured to perform a retrain function configured to collect image data as training data for a machine learning model and retrain the machine learning model using the training data, and a second assignment function configured to assign the mode of the roll-fed packaging machine to an autonomous mode.

[0027] The features described above in the first aspect also apply to this second aspect, where applicable, and to avoid undue repetition, reference is made thereto.

[0028] According to a third aspect, there is provided a food production system comprising: a roll-fed packaging machine configured to produce packages; an image data capture device disposed within the roll-fed packaging machine and configured to capture image data depicting at least a portion of the packages; and a quality control device for quality control of the packages produced by the roll-fed packaging machine according to the second aspect.

[0029] The above-mentioned features of the first and second aspects also apply to the third aspect, where applicable, and to avoid undue repetition, reference is made to the above.

[0030] According to a fourth aspect, there is provided a non-transitory computer-readable storage medium storing one or more programs configured to be executed by one or more processors of a processing system, the one or more programs including instructions for performing a method according to the first aspect.

[0031] The above-mentioned features of the first aspect also apply to the fourth aspect, where applicable, and to avoid undue repetition, reference is made to the above.

[0032] According to a fifth aspect, there is provided a computer program product comprising instructions which, when executed by a computer, cause the computer to perform the steps of the method of the first aspect.

[0033] The above-mentioned features of the first aspect also apply to this fifth aspect, where applicable, and to avoid undue repetition, reference is made to the above.

[0034] Further scope of applicability of the present disclosure will become apparent from the detailed description set forth below. However, it should be understood that the detailed description, while illustrating preferred variations of the inventive concepts, is given by way of example only, since various changes and modifications within the scope of the inventive concepts will become apparent to those skilled in the art from this detailed description.

[0035] Therefore, it should be understood that the inventive concept is not limited to the particular steps of the described method or components of the described system, as such methods and systems may vary. It should also be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting. It should be noted that, as used in the appended claims, the articles "a," "an," "the," and "said" mean that there are one or more elements, unless the context clearly dictates otherwise. Thus, for example, the phrase "a device" or "the device" may include multiple devices, etc. Furthermore, the use of words such as "comprising," "including," and "containing" does not exclude other elements or steps.

[0036] Other aspects of the inventive concept will now be described in more detail with reference to the accompanying drawings which show variations of the inventive concept, and which should not be construed as limiting the invention to the particular variations, but are used to explain and understand the inventive concept.

[0037] As shown in the figures, the sizes of layers and regions are exaggerated for illustrative purposes and, therefore, are provided to illustrate the general structure of variations of the inventive concepts. Like reference numerals refer to like elements throughout. [Brief explanation of the drawings]

[0038] [Figure 1] 1 is a flow chart illustrating steps in a method for quality control of packages produced on a roll-fed packaging machine. [Figure 2] FIG. 2 is a diagram illustrating a quality control device. [Figure 3] FIG. 1 is a diagram illustrating an example of a food production system including a quality control device. DETAILED DESCRIPTION OF THE INVENTION

[0039] The present invention will now be described in more detail with reference to the accompanying drawings, in which preferred variations of the present invention are shown. However, the present invention may be embodied in many different forms and should not be construed as limited to the variations set forth herein; rather, these variations are provided for thoroughness and completeness and will fully convey the scope of the present invention to those skilled in the art.

[0040] It will also be understood that where the present disclosure is described in terms of a method, the present disclosure may be embodied in an apparatus or device that includes one or more processors and one or more memories coupled to the one or more processors, the memories being loaded with computer code to implement the method. For example, the one or more memories may, in some embodiments, store one or more computer programs that, when executed by the one or more processors, perform the steps, services, or functions disclosed herein.

[0041] It should also be understood that the terms used herein are for the purpose of describing particular embodiments only and are not intended to be limiting. It should be noted that, when used in the appended claims, the articles "a," "an," "the," and "said" are intended to mean that there are one or more elements, unless the context clearly dictates otherwise. Thus, for example, the terms "a unit" or "the unit" may refer to multiple units, depending on the context. Furthermore, the terms "comprising," "including," and "containing" do not exclude other elements or steps. It should be emphasized that, as used herein, the term "comprises / comprising" is used to specify the presence of stated features, integers, steps, or components. This does not exclude the presence or addition of one or more other features, integers, steps, components, or groups thereof. The term "and / or" is to be construed as either an alternative or to mean "both." The term "obtain" is to be interpreted broadly herein and includes receiving, retrieving, collecting, obtaining, and the like.

[0042] It should also be understood that although terms such as "first," "second," etc. may be used herein to describe various elements or features, these elements should not be limited by these terms. These terms are merely used to distinguish one element from another. For example, a first function could be referred to as a second function, and similarly, a second function could be referred to as a first function, without departing from the scope of the embodiments. A first function and a second function are both functions, but are not the same function.

[0043] Hereinafter, a quality control method, a quality control device, and a food production system for packages produced by a roll-fed packaging machine will be described with reference to FIGS.

[0044] FIG. 1 is a flowchart illustrating steps in a method 100 for quality control of packages produced on a roll-fed packaging machine. The packages are carton-based packages containing food products. The packages may include a carton layer and at least one plastic layer. When producing packages on a roll-fed packaging machine, a tube filled with food products can be formed. Packages can be continuously formed from the bottom end of the tube. The advantage of having such a tube is that the formed packages are interconnected, and therefore, quality issues detected in previous packages may be relevant and should be taken into account when evaluating the current package. The same situation may not occur with blank-fed systems that form packages based on blanks (i.e., pre-fabricated sleeves of packaging material) or PET bottling systems that use preforms. Although the link between successive packages may be stronger on a roll-fed packaging machine, the principles described herein may also be used on blank-fed packaging machines or PET bottling systems. In these two systems, the link between successive packages may be related to changes in machine settings or to the continuous deterioration of machine components.

[0045] 1 illustrates a preferred variation of method 100. Additionally, many optional steps of method 100 are shown in dashed lines, forming many alternative embodiments.

[0046] Method 100 includes a number of iterative steps. Stated differently, method 100 may be repeated for multiple subsequent packages produced by a roll-fed packaging machine. By recursively executing method 100, the performance of a machine learning model (described further below) may be monitored to detect whether performance begins to degrade. The different steps are described in more detail below with reference to FIG. 1. Although illustrated in a particular order, the steps of method 100 may be performed in any suitable order, in parallel, and multiple times.

[0047] Image data depicting at least a portion of a package is acquired from an image data capture device (S102). The package, as used herein, refers to a package being produced or produced by a roll-fed packaging machine and currently being evaluated for quality. Thus, the package can be evaluated at any stage during the manufacturing process. Thus, the image data may depict the package in an unformed state (e.g., as a web of packaging material), a partially formed state (e.g., as a tube), or a fully formed state (e.g., as a finished package).

[0048] The image data can depict a portion of the package relevant to the quality control performed on the package. As an example, the image data depicts the longitudinal seal of the package. Therefore, the quality measurement (see below) determined later can be a quality measurement of the longitudinal seal. In roll-fed packaging machines, such as the Tetra Brik® system sold by Tetra Pak®, the longitudinal seal is formed by joining two ends of the packaging material to form a tube. The tube can then be filled with food and closed with top and bottom transverse seals. More specifically, the longitudinal seal can be formed by forming a tube of a web of packaging material such that a first longitudinal edge of the web overlaps a second longitudinal edge of the web, attaching the first and second longitudinal edges, and applying a protective strip to the overlap between the first and second longitudinal edges inside the tube. The longitudinal seal can be an important part of ensuring the food safety of the package. Image data depicting the longitudinal seal can be captured from inside the tube by an image data capture device. In other words, the image data capture device may be located within the tube, as further described in connection with FIG.

[0049] However, the image data may also describe other portions of the package, such as the sidewalls of the package, the top surface of the package, the bottom surface of the package, or the web of packaging material. The image data may also describe the entire package.

[0050] The image data may include a single frame of the package. Alternatively, the image data may include a series of frames of the package. In other words, the image data may include multiple subsequent frames. As an example, the image data may include 20 to 40 frames of the package. More specifically, the image data may include 32 subsequent frames of the package.

[0051] The image data capture device may be a camera configured to capture an image of the package. Alternatively, the image capture device may include multiple cameras, each configured to capture an image of a different portion of the package. In this case, the resulting image data may include multiple images of the package captured by the multiple cameras. The multiple images may be combined to form a single image before being used in subsequent steps of method 100.

[0052] Subsequently, in response to the roll-fed packaging machine being in an autonomous mode, a quality measurement of the package is determined by inputting the image data into a machine learning model (S106). The machine learning model may be a neural network, such as a convolutional neural network. The machine learning model may be trained to output a predicted quality measurement based on the input image data. The machine learning model may be trained using supervised learning. In other words, the machine learning model may be trained based on image data having known associated quality measurements.

[0053] Alternatively, the quality measure may be a range of continuous values, such as a range from 0 to 1. Alternatively, the quality measure may be a set of discrete values. For package defects, the value of the quality measure may indicate the severity of the detected defect. Alternatively, the quality measure may be a binary value, such as "OK" or "Not OK." The quality measure may further include information indicating what type of defect, if any, the package has. Information indicating the type of defect, for example, that the longitudinal seal is not properly formed or that the packaging material is damaged, both of which can result in a non-sterile package. The quality measure may also indicate a confidence score of the output from the machine learning model, for example, how confident the machine learning model is that the detected defect is actually a defect.

[0054] Method 100 may further include determining modified image data (S104) by inputting the image data into an image pre-processing module prior to determining the quality measure (S106). Determining the quality measure of the package (S106) may then be performed by inputting the modified image data into a machine learning model. In other words, the pre-processing module may perform one or more pre-processing steps on the image data to form the modified image data. The one or more pre-processing steps may include, but are not limited to, reshaping, normalizing, and / or cropping the image data.

[0055] Further, based on the determined quality measurements, an action to be taken is determined (S108). This action may be one of: (i) rejecting or accepting the package; (ii) sending a warning signal to an operator of the packaging machine; (iii) stopping the packaging machine; and (iv) adjusting the settings of the packaging machine. It should be noted that more than one action may be determined to be performed. An example is to reject the package and adjust the settings of the packaging machine.

[0056] The action to be taken may, for example, be by comparing the quality measurements to one or more thresholds to determine whether the package meets quality control and what action should be taken. As a non-limiting example, if the quality measurements exceed a first threshold (e.g., indicating a minor defect), the action may be to send a warning signal to an operator. If the quality measurements exceed a second threshold (e.g., indicating a defect that may compromise food safety), the action may be to reject the package. Finally, if the quality measurements exceed a third threshold (e.g., indicating a major defect), the action may be to stop the packaging machine. As noted above, information indicating the type of defect may also be used in determining the action to be taken. For example, if it is determined that the longitudinal seal is not properly formed, the action to be taken may be to adjust the settings of the packaging machine.

[0057] This action is then communicated to the control system of the roll-fed wrapper (S110). The control system of the roll-fed wrapper may then execute the action. If the action is to accept or reject the package, the action may be communicated to a package rejection unit located downstream of the roll-fed wrapper. Alternatively, the control system of the roll-fed wrapper may instruct the package rejection unit to discard the package.

[0058] The image data may be associated with identification data of the package. The identification data may be printed on the package, for example, as a two-dimensional code (such as a barcode or QR code) or as a unique ID number. The identification data may be depicted in the image data of the package and extracted therefrom. Alternatively, the identification data may be read by a scanner or the like. Alternatively, the identification data may be written on a tag such as an RFID or NFC tag and acquired by an RFID or NFC tag reader. Communicating the action to the control system (S110) may further include communicating the identification data to the control system. Thus, the control system can know which package the action is associated with.

[0059] Further, a performance metric for the machine learning model is determined based on the quality measurements of the package and the quality measurements of previously evaluated packages (S112). The term "previously evaluated packages" herein refers to either packages the machine learning model has seen during training or packages the machine learning model has evaluated on the roll-fed packaging machine before the package being currently evaluated. The performance metric should be understood as a measure of how well the machine learning model performs on the image data. The performance metric may be determined based on the number of false positives and false negatives of previously evaluated packages (S112). Alternatively, the performance metric may be based on the rate of false positives and the rate of false negatives in a batch of previously evaluated packages. The performance metric may be precision and / or recall based on the number of false positives and the number of false negatives. As another example, the performance metric may be the so-called F1 score. Using the F1 score as a performance metric is advantageous when there is an imbalance in the dataset, i.e., typically when there are a large number of "good" packages and a small number of "bad" packages.

[0060] The performance metric may indicate the distribution of the output quality measures (i.e., inference scores). More specifically, if the image data includes multiple subsequent frames of a package, the distribution of inference scores corresponding to the number of subsequent frames may be determined as the performance metric. This distribution may be compared to the distribution of inference scores corresponding to the training set of data used in training the machine learning model. If the distribution of the image data deviates from the distribution of the training data set by more than a certain amount, this may indicate that the machine learning model needs to be retrained. A change in the distribution of inference scores may be due to the image data depicting a new type of package defect that the machine learning model did not detect during training. Alternatively, or in combination with the above, the performance metric may indicate how the quality measures change over time (i.e., with the series of packages evaluated). A sudden increase or decrease in the quality measures over time may indicate that the machine learning model needs to be retrained.

[0061] The performance index may include multiple sub-indicators. The sub-indicators may relate to different quality aspects of the package. For example, the sub-indicators may relate to longitudinal sealability of the package, transverse sealability of the package, decoration of the package, etc. In response to a sub-indicator being outside a corresponding sub-indicator interval, the method 100 may further include generating a task associated with the sub-indicator, the task including image data and settings of the roll-fed packaging machine, and sending the task to a user group associated with the sub-indicator. The user group may be a team of experts on the corresponding quality aspect. In response to the task, the method 100 may further include receiving updated settings of the roll-fed packaging machine from the user group. In this manner, the updated settings may be determined by the user group to mitigate the reason for the quality defect. The method 100 may further include adjusting the settings of the roll-fed packaging machine according to the updated settings.

[0062] Furthermore, if the performance index is outside the performance index interval, the mode of the roll-fed packaging machine is assigned to a re-learning mode (S114). The performance index interval can be interpreted as an operating window in which the machine learning model is allowed to perform quality control. By monitoring whether the performance index of the machine learning model is within the performance index interval, it is possible to detect when quality control is no longer reliable.

[0063] Alternatively, the roll-fed packaging machine may be assigned to a re-learn mode if the performance index deteriorates as a function of the number of consecutively evaluated packages (i.e., the number of consecutive iterations) (S114).

[0064] In response to the roll-fed packaging machine being in the re-learning mode, image data is collected as training data for the machine learning model (S116). Image data that caused the machine learning model's performance index to fall outside the performance index interval is particularly useful as training data because it may indicate that the machine learning model is composed of new cases that it has not seen before. The roll-fed packaging machine may remain in the re-learning mode depending on the number of iterations of method 100, i.e., the number of packages produced. While in the re-learning mode, image data of every package that passes by the image data capture device may be collected as training data.

[0065] Further, the machine learning model is retrained using the training data (S118). The machine learning model may be retrained until the machine learning model reaches a certain level of performance (e.g., a certain level of precision, recall, and / or F1 score). As one example, the machine learning model may be retrained until it reaches a precision of 95% or higher and a recall of 80% or higher. As another example, the machine learning model may be retrained until it reaches an F1 score above a certain threshold. As another example, the machine learning model may be retrained until the distribution of its inference scores is within a certain amount of the distribution of inference scores of the original training set used to initially train the machine learning model.

[0066] Furthermore, the mode of the roll-fed packaging machine is assigned to the autonomous mode (S120). Once the retraining of the machine learning model is complete, the roll-fed packaging machine is assigned to the autonomous mode.

[0067] While the roll-fed packaging machine is in the re-learn mode and during a time to accept or reject a package, the method 100 may further include instructing a secondary reject system (S122) to perform a quality assessment of the package. The secondary reject system may be, for example, a manual quality control performed by a user. Even when the roll-fed packaging machine is in the re-learn mode, image data depicting at least a portion of the package may be acquired and communicated to the secondary reject system.

[0068] Optionally, method 100 may further include adjusting (S124) settings of the roll-fed packaging machine based on the determined quality measurements. If the longitudinal seal of the package is evaluated, adjusting (S124) settings of the roll-fed packaging machine is preferably performed by increasing or decreasing the heating temperature and / or increasing or decreasing the amount of electricity during formation of the longitudinal seal.

[0069] FIG. 2 is a schematic diagram of a quality control device 200 consistent with the concepts of the present invention. The quality control device 200 is configured to perform quality control of packages produced by a roll-fed packaging machine. In particular, the quality control device 200 is configured to perform the method 100 described above in connection with FIG. 1. Any features, aspects, or advantages described above in connection with the method 100 also apply to the quality control device 200 described below, and vice versa. The quality control device 200 may be integrated with the roll-fed packaging machine. More specifically, the quality control device 200 may be part of the control system of the roll-fed packaging machine. Alternatively, the quality control device 200 may be external to the roll-fed packaging machine. In that case, the quality control device 200 may be communicatively connected to the roll-fed packaging machine.

[0070] The quality control device 200 includes a circuit 202. The circuit 202 may physically comprise one single circuit. Alternatively, the circuit 202 may be distributed across multiple circuits. As shown in the example of FIG. 2, the quality control device 200 may further include a transceiver 206 and a memory 208. The circuit 202 is communicatively connected to the transceiver 206 and the memory 208. The circuit 202 may include a data bus, and the circuit 202 may communicate with the transceiver 206 and / or the memory 208 via the data bus (not shown in FIG. 2).

[0071] The circuitry 202 may be configured to perform overall control of the functionality and operation of the quality control device 200. The circuitry 202 may include a processor 204, such as a central processing unit (CPU), microcontroller, or microprocessor. The processor 204 may be configured to execute program code stored in a memory 208 to perform the functional operations of the quality control device 200.

[0072] The transceiver 206 may be configured to enable the quality control device 200 to communicate with other devices or equipment. For example, the quality control device 200 may be communicatively coupled to a control system of a roll-fed packaging machine via the transceiver 206. In another example, the quality control device 200 may be communicatively coupled to an image data capture device, for example, via the transceiver 206. The transceiver 206 may both transmit data to and receive data from the quality control device 200. The transceiver 206 may communicate via wired or wireless communication protocols (e.g., Bluetooth, Wi-Fi, cellular communication, etc.).

[0073] The memory 208 may be a non-transitory computer-readable storage medium. The memory 208 may be one or more of a buffer, flash memory, a hard drive, removable media, volatile memory, non-volatile memory, random access memory (RAM), or other suitable devices. In a typical configuration, the memory 208 may include non-volatile memory for long-term data storage and volatile memory that serves as the system memory of the quality control device 200. The memory 208 may exchange data with the circuit 202 via a data bus. Associated control lines and address buses may also exist between the memory 208 and the circuit 202.

[0074] Although not explicitly shown in FIG. 2 , the quality control device 200 may include one or more input devices, such as a keyboard, a mouse, or a touchscreen. User input may be used, for example, when collecting image data as training data, as described further below. A user may provide user input as labels for the image data. Thus, user input may serve as a method for improving performance or as feedback to the quality control device. The quality control device 200 may further include a display for providing output, such as captured image data, to a user.

[0075] The functions and operations of the quality control device 200 may be implemented in the form of executable logic routines (e.g., lines of code, software programs, etc.) stored on a non-transitory computer-readable storage medium (e.g., memory 208) of the quality control device 200 and executed by the circuit 202 (e.g., using the processor 204). In other words, if the circuit 202 is configured to perform a particular operation or function, the processor 204 of the circuit 202 may be configured to execute program code portions stored on the memory 208, the stored program code portions corresponding to the particular operation or function. Furthermore, the functional operations of the circuit 202 may be independent software applications or may form part of a software application that performs additional tasks related to the circuit 202. The described functional operations can be considered as methods that a corresponding device is configured to perform, such as the method 100 described above in connection with FIG. 1. The described functional operations may also be implemented in software, or such functions may be performed via dedicated hardware or firmware, or some combination of one or more of hardware, firmware, and software. The following operations may be performed by quality control device 200 and stored as functions on a non-transitory computer-readable storage medium: acquisition function 210, first determination function 212, second determination function 214, communication function 216, third determination function 218, first allocation function 220, relearning function 222, and second allocation function 224. First, second, and third determination functions 212, 214, and 218 may be implemented as a common determination function. Similarly, first and second allocation functions 220 and 224 may be implemented as a common allocation function. First determination function 212, second determination function 214, communication function 216, third determination function 218, and first allocation function 220 are executed by circuitry 202 in response to the roll-fed packaging machine being in autonomous mode. In other words, the functions may be repeatedly executed to autonomously evaluate package quality as long as the roll-fed packaging machine is in autonomous mode.A relearn function 222 and a second allocation function 224 are performed by the circuit 202 in response to the roll-fed packaging machine being in the relearn mode.

[0076] The acquisition function 210, the first determination function 212, the second determination function 214, and the communication function 216 may be considered as a quality assessment module of the quality control device 200. The quality control module may be tasked with evaluating different quality aspects of the package and, in response, determining what action to take. The third determination function 218, the first allocation function 220, the retraining function 222, and the second allocation function 224 may be considered as a drift module of the quality control device 200. The drift module may be tasked with monitoring the performance and operating conditions of the machine learning model to detect when conditions change (e.g., when the performance of the machine learning model deteriorates or when new types of input data begin to appear). In this way, the proposed quality control device 200 provides a robust and effective way to perform quality assessment of packages in a roll-fed packaging machine. The same is true for the method 100 and corresponding method steps.

[0077] The circuitry 202 may be further configured to perform a directing function 226 , a coordinating function 228 , and / or a task generating function 230 .

[0078] The acquisition function 210 is configured to acquire image data depicting at least a portion of the package from an image data capture device. The acquisition function may acquire the image data in response to sending an instruction signal to the image data capture device to capture the image data. Alternatively, the image data capture device may be included within the quality control device 200. As described above in connection with FIG. 1 , the image data may depict the longitudinal seal of the package. Thus, the determined quality measurement may be a quality measurement of the longitudinal seal.

[0079] The first determination function 212 is configured to determine a quality measurement value of the package by inputting the image data into a machine learning model, such as a neural network. The machine learning model may be provided in a machine learning module of the quality control device 200. Alternatively, the machine learning model may be provided on a remote server, such as a cloud.

[0080] The second decision function 214 is configured to determine an action to be taken based on the quality measurement, which can be one of: (i) rejecting or accepting the package, (ii) sending a warning signal to an operator of the packaging machine, (iii) stopping the packaging machine, and (iv) adjusting the settings of the packaging machine.

[0081] The communication function 216 is configured to communicate the action to a control system of the roll-fed packaging machine. The communication function 216 may be configured to communicate the action to a package rejection unit located downstream of the roll-fed packaging machine. The communication function 216 may further be configured to communicate package identification data associated with the image data along with the action.

[0082] The third determination function 218 is configured to determine a performance metric for the machine learning model based on the quality metrics of the package and the quality metrics of previously evaluated packages. The performance metric may be determined as described above in connection with FIG. 1.

[0083] The first assigning function 220 is configured to assign the mode of the roll-fed packaging machine to a relearning mode if the performance index is outside the performance index interval. Assigning the roll-fed packaging machine to a relearning mode is further described above in connection with FIG.

[0084] The re-learning function 222 is configured to collect image data as training data for the machine learning model and re-learn the machine learning model using the training data.

[0085] The second assigning function 224 is configured to assign the mode of the roll-fed packaging machine to the autonomous mode. As described above in connection with FIG. 1, the roll-fed packaging machine may be assigned to the autonomous mode once the machine learning model has been retrained.

[0086] The prompting function 226 may be configured to prompt the secondary reject system to perform a quality assessment of the package. The prompting function 226 may be executed in response to the roll-fed packaging machine being in a relearn mode.

[0087] The adjustment function 228 may be configured to adjust settings of the roll-fed packaging machine based on the determined quality measurements. For example, if the longitudinal seal is evaluated, the settings may be adjusted by increasing or decreasing the heating temperature and / or increasing or decreasing the amount of power during the formation of the longitudinal seal.

[0088] A performance index may include multiple sub-indicators. The task generation function 230 may be configured to generate a task associated with the sub-indicator in response to the sub-indicator being outside a corresponding sub-indicator interval and send the task to a user group associated with the sub-indicator. The task may include image data and settings of the roll-fed packaging machine to be used in producing the associated package. The task generation function 230 may be further configured to receive updated settings of the roll-fed packaging machine from the user group in response to the task. The task generation function 230 may be further configured to adjust the settings of the roll-fed packaging machine according to the updated settings.

[0089] It should be noted that the features, aspects and advantages of the method 100 described above in relation to Figure 1 are also applicable to the quality control device 200 described herein. To avoid undue duplication, reference is made to the above.

[0090] 3 shows, by way of example, a schematic illustration of a food production system 300 in accordance with the concepts of the present invention. Dashed lines indicate optional parts. Food production system 300 may be considered any type of food processing line for filling and packaging food products.

[0091] The food production system 300 includes a roll-fed packaging or filling machine 304. The packaging machine 304 is a roll-fed packaging machine used to produce food packages. More specifically, the packaging machine 304 may be used to package liquid foods in carton-based packages. This type of packaging machine was introduced by Tetra Pak as early as the 1940s and is today a well-known method for safely and cost-effectively packaging milk and other liquid foods. This general approach can also be used for non-liquid foods, such as potato chips. Furthermore, this general approach can also be used for carton-free packaging, such as PET bottles. Thus, the packaging machine 304 may be a PET bottling system.

[0092] Packaging material is often printed and prepared at a packaging production center, also known as a converting plant, and shipped to a site, e.g., a dairy, where a packaging machine 304 is located. Typically, the packaging material is wound onto a reel 302 before being transported. After arriving at the site, the reel 302 is either placed on the packaging machine 304 or fed into the packaging machine 304 as shown. Alternatively, the packaging material is fed to the packaging machine 304 as a blank.

[0093] During production, a web 302 of packaging material may be fed from a reel 302 to and passed through a packaging machine 304. Although not shown in FIG. 3 , the packaging material may be passed through a sterilization device, such as a hydrogen peroxide bath or a low-voltage electron beam (LVEB) station, to ensure that the web 303 is free of unwanted microorganisms. A longitudinal seal may be formed from the web 302 to form a tube 308 before the food product is delivered. The food product may be delivered to the tube 308 through a fill pipe (not shown), and a valve (not shown) may be used to regulate the flow through the fill pipe. The lower end of the tube 308 may be fed into a folding device (not shown), where a transverse seal is formed and the tube is folded and cut along fold lines (also called lines of weakness) to form a package 310.

[0094] The food production system 300 further includes image data capture devices 306a-c. The image data capture devices 306a-c are configured to capture image data depicting at least a portion of a package produced in the packaging machine 304. The image data capture devices 306a-c may be located within the packaging machine 304. However, as will be readily understood by those skilled in the art, the image data capture devices may also be located external to the packaging machine 304. In the illustrated example, first, second, and third image data capture devices 306a, 306b, and 306c are shown. These should be considered merely as non-limiting examples to illustrate that image data depicting a package may be captured at different stages of the manufacturing process. For example, the first image data capture device 306a illustrates an example in which image data is captured in the form of the web 303 of packaging material, i.e., the package before it is formed into the tube 308. Furthermore, the second image data capture device 306b illustrates an example in which image data is captured from the interior of the tube 308, i.e., from the interior of the longitudinal seal. Thus, the image data capture device may be located inside the tube 308 (as indicated by the dashed-dot pattern). Also, although not shown, a product fill pipe may be provided inside the tube, with the second image data capture device 306b positioned on the product fill pipe. Furthermore, the third image data capture device 306c illustrates an example in which image data of a completed package 310 (i.e., a filled, sealed, and folded package) is captured. The food production system 300 may include one or more image data capture devices (e.g., any one of the first, second, and third image data capture devices 306a-c) located at different locations on the packaging machine 304. The image data capture devices 306a-c may be any camera or other image capture device suitable for generating image data that can be used to determine the quality measurements described herein. The image data capture devices 306a-c may include a light source adapted to illuminate the portion of the package from which image data is captured. The light source may use visible light, infrared (IR) light, ultraviolet (UV) light, or any combination thereof. The light source may provide visible light at a wavelength of approximately 380 to 740 nanometers (nm).The light source may provide IR light with a wavelength of about 740 nm to 1 mm. The light source may provide UV light with a wavelength of about 1 to 380 nm.

[0095] The food production system 300 further includes a quality control device 200 for quality control of packages produced by the roll-fed packaging machine 304. The quality control device 200 is the quality control device 200 described above in connection with FIG. 2. As shown, the quality control device 200 may be a separate unit from the packaging machine 304 or may be communicatively connected thereto. Alternatively, the quality control device 200 may be an integral unit of the packaging machine 304, as part of its control system or communicatively connected thereto. As described above in connection with FIGS. 1 and 2, the quality control device 200 can acquire image data depicting at least a portion of a package. Here, the quality control device 200 receives image data from the packaging machine 304 or the image data capture devices 306a-c via any suitable communication protocol. Optionally, the quality control device 200 may receive package identification data (ID) associated with the image data. In return, the quality control device 200 transmits a determined action to be performed. Optionally, the quality control device 200 sends updated settings to the packaging machine 304. The quality control device 200 may also track what mode the roll-fed packaging machine 304 is in (i.e., autonomous mode or re-learn mode). In other words, the mode may be associated with the quality control device 200. That is, if the quality control device 200 is in autonomous mode, quality control can be performed automatically. If the quality control device 200 is in re-learn mode, the quality control device 200 re-trains the machine learning model, and quality control of the packages is performed by other means (e.g., a second reject system or manual quality control).

[0096] The food production system 300 may further include a package rejection unit 312. The package rejection unit 312 may be configured to store or discard the package based on the quality evaluation. The quality control device 200 may be communicatively connected to the package rejection unit 312. The quality control device 200 sends an action to reject or accept the package to the package rejection unit 312. Package identification data related to the action is also sent. Based on the action, the package rejection unit 312 discards or stores the package. Rejected packages 314 may be collected by a recycling unit 316 to recycle the product of the discarded package. Accepted packages 320 may be sent to an end station 318 of the food production system 300, where they may be collected and transported (e.g., to a warehouse). Rejected packages 314 may be evaluated by a user (e.g., an operator) to determine whether they were correctly rejected. This information (i.e., if the package is a false negative FN (or true negative TN)) may be sent to the quality control device 200. Similarly, one or more of the accepted packages 320 may be evaluated by a user to determine whether they were correctly accepted. This information (i.e., if the package is a false positive FP (or true positive TP)) may be sent to the quality control device 200. Information regarding the false negatives and / or false positives may be used by the quality control device 200 in determining performance metrics for the machine learning model, as described above in connection with Figures 1 and 2. User feedback may also be used in labeling image data collected for retraining the machine learning model.

[0097] Moreover, from a study of the drawings, the disclosure, and the appended claims, one skilled in the art can appreciate variations to the disclosed modifications and practice the claimed invention.

Claims

1. A method (100) for quality control of packages produced on a roll-fed packaging machine, the method (100) comprising a series of repeated steps, the series of repeated steps comprising: acquiring image data depicting at least a portion of a package from an image data acquisition device (S102); In response to the roll-fed packaging machine being in an autonomous mode, determining a quality measurement of the package by inputting the image data into a machine learning model (S106); determining an action to be performed based on the determined quality measures (S108); communicating the action to a control system of the roll-fed packaging machine (S110); determining a performance metric for the machine learning model based on the quality measure of the package and quality measures of previously evaluated packages (S112); and assigning the mode of the roll-fed packaging machine to a re-learn mode if the performance index is outside a performance index interval (S114); Including, in response to the roll-fed packaging machine being in a relearn mode; Collecting image data as training data for the machine learning model (S116); Retraining the machine learning model using the training data (S118); and assigning the mode of the roll-fed packaging machine to an autonomous mode (S120); Including, Method (100).

2. The package includes a carton layer and at least one plastic layer. The method (100) of claim 1.

3. the image data depicts a longitudinal seal of the package, and the determined quality measure is a quality measure of the longitudinal seal.

3. The method (100) of claim 1 or 2.

4. a longitudinal seal is formed by forming a tube of the web of packaging material such that a first longitudinal edge of the web overlaps a second longitudinal edge of the web, attaching the first and second longitudinal edges, and applying a protective strip to the overlap between the first and second longitudinal edges inside the tube; image data of the longitudinal seal is acquired from inside the tube by the image data capture device; The method (100) of claim 3.

5. The action is one of: (i) rejecting or accepting the package; (ii) sending a warning signal to an operator of the packaging machine; (iii) stopping the packaging machine; and (iv) adjusting settings of the packaging machine. The method (100) according to any one of claims 1 to 4.

6. the action is to accept or reject the package, and communicating the action (S110) includes communicating the action to a package rejection unit located downstream of the packaging machine. The method (100) according to any one of claims 1 to 5.

7. When the action is to accept or reject the package and the roll-fed packaging machine is in the relearn mode, the method (100) further comprises: instructing a secondary rejection system to perform a quality assessment of the package (S122); The method (100) according to any one of claims 1 to 6.

8. The performance index is determined based on the number of false positives and false negatives of previously evaluated packages (S112). The method (100) according to any one of claims 1 to 7.

9. The method (100) further comprises adjusting (S124) settings of the roll-fed packaging machine based on the determined quality measurements, preferably by increasing or decreasing the heating temperature and / or increasing or decreasing the amount of electricity during formation of the longitudinal seal. The method (100) according to any one of claims 1 to 8.

10. the image data is associated with identification data of the package; Communicating the action to the control system (S110) includes communicating the identification data to the control system. The method (100) according to any one of claims 1 to 9.

11. assigning the roll-fed packaging machine to a re-learn mode if the performance index deteriorates in response to the number of consecutively evaluated packages (114); The method (100) according to any one of claims 1 to 10.

12. The performance index includes a plurality of sub-indexes, and the method (100) further comprises: In response to the sub-index being outside a corresponding sub-index interval, generating a task associated with the sub-indicator, the task including the image data and settings of the roll-fed packaging machine; sending a task to a user group associated with the sub-indicator; receiving updated settings for the roll-fed packaging machine from the user group in response to the task; adjusting settings of the roll-fed packaging machine according to the updated settings; Including, The method (100) according to any one of claims 1 to 11.

13. A quality control device (200) for quality control of packages produced by a roll-fed packaging machine, said quality control device (200) comprising a circuit (202) comprising: an acquisition function (210) configured to acquire image data indicative of at least a portion of the package from an image data capture device; In response to the roll-fed packaging machine being in an autonomous mode, the circuit (202) a first determination function (212) configured to determine a quality measurement of the package by inputting the image data into a machine learning model, such as a neural network; a second decision function (214) configured to decide an action to be taken based on said quality measures; a communication function (216) configured to communicate the action to a control system of the roll-fed packaging machine; a third determination function (218) configured to determine a performance indicator for the machine learning model based on the quality measure of the package and quality measures of previously evaluated packages; a first assigning function (220) configured to assign a mode of the roll-fed packaging machine to a relearn mode if the performance index is outside a performance index interval; further configured to perform In response to the roll-fed packaging machine being in a relearn mode, the circuit (202) a re-learning function (222) configured to collect the image data as training data for the machine learning model and re-learn the machine learning model using the training data; a second assigning function (224) configured to assign a mode of the roll-fed packaging machine to an autonomous mode; further configured to perform Quality control device (200).

14. A food production system (300), comprising: a roll-fed packaging machine (304) configured to produce packages (310); an image data capture device (306a-c) disposed within the roll-fed packaging machine (304) for capturing image data depicting at least a portion of the package; A quality control device (200) according to claim 13 for quality control of packages produced by the roll-fed packaging machine; A food production system (300) comprising:

15. 13. A non-transitory computer-readable medium having stored thereon one or more programs configured to be executed by one or more processors of a processing system, the one or more programs comprising instructions for performing the method (100) of any one of claims 1 to 12.

Images