Implementation of web tension adjustment in food packaging systems based on reinforcement learning

Abstract

A method and apparatus, including a computer program product, are described for controlling web tension in a food packing machine (100) consisting of multiple subsystems. One or more local variable values ​​(116) are received, representing measurements by the food packing machine (100) of one or more physical parameters for a web tension adjustment subsystem (200). One or more remote variable values ​​(204) are received, representing measurements by the food packing machine (100) of one or more physical parameters for the one or more remote subsystems. One or more control parameter values ​​for the web tension adjustment subsystem (200) are determined by processing the remote and local variable values ​​(204, 116) using a reinforcement learning model (206) and a local control model (210). One or more control parameters of the web tension adjustment subsystem (200) are adjusted according to the determined control parameter values.

Description

[Technical Field]

[0001] FIELD OF THE INVENTION The present invention relates to food packaging systems, and more particularly to adjusting web tension in food packaging systems. [Background technology]

[0002] Automation control systems are currently used in a variety of manufacturing and processing sites and are continually becoming more complex. A common approach to managing this complexity is to divide the system into subsystems and develop appropriate control mechanisms for each subsystem. However, this approach does not always yield an optimal solution for the system as a whole.

[0003] As systems become more complex and the number of influencing factors increases, capturing these influencing factors from various sources becomes increasingly difficult. This complexity is further compounded when the relationships between influencing factors, control variables, and the system itself are nonlinear and difficult to model.

[0004] There are two main perspectives regarding abstraction levels in industrial control: low-level control and high-level control. Low-level control refers to the management of individual automation components (e.g., actuators, servo motors, heaters, and many other devices). High-level control can increase the level of abstraction from the subsystem level to the system level, and even to the orchestration of an entire plant with multiple systems and subsystems that need to work together.

[0005] As an example, food processing and packaging equipment typically includes several subsystems, such as a filling system, a sterilization system, a package folding system, etc. Each subsystem includes many different elements (e.g., pneumatic actuators, servo motors, DC motors, AC motors, sensors, other actuators, etc.). These individual elements are typically controlled by low-level local control systems that utilize traditional control techniques, such as proportional integral derivative (PID) controllers, to control target variables. Feedback loops are used to keep the controller error low relative to the target operating point of the element, system, or subsystem.

[0006] However, PID controllers must be tuned for their application and are typically optimized for a specific operating range and dynamics. They are also poorly suited to adapting to unexpected events or operating conditions outside of their traditional operating range. When these conditions change (e.g., a different operating environment, changes in automation elements, changes in the manufacturing process, etc.), the PID controller parameters must be adjusted and recalibrated. This can be a time-consuming and complex process, requiring significant manual input from experienced personnel, especially when numerous elements and / or subsystems are involved, as is typical with food processing and packaging equipment.

[0007] A filling machine is an example of a composite system that packages liquid, semi-liquid, or pourable foods, such as fruit juice, UHT (ultra-high temperature processed) milk, wine, and tomato sauce, into composite packages made of multilayer composite packaging materials for distribution and sale. A typical example is the parallelepiped pourable food package known as "Tetra Brik Aseptic®," which is made by sealing and folding laminated strips of packaging material. This packaging material has a multilayer structure, with a carton or paper substrate layer covered on both sides with a heat-sealable plastic material such as polyethylene. For long-term, aseptic packaging, the packaging material also includes a layer of oxygen barrier material, e.g., aluminum foil, which overlaps the heat-sealable plastic layer and is then covered with another layer of heat-sealable plastic to form the inner surface of the package that ultimately contacts the food.

[0008] The filler starts with a web of multi-layer composite packaging material (wound from a reel). The web is fed through the filler, which creates a longitudinal seal to form a tube. Liquid food is fed through a pipe into the tube, and the bottom end of the tube is fed into a folding device, which creates a transverse seal and folds the tube along fold lines, also known as lines of weakness, and cuts it to form a composite package filled with the liquid food.

[0009] The machine module or subsystem responsible for forming, transverse sealing and cutting the package is called the "jaw system" and consists of two pairs of jaws whose synchronized movement allows them to pull down the tube of packaging material and completely close the filled package. The jaw system is a critical part of the filling machine, as their coordinated movement is responsible for the correct formation of the package and for feeding the web through the machine.

[0010] The jaw system pulls the packaging material not continuously, but in a "pulsating" fashion as individual packages are created. This intermittent web tension occurs at the end of the packaging machine, while the packaging material is placed on a large roll at the beginning of the machine. Therefore, the intermittent tensioning of the packaging material by the jaw system creates varying forces on the packaging material, some due to the inertia of the packaging material roll and some due to the packaging material itself. This can be problematic because certain sections of the packaging machine require the web to move at a constant speed. Furthermore, excessive web tension can cause package integrity issues. Furthermore, web tension can be affected by other factors that pull on the web, such as the length of the web through the packaging machine from start to finish and the mass of the product being filled into the packaging material tube. The web tensioning system is designed to maintain the packaging material web at a tension appropriate for the filling machine's packaging process. Sagging webs degrade performance, while excessive web tension can result in package defects or damage. Currently, mechanical engineers typically have to rely on trial and error manual settings to achieve the appropriate web tension with web tensioning systems. Furthermore, there is currently no way to consider the effects of factors such as jaw system motion profile, filling conditions (filling flow, product level, etc.), and packaging material properties (thickness and mechanical properties) on web tension.

[0011] Therefore, there is a need for improved techniques for controlling web tension that also take into account various events that occur within a packaging machine and that may affect web tension so that the proper web tension is always maintained. Summary of the Invention [Problem to be solved by the invention]

[0012] It is an object of the present invention to at least partially overcome one or more limitations of the prior art. In particular, it is an object of the present invention to provide a method and system that allows for controlling web tension in a food packer by taking into account measured parameter values ​​of not only the local web tension adjustment subsystem but also other remote subsystems of the food packer. As a result, proper web tension is achieved, allowing for both a faster setup process when the food packer is initially configured and a more reliable production process in which fewer packages need to be discarded.

[0013] In one aspect of the present invention, this is accomplished by a method for controlling web tension in a food packing machine, the food packing machine being comprised of multiple subsystems, the method including: · receiving one or more local variable values ​​indicative of measurements by the food packaging machine of one or more physical parameters related to the web tension adjustment subsystem; receiving one or more remote variable values ​​indicative of one or more remote subsystem measurements of one or more physical parameters by the food packaging machine; · determining one or more control parameter values ​​for a web tension adjustment subsystem by processing the remote variable values ​​and the local variable values ​​using the reinforcement learning model and the local control model; Adjusting one or more control parameters of the web tension adjustment subsystem according to the determined control parameter values.

[0014] Utilizing both local variables and inputs from remote subsystems allows for more precisely controlled web tension and resilient behavior in the food packaging machine when unexpected adverse events occur. The result is less packaging (and food) wasted, making food packaging machine operations more efficient and environmentally friendly. Greater control over the packaging process also reduces manual testing and enables faster time-to-market for new products and configurations. This benefit is further enhanced because control strategies can be learned in a simulation environment, eliminating the need to manually configure the food packaging machine from scratch.

[0015] In one embodiment, the reinforcement learning model is a deep reinforcement learning model that includes a neural network. Deep reinforcement learning is particularly useful when evolving control policies for subsystems that must consider a large number of variables whose internal relationships and effects on the subsystems may be unknown, and offers a more sophisticated approach to determining one or more control parameter values ​​for a local subsystem of a food packaging machine than is possible using traditional reinforcement learning that does not include a neural network.

[0016] In one embodiment, the web tension adjustment subsystem includes two fixed guide rolls and one movable guide roll. The inclusion of the movable guide roll allows the distance between the movable guide roll and each of the fixed guide rolls to be varied, thereby applying a force to the web as the guide rolls move (typically using a servo motor), thus creating a simple method for adjusting the tension in the web without changing other parameters of the web (e.g., speed, etc.).

[0017] In one embodiment, the movable guide roll is located between two fixed guide rolls along the path of the web through the packaging machine and is movable to increase or decrease web tension in response to received control parameter value indications, thereby providing similar benefits as those described above and also distributing web tension changes evenly between the two fixed guide rolls.

[0018] In one embodiment, the neural network is a convolutional neural network, a recurrent neural network, a long short-term memory neural network, or a fully connected neural network, all of which are different types of conventional neural networks that are well known to those skilled in the art and therefore may be more easily integrated into existing food packing machine setups.

[0019] In one embodiment, the one or more local variable values ​​include measurements related to a web tension set point or a current web tension setting system position, and the one or more remote variable values ​​include measurements related to a web movement control variable, a jaw system motion profile, a packaging material property, or a fill state. As achieved by the data-driven approach of various embodiments described herein, these parameters are used to better control the subsystems of the food packer, significantly improving the operation of the tube forming subsystem, and even the overall operation of the food packer.

[0020] Other aspects of the invention include systems and computer programs for controlling web tension in a food packing machine. The features and advantages of these aspects of the invention are substantially the same as those for the method described above.

[0021] Further objects, features, aspects and advantages of the present invention will become apparent from the following detailed description and drawings. [Means for solving the problem]

[0022] Embodiments of the present invention will now be described, by way of example only, with reference to the accompanying drawings, in which: FIG. [Brief explanation of the drawings]

[0023] [Figure 1] 1 is a schematic diagram showing a part of a food packaging machine according to an embodiment. [Figure 2] FIG. 2 is a schematic diagram of a controller in the food packaging machine according to one embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0024] As discussed above, a goal of various embodiments of the present invention is to improve control techniques for devices and systems related to food processing and packaging, particularly with respect to web tension. Having proper web tension is important not only from the perspective of mechanical packaging machine operation, but also from the perspective of functionality, as improper web tension can result in integrity issues for packages formed by the packaging machine. By applying the general concepts of reinforcement learning and / or deep reinforcement learning techniques to control the web tension adjustment system of a food packaging machine, a wider range of factors can be considered compared to what is possible with existing systems, allowing for very precise adjustment of web tension, thereby improving the operation of the food packaging machine and ensuring package integrity, formation, and appearance.

[0025] Reinforcement learning and deep reinforcement learning are both examples of machine learning techniques. In general, reinforcement learning (RL) is characterized by dynamic learning using positive or negative rewards. A system's performance is evaluated against a desired goal. If the goal is reached, a positive reward is given, and if the goal is not reached, a negative reward is given. As positive and negative rewards accumulate over time, the RL model evolves the system's control policy with the goal of maximizing the outcome. Deep reinforcement learning (DRL) is an enhancement of RL, characterized by the combined use of RL and neural networks to evolve the system's control policy.

[0026] In the context of food processing and packaging, RL (i.e., agent-environment interaction) can be used to evolve control policies for food processing and / or packaging machines. DRL (i.e., RL combined with neural networks) is particularly effective when evolving control policies for subsystems, such as the filling subsystem, that must consider a large number of variables whose internal relationships and influences on the subsystem are unknown. Furthermore, it is noteworthy that RL and DRL techniques can also be used to improve existing local control techniques, allowing this data-driven approach to "fill in the gaps" of traditional control techniques. Thus, DRL algorithms can directly (or through other control layers, e.g., by adjusting gains so that traditional PID controllers can operate more efficiently compared to traditional control techniques) control actuators (e.g., servo motors, pneumatic actuators, or other actuators) that prevent tube twisting and keep the tube orientation stable.

[0027] To further illustrate these principles, various embodiments of the present invention will now be described in more detail, using the control of a web tension adjustment subsystem in a food packaging machine as an example, with reference to the accompanying drawings, in which some, but not all, embodiments of the present invention are shown. The present invention may be embodied in many different forms and should not be construed as limited to the embodiments set forth herein.

[0028] As mentioned above, the web tensioning subsystem is a critical part of the food packing machine and its operation must be carefully controlled to allow the web to move smoothly at the desired speed throughout the food packing machine, despite the intermittent operation of the jaw system described above, to ensure proper package integrity, formation, and appearance.

[0029] 1 is a schematic diagram of a subsection of a food packaging machine 100 in which a web 102 of packaging material, preferably including at least one sealable surface thereon, is fed forward over guide rolls 106, 108, 110 in an S-shaped pattern through a web feeder and formed into a tube 112. The overlapping longitudinal side edges of the web 102 are sealed to close the tube along the longitudinal edges. The side edges can be overlapped with their undersides facing each other or with their undersides facing in the same direction. A strip of tape can also be provided along one or both of the longitudinal edges to assist in tube formation.

[0030] After the tube is formed from the web, a filling station (not shown) delivers food to the formed tube. Food, in this context, refers to anything ingested, eaten, and / or drunk by humans or animals, or absorbed by plants, and includes, but is not limited to, liquid, semi-liquid, viscous, dry, powdered, and solid food and beverage products, and water. For the avoidance of doubt, food also includes ingredients for preparing food. Examples of food products include milk, water, and juice. The filled tube is then fed to a jaw system, which transversely seals the filled tube and cuts the sealed tube transversely along its length and within the boundaries of the transversely sealed area to form individual product-filled packages. In this manner, the jaw system exerts a forward pulling force on the web 102. As mentioned above, a corresponding opposing force on the web 102 is exerted as a result of the inertia of the large rolls holding the web 102 during start-up of the packaging machine.

[0031] The right side of FIG. 1 shows a more detailed view of the web tension adjustment subsystem 200 of the food packaging machine 100. In the illustrated embodiment of the web tension adjustment subsystem 200, the intermediate guide roll 108 is vertically movable to adjust the tension on the web 102, thereby decreasing the tension on the web 102 during times when the jaw system is actively pulling the web 102 forward and increasing the tension on the web 102 during times when the jaw system is not actively pulling the web 102. FIG. 1 illustrates only one embodiment of the web tension adjustment subsystem 200; other embodiments may have multiple guide rolls. Similarly, references to vertical movement of the intermediate guide roll 108 are made solely for illustrative purposes. In other embodiments, the guide rolls may be rotated, for example, 90 degrees, compared to those illustrated in FIG. 1, and the movement of the intermediate guide roll 108 may instead be side-to-side. As such, many variations will occur to those skilled in the art. Because of its movement, the intermediate guide roll 108 is also referred to as a pendulum roll.

[0032] 1, movement of the intermediate guide roll 108 is performed in response to a signal from a controller 114. The controller 114 receives input from a sensor 116 in the web tension adjustment subsystem 200, which measures the current tension in the web.

[0033] Additionally, the controller receives input from one or more remote subsystems of the food packer 100 that may experience an event that affects the operation of the web tension adjustment subsystem 200. Some examples of such events may include a splice event (i.e., when the trailing end of packaging web of a used roll of packaging web at the beginning of the food packer is spliced ​​with the leading end of packaging web of a fresh roll of packaging web to create a continuous packaging web, resulting in a section of web having a thickness of two layers rather than a single layer), acceleration, deceleration, or stopping of the web 102 due to jaw movement or other reasons, a change in packaging material, a change in product, package fill, web length, etc.

[0034] These events can be represented by a set of variables whose values ​​indicate various conditions in different subsystems of the food packer. This is shown schematically in Figure 2, which shows how input from local sensor 116 of web tension adjustment subsystem 200 is input to controller 114, along with input values ​​204 from other subsystems of the food packer.

[0035] In one embodiment, some examples of variables representing physical parameters from the local web tension adjustment subsystem 200 include: · Web tension setpoint (i.e., the desired web tension for the particular type of web used in the food packaging machine). The current position of the web tensioning system (e.g., represented by the physical displacement of the movable guide roller from its neutral position).

[0036] In one embodiment, some examples of variables from other subsystems include: Web movement and its associated control variables (e.g. splice detection, package size, speed, etc.) The jaw system's motion profile (e.g., how often and with what force the jaws pull the web) Packaging material characteristics (e.g., hardness of packaging material, presence or absence of closure, package volume, web length, etc.) Filling status (e.g., filling flow, product level, etc.)

[0037] As can be appreciated, these are just a few examples of influencing factors from other subsystems and should not be considered an exhaustive list. However, these factors represent influencing factors that cannot be taken into account in conventional web tension adjustment control systems because it is difficult or impossible to determine how the various possible combinations of these factors should affect the operation of the web tension adjustment subsystem 200.

[0038] According to various embodiments described herein, the controller 114 uses a local control model 210 to process the local subsystem input variables 116, in combination with a reinforcement learning model 206 to process input values ​​from other subsystems, to determine how all measured variables collectively affect the operation of the web tensioning subsystem 200. The local control model 206 is an algorithm implemented by a PID controller. The reinforcement learning model 206 may be a deep reinforcement learning model including one or more neural networks, as described above. In some embodiments, the local subsystem input variables 116 may be processed by the reinforcement learning model 206. In some embodiments, the reinforcement learning model 206 can figure out how different combinations of local and remote variables should affect the web tensioning subsystem and use this insight to improve the local control model 210. Based on the results of this processing and determination, the controller 114 controls the position of the intermediate guide roll 108 to achieve the appropriate web tension and generates a set of output control signals 208 for the local web tensioning subsystem 200.

[0039] Examples of neural networks that can be used in embodiments using deep reinforcement learning models include, for example, convolutional neural networks (CNNs) trained using reinforcement learning and deep reinforcement learning, recurrent neural networks (RNNs) such as long short-term memory (LSTM) neural networks commonly used in the field of deep learning, or fully connected neural networks. LSTM networks are considered particularly useful because, unlike standard feedforward neural networks, LSTMs have feedback connections. This allows LSTMs to process entire sequences of data, not just single data points, which is particularly useful in the context of food packaging machines designed to produce large numbers of packages.

[0040] Thus, if the tension of the web 102 changes, for example, due to changes in the nature of the jaw movement or due to imprecise functioning of one or more mechanical components of the filling machine, the data-driven approach allows the control device 114 to detect such fluctuations in web tension and adjust the position of the intermediate guide roll 108 to ensure proper web tension at all times, thereby avoiding potential damage to the packaging material and ensuring sealing and forming quality. Furthermore, conventional control techniques often require manual calibration for different operational settings. In contrast, this embodiment of the present invention provides a learning environment that allows the controller 114 to learn the optimal control strategy given goals for the web tension adjustment subsystem 200. This can significantly reduce the labor required to set up the packaging machine, thereby shortening the time to market for new packages and products. Furthermore, in some embodiments, outputs from the reinforcement learning model can be used to adjust the gains of a conventional PID controller, allowing the PID controller to operate more efficiently compared to conventional control techniques in which the PID controller relies only on local variable values. Thus, embodiments of the present invention may also be beneficial in situations where the only means for controlling the web tensioning subsystem 200 is a PID controller.

[0041] It should be noted that while subsystems are referred to above as web tensioning systems, filling systems, sterilization systems, package folding systems, etc., these may refer to parts of the above subsystems or individual elements.

[0042] Note that in some embodiments, the control models for the controller 140 may reside in the controller 140 itself, as shown in Figure 1. In other embodiments, they may reside in and operate from external hardware / software (e.g., an external computer or similar processing device) to further accelerate the necessary calculations, and the controller 140 within the food packing machine may be a simpler controller that simply performs functions determined by the external hardware / software.

[0043] The systems and methods disclosed herein can be implemented as software, firmware, hardware, or a combination thereof. In a hardware implementation, the division of tasks among functional units or components mentioned in the above description does not necessarily correspond to a division into physical units; conversely, one physical component may perform multiple functions, and one task may be performed jointly by multiple physical components.

[0044] Particular components, or all components, may be implemented as software executed by a digital signal processor or microprocessor, or as hardware or application-specific integrated circuits. Such software may be distributed on computer-readable media, including computer storage media (or non-transitory media) and communication media (or transitory media). As known to those skilled in the art, the term computer storage media includes both volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules, or other data. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, optical or magnetic storage devices, or any other medium that can be used to store the desired information and that can be accessed by a computer.

[0045] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowcharts or block diagrams may represent a module, segment, or portion of instructions, which includes one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions shown in the blocks may be executed out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may be executed in the reverse order, depending on the functionality involved. Each block of the block diagrams and / or flowchart diagrams, and combinations of blocks in the block diagrams and / or flowchart diagrams, may be implemented by a special-purpose hardware-based system that performs the specified functions or operations or executes a combination of special-purpose hardware and computer instructions.

[0046] From the foregoing description, while various embodiments of the present invention have been described and illustrated, the present invention is not limited thereto and may be embodied in other ways within the scope of the subject matter defined in the following claims.

Claims

1. A method of controlling web tension in a food packing machine (100), the food packing machine (100) comprising a plurality of subsystems, the method comprising: receiving one or more local variable values ​​(116) indicative of measurements by the food packer (100) of one or more physical parameters related to the web tension adjustment subsystem (200); receiving one or more remote variable values ​​(204) indicative of measurements by the food packer (100) of one or more physical parameters related to one or more remote subsystems; determining one or more control parameter values ​​for the web tension adjustment subsystem (200) by processing the remote variable values ​​(204) and the local variable values ​​(116) using a reinforcement learning model (206) and a local control model (210); adjusting one or more control parameters of the web tension adjustment subsystem (200) according to the determined control parameter values; Prepare for this. the one or more local variable values ​​(116) include measurements related to one or more of a web tension set point and a current web tensioning system position; the one or more remote variable values ​​(204) include measurements related to one or more of a web movement control variable, a jaw system motion profile, a packaging material property, and a fill state; method.

2. The reinforcement learning model (206) is a deep reinforcement learning model including a neural network. The method of claim 1.

3. The web tension adjustment subsystem (200) includes two fixed guide rolls (106, 110) and one movable guide roll (108).

3. The method according to claim 1 or 2.

4. the movable guide roll (108) is disposed between the two fixed guide rolls (106, 110) along a path traversed by the web (102) through the food packing machine (100), and is movable to increase or decrease the tension in the web (102) in response to received control parameter value indications; The method of claim 3.

5. the neural network is one of a convolutional neural network, a recurrent neural network, a long short-term memory neural network, and a fully connected neural network; The method of claim 2.

6. A computer program product comprising a computer readable storage medium having instructions adapted to perform the method of any one of claims 1 to 5 when executed by a processor.

7. A system for controlling web tension in a food packing machine (100) having multiple subsystems, the system comprising: Memory and a processor; Equipped with The memory includes instructions that, when executed by the processor, cause the processor to perform a method including: receiving one or more local variable values ​​(116) indicative of measurements by the food packer (100) of one or more physical parameters related to the web tension adjustment subsystem (200); receiving one or more remote variable values ​​(204) indicative of measurements by the food packer (100) of one or more physical parameters related to one or more remote subsystems; determining one or more control parameter values ​​for the web tension adjustment subsystem (200) by processing the remote variable values ​​(204) and the local variable values ​​(116) using a reinforcement learning model (206) and a local control model (210); adjusting one or more control parameters of the web tension adjustment subsystem (200) according to the determined control parameter values; Prepare for this. the one or more local variable values ​​(116) include measurements related to one or more of a web tension set point and a current web tensioning system position; the one or more remote variable values ​​(204) include measurements related to one or more of a web movement control variable, a jaw system motion profile, a packaging material property, and a fill state; system.

8. The reinforcement learning model (206) is a deep reinforcement learning model including a neural network. The system of claim 7.

9. The web tension adjustment subsystem (200) includes two fixed guide rolls (106, 110) and one movable guide roll (108).

9. A system according to claim 7 or 8.

10. the movable guide roll (108) is disposed between the two fixed guide rolls (106, 110) along a path traversed by the web (102) through the food packing machine (100), and is movable to increase or decrease the tension in the web (102) in response to received control parameter value indications; The system of claim 9.

11. The neural network is one of a convolutional neural network, a recurrent neural network, a long short-term memory neural network, and a fully connected neural network. The system of claim 8.

Images