Method and apparatus for automatic packaging

Abstract

The invention provides in other aspects a method for generating first and second instructions for manipulating a physical structure comprising a first part and a second part, comprising: receiving data about a reference volume, the reference volume comprising the first part and the second part of the physical structure; iteratively generating first instructions for causing a first robotic arm to manipulate the physical structure at the first part relative to the reference volume; iteratively generating second instructions for causing a second robotic arm to manipulate the physical structure at the second part relative to the reference volume; wherein the first and second instructions comprise programmable logic controller, PLC, instructions; and wherein the first and second instructions are generated based on at least a measured physical interaction between the first robotic arm and the second robotic arm measured by a feedback device.

Description

Technical Field

[0001] This invention relates to improved automation and industrial computers, including PLCs. Specifically, this invention relates to automation achieved through the combination of two or more robotic arms. Background Technology

[0002] With the advent of industrial computers, many tasks previously performed manually by operators have been partially or fully automated. This improves operator ergonomics, produces higher accuracy, and allows for faster task processing, especially in tasks involving multiple manual operations. In cases of manipulation-intensive tasks requiring numerous maneuvering steps by human operators, the opportunity for automation can be particularly high, as automation can provide a significant speedup in such situations.

[0003] One example of this operation-intensive task is folding instrument trays into packaging material. This packaging involves ensuring that medical tools and instruments used during medical interventions are delivered aseptically. This folding is inherently operation-intensive, which can cause physical discomfort for operators who frequently perform folding. Other challenges exist, involving, for example, improper folding or material defects. For these reasons, automation of folding is highly desirable.

[0004] Other examples of operation-intensive tasks may involve the repeated folding of sheet materials (also for applications beyond packaging) or the collection and storage of multiple objects in containers (e.g., collecting and storing instruments in an instrument tray). For such instances, automation is highly desirable from an ergonomic, reliability, and / or speed perspective.

[0005] EP 3648714 discloses related methods and systems, but only discloses continuous manipulation without direct interaction between robotic arms, which may not be optimal in terms of speed.

[0006] US 20140158141 A1 describes a scheme with four robotic arms in a medical setting and related to packaging. However, it does not describe the control of the robotic arms. Furthermore, it clearly involves a specific form of robotic arm control, see US11185380, which is complex to implement.

[0007] CN 105729472 describes PLC-based control of a robotic arm for packaging. CN 105729472 thus describes PLC control with a 2D camera and vision software. However, CN 1057 only describes the control of a single robotic arm and does not describe schemes with more than one robotic arm.

[0008] WO 2019121571 A1 describes a clamping tool for packaging. However, WO 2019121571A1 thus involves the use of a single clamping tool, which limits the speed of operation due to the sequential execution of all operating steps through this clamping tool, resulting in low operational efficiency.

[0009] The present invention aims to solve problems such as those mentioned above. Summary of the Invention

[0010] According to a first aspect, the present invention provides a method for generating first instructions and second instructions for manipulating a physical structure comprising a first part and a second part, the method comprising: Receive data about a reference volume, which includes a first part and a second part of the physical structure; The first instructions for manipulating the physical structure at the first part are repeatedly generated relative to the reference volume. The second command is repeatedly generated relative to the reference volume to enable the second robotic arm to manipulate the physical structure at the second part. The first and second instructions include programmable logic controller (PLC) instructions. The first and second instructions are generated based at least on the measured physical interaction between the first and second robot arms, as measured by a feedback device.

[0011] By taking into account the measured physical interactions between the robotic arms, this approach can advantageously achieve faster and / or more reliable manipulation. This can be particularly advantageous for operation-intensive tasks. For such tasks, existing methods with multiple robotic arms and PLC-based instruction manipulation typically rely on sequential steps, often with only slight or no temporal overlap, whereby one robotic arm performs a first manipulation step according to a first predetermined program, and when the first program is almost or completely completed, another robotic arm initiates the next manipulation step according to its appropriate predetermined program. In contrast, by taking into account the measured physical interactions and their effects, a more parallel approach is enabled, where, for example, the robotic arms can move toward different coordinates within the same reference volume and simultaneously perform manipulation steps within a directly adjacent range. This, in turn, can enable faster manipulation and / or improved robustness in defective or unpredictable situations where the distance between the robotic arms may deviate from the expected distance. In practice, by detecting such deviations, in embodiments, the method can be executed adaptively based on this information, rather than simply detecting deviations and performing some default response action (e.g., stopping all robotic arms or retracting all robotic arms, merely for illustrative purposes).

[0012] According to a second aspect, the present invention provides an apparatus for generating a first instruction and a second instruction, each of the first instruction and the second instruction comprising PLC instructions for manipulating a physical structure comprising a first part and a second part, the apparatus comprising a PLC module (preferably a PLC module) comprising a processor for executing the method according to the present invention.

[0013] According to another aspect, the present invention provides a system for generating first and second instructions for manipulating a physical structure comprising a first part and a second part, the system comprising: Apparatus, preferably an apparatus according to the invention; A first robotic arm and a second robotic arm are connected to the device and are preferably positioned at different corresponding angles relative to the physical structure; A feedback device is used to measure the physical interaction between the first and second robot arms and is connected to the device. The device is configured as follows: Receive data about a reference volume, which includes a first part and a second part of the physical structure; The physical interactions measured between the first and second robot arms are repeatedly received from the feedback device; The first command is repeatedly generated relative to a reference volume and based on the measured physical interaction to enable the first robotic arm to manipulate the physical structure at the first part. Relative to a reference volume and based on measured physical interactions, second commands are repeatedly generated to enable the second robotic arm to manipulate the physical structure at the second part. The first and second robotic arms are respectively configured as follows: The first and second instructions are repeatedly received from the device, respectively. The physical structure is repeatedly manipulated at the first part and the second part, respectively, based on the first instruction and the second instruction. The feedback device is configured as follows: The physical interaction between the first and second robotic arms was repeatedly measured; and The device repeatedly sends physical interaction measurements; the instructions include programmable logic controller (PLC) instructions.

[0014] According to another aspect, the present invention provides a computer program product comprising a medium for storing instructions for performing the method according to the present invention.

[0015] Preferred embodiments and their advantages are provided in the specification and dependent claims. Attached Figure Description

[0016] The invention will now be discussed in more detail with reference to the accompanying drawings.

[0017] Figures 1 to 5 Different perspective views relating to an exemplary system according to the present invention are shown. Therefore, Figure 1 and Figure 2 The rear view and side view are shown respectively, while Figures 3 to 5 Three close-up views of the front are shown.

[0018] Figures 6 to 7 Two different exemplary types of labyrinthine packaging relating to various aspects of the present invention are shown. Therefore, Figure 6 A maze-like package based on a modified packaging method is shown, while Figure 7 A maze-like package based on the envelope-style packaging method is shown. Detailed Implementation

[0019] The following description depicts exemplary embodiments only and is not intended to limit the scope. Any references to this disclosure herein are not intended to limit or restrict this disclosure to the exact features of any one or more exemplary embodiments disclosed in this specification.

[0020] Furthermore, the terms first, second, third, etc., used in the specification and claims are used to distinguish similar elements and are not necessarily used to describe an order or chronological sequence. Where appropriate, these terms are interchangeable, and embodiments of the invention may operate in a different order than that described or shown herein.

[0021] Furthermore, the different embodiments (though referred to as “preferred”) should be interpreted as exemplary ways in which the invention can be implemented, rather than limiting the scope of the invention.

[0022] The term "comprising" as used in the claims should not be construed as limited to the elements or steps listed thereafter; it does not exclude other elements or steps. It should be interpreted as specifying the presence of the mentioned feature, integral, step, or component, but does not exclude the presence or addition of one or more other features, integrals, steps, or components, or combinations thereof. Therefore, the scope of the expression "a device comprising A and B" should not be limited to a device consisting solely of components A and B, but rather, in relation to the invention, the device is defined by listing only components A and B, and further, the claims should be interpreted as including equivalents of those components.

[0023] In one embodiment, the physical structure relates to a single physical structure. In other embodiments, the physical structure relates to two separable substructures, or to more than two separable substructures. In any of these embodiments, the reference volume may be defined statically or dynamically, preferably such that at least a first portion and a second portion are included in the reference volume, more preferably such that the entire physical structure is included in the reference volume.

[0024] In embodiments, the physical structure relates to a physical object and packaging material. In such embodiments, the physical structure is a general term used to encompass all embodiments that include at least a physical object and packaging material. This can, for example, refer to a single physical structure, such as in embodiments where a portion of the packaging material has been “attached” to a portion of the object by a permanent connection (e.g., a drop of removable glue) and / or a non-permanent connection (e.g., adhesion based on electrostatic charge difference or any other related force). However, a physical structure can also similarly refer to a main physical structure comprising separable substructures, such as a separate physical object for a first substructure and separate packaging for a second substructure. In embodiments, the physical structure relates to a single physical structure integrally formed but having a first and a second portion suitable for manipulation by a robotic arm. This can, for example, refer to embodiments having a single physical structure comprising a main portion and one or more extensions (e.g., foldable protrusions) extending from the main portion, wherein manipulation according to the invention can involve manipulating one of these extensions and the main portion as the first and second portions, or involving manipulating the first and second extensions as the first and second portions, respectively. In each of these embodiments, the reference volume may be defined statically or dynamically, preferably such that at least the first and second portions are included in the reference volume, and more preferably such that the entire physical structure is included in the reference volume.

[0025] In this document, the terms "packing" and "packaging" are used interchangeably. Similarly, the terms "packing material" and "packaging material" are used interchangeably. In this document, the term "PLC" refers to a programmable logic controller. In a preferred embodiment, PLC thus refers to an apparatus, method, programming language, and system conforming to the IEC 61131 standard. In this document, the terms "controller" and "PLC module" are used interchangeably. In an embodiment, the apparatus includes a PLC module and additional modules. In an embodiment, the apparatus is a PLC module.

[0026] In embodiments, the device (preferably a PLC module) includes input / output (I / O) points for connection to other devices, such as robot arms, sensors, and / or actuators. In embodiments where the feedback device includes an external feedback device, the I / O points may allow connection to the external feedback device.

[0027] In embodiments, these input / output points allow inputs from any one or any combination of feedback devices, one or more cameras for computer vision, analog variables from process sensors (such as weight sensors), or sensors for measuring the physical dimensions of a portion of the physical structure (such as depth sensors, surface sensors, or volume sensors). In embodiments, the input / output points allow outputs toward any one or any combination of actuators, indicator lights, alarms, pneumatic or hydraulic cylinders, magnetic relays, electromagnetic coils, or analog outputs for enabling a robotic arm to manipulate the physical structure. Actuators may involve, for example, electric motors and / or pneumatic motors and / or variable frequency drives (VFDs) and / or servo systems and / or stepper motors.

[0028] In this embodiment, the device (preferably a PLC module) includes an Ethernet interface.

[0029] In this document, the term "PLC instruction" refers to any instruction provided directly or indirectly, either as an input or as an output to a PLC module in the device. The term can therefore refer to, for example, instructions written in a PLC programming language format. The term can also refer to data format instructions compatible with reading as input via an I / O point of the PLC module, i.e., instructions entering the PLC module, but not limited to instructions written in a PLC programming language. The term can also refer to instructions generated as an output via an I / O point of the PLC module, i.e., instructions originating from the PLC module, but not limited to instructions written in a PLC programming language.

[0030] In this document, the term "instrument tray" refers to an instance of a physical structure (physical object and packaging material) from a medical context, wherein the physical object is an instrument tray, and the manipulation involves packaging the physical object in the packaging material. The "instrument" present in the instrument tray is merely an exemplary medical device not involving packaging and should not be confused with the first, second, and third instruments involved in actual operation according to the invention in the embodiments.

[0031] In an embodiment, the device of the present invention includes a PLC module, preferably a PLC module. In an embodiment, the PLC module may relate to a controller for an industrial computer, the controller including a CPU and a memory including PLC instructions that, when executed on a processor, cause the PLC module to send control instructions to two or more robotic arms including actuators. In an embodiment, sending control instructions to the robotic arms involves sending control instructions to actuators included in the robotic arms.

[0032] In this embodiment, the device, preferably at least the controller, conforms to IEC 61131. In this embodiment, the device (preferably at least the controller) conforms to at least two parts of IEC 61131, from parts 1 to 10. In this embodiment, the device (preferably at least the controller) conforms to part 3 of IEC 61131, hereinafter referred to as "61131-3," which relates to PLC programming languages.

[0033] In embodiments, the preferred device conforms to Part 3 of IEC 61131, and the PLC instructions relate to a PLC-related programming language conforming to IEC 61131-3. This may involve any of ladder diagrams (LD), function block diagrams (FBD), structured text (ST), instruction lists (IL), and sequential function charts (SFC). In a preferred embodiment, PCL instructions relate to structured text including SCL-defined instructions.

[0034] In an embodiment, the device includes a controller belonging to either the Siemens® Simatic® S7-1500 series (e.g., S7-1510, S7-1511, S7-1512, S7-1513, S7-1514, S7-1515, S7-1516, S7-1517, S7-1518) or the S7-1200 series (e.g., S7-1210, S7211, S7-1212, S7-1213).

[0035] In one embodiment, the feedback device may include or be an external feedback device not belonging to the device. In another embodiment, the feedback device may include or be an internal feedback device belonging to the device. In yet another embodiment, the feedback device may include both internal and external feedback devices.

[0036] In one embodiment, the feedback device (internal or external) involves a sensor dedicated to a single robotic arm and present (and preferably belonging to) or located near that robotic arm, wherein the sensor provides information about the position (e.g., coordinates), velocity, and / or acceleration of the robotic arm. In this embodiment, the measured physical interaction may involve the device receiving data from corresponding feedback devices of different robotic arms and measuring the physical interaction between the first and second robotic arms from this information. This may involve, for example, repeatedly receiving 3D position information from each of the first and second robotic arms and determining, based on the relationship between the 3D position information of the first and second robotic arms, whether a physical interaction is occurring or will occur.

[0037] In one embodiment, the feedback device (internal or external) is dedicated to more than one robotic arm and positioned near more than one robotic arm (e.g., on one of more than one robotic arm), wherein the sensor provides information about the position (e.g., coordinates), velocity, and / or acceleration of the robotic arm relative to each of the more than one robotic arm. In such embodiments, the measured physical interaction may involve the device receiving data from a single feedback device and measuring the physical interaction between a first robotic arm and a second robotic arm based on this information. This may involve, for example, repeatedly receiving 3D position information about each of the first and second robotic arms and determining, based on the relationship between the 3D position information of the first and second robotic arms, that a physical interaction is occurring or will occur.

[0038] In one embodiment, the feedback device and the measured physical interaction involve calculating a 3D path for each of the robot arms based on the desired position of the robot arms. In another embodiment, repeated reception of information from the feedback device allows for correction of the 3D path of one of these robot arms. This is applicable to applications requiring correction in case of accidental operation or, for example, collisions between arms.

[0039] In this embodiment, the feedback device and the measured physical interaction involve calculating a 3D path for each of the robotic arms based on the desired position of the robotic arms, thereby enabling feedback between the robotic arms without receiving any further information during the execution of that path. This can be suitable for applications with high predictability, where multiple robotic arms are expected to move simultaneously (e.g., for increasing speed), but feedback between the robotic arms according to a predetermined 3D path may be sufficient without requiring any (real-time) correction.

[0040] In this embodiment, the first and second instructions consist of PLC instructions.

[0041] In an embodiment, the measured physical interaction relates to the detected physical proximity between the first and second robotic arms, preferably involving a distance between the first and second robotic arms that is less than a predetermined threshold. This can involve, for example, a distance less than 50 mm, 40 mm, 30 mm, 20 mm, 15 mm, 10 mm, or 5 mm.

[0042] In an embodiment, the method further includes repeatedly generating third instructions relative to a reference volume for manipulating a physical structure at a third portion included in the reference volume; wherein the third instructions are generated based at least on measured interactions between the third robot arm and one of the first and second robot arms.

[0043] In an embodiment, the method further includes repeatedly generating fourth instructions relative to a reference volume for manipulating the physical structure at a fourth portion included in the reference volume; wherein the fourth instructions are generated based at least on measured interactions between the fourth robot arm and one of the first, second, and third robot arms.

[0044] In an embodiment, receiving data on the reference volume involves automatically detecting a two-dimensional or three-dimensional bounding box comprising a first and a second portion of the physical structure. This can advantageously allow for automatic initiation when a new physical structure is set up for manipulation.

[0045] In this embodiment, the bounding box is automatically detected as a 3D bounding box surrounding the physical structure. This allows a reference volume to be dynamically defined as a cuboid with a base (depth D and width W) and a height H, relative to a portion of the working region, where each of D, W, and H is defined relative to the detected bounding box dimensions DB, WB, and HB. For example, the reference volume can be centered such that the center of the bounding box coincides with the base of the reference volume, where D = μ DB, W= μ WB, and H = μ HB, where p can be 1.2 or 1.3 or 1.5 or 1.6 or 1.7 or 1.8 or 1.9 or 2 or 2.5 or 3 or 4 or 5 or 6 or 7 or 8 or 9 or 10.

[0046] In an embodiment, the reference volume may be statically defined as a cuboid with a base of a portion of the working area and a height of H, wherein the height may be expressed in absolute terms (e.g., 20 mm, 30 mm, 40 mm, 50 mm, 100 mm, 150 mm, 200 mm, 300 mm, 400 mm, 500 mm, 600 mm, or 700 mm), but may also be expressed relative to a predetermined maximum height HT of the physical structure, for example, 4 / 3. HT or 3 / 2 HT or 2 HT or 3 HT or 4 HT.

[0047] In one embodiment, the automatic detection of the two-dimensional or three-dimensional bounding box involves acquiring sensor data from at least two sensors and applying a bounding box detection algorithm to the sensor data. In a related embodiment, the at least two sensors include at least two cameras, wherein the sensor data is related to image data, and wherein the bounding box detection algorithm involves a computer vision detection algorithm.

[0048] In an embodiment, the PLC instructions relate to a PLC module that includes devices for controlling a first robotic arm to manipulate a first part and simultaneously controlling a second robotic arm to manipulate a second part, the control including calculating a 3D path such that the robotic arms do not collide during the simultaneous manipulation.

[0049] In embodiments, the physical structure involves physical objects and packaging material, and manipulation involves manipulating the packaging material at its first and second portions, which are end portions for realizing a package with multiple turns (preferably according to a maze-like packaging). The present invention relates to the operation-intensive task of wrapping an instrument tray in packaging material. This packaging involves ensuring that medical tools and instruments used during medical interventions are provided in a sterile manner. Thus, an instrument tray comprising cleaned instruments is packaged into packaging material, wherein the instrument tray may be sterilized before or after packaging. Therefore, the packaging, intended to ensure that sterilization conditions can be maintained for as long as possible over time, is typically accomplished using a maze-like packaging pattern, thus requiring numerous folds. This folding is inherently operation-intensive and can be physically uncomfortable for operators who repeatedly perform this operation. Furthermore, this folding can be complicated by practical factors. For example, it may occur that the instrument tray is not manually positioned in the correct location on the packaging material, thereby preventing the required overlap of the packaging portions from being achieved during folding. There is also a risk that the correct folding pattern cannot be achieved due to defects, such as the inability to achieve the minimum number of folds required for the envisioned maze-like packaging. For these reasons, precise automation of folding is highly desirable.

[0050] In some embodiments, the labyrinthine packaging involves a "modified packaging method" (Dutch: gemodificeerde pakket methode). This is further illustrated by Example 2. In other example embodiments, folding is as follows: Figure 2 a to Figure 2 As shown in the corresponding disclosures of i and NL2019198B1. It is clear, implementing packaging according to the modified packaging method involves many consecutive operations, and the present invention can provide a significant acceleration, such as that disclosed in NL2019198B1, by allowing multiple arms to operate simultaneously within the same reference volume, as achieved by measuring the physical interactions between the robot arms by means of feedback devices.

[0051] In this embodiment, the maze-like packaging involves an "enveloppe techniek" packaging method. This is further illustrated by Example 2.

[0052] In an embodiment, manipulation at the first and second portions involves corresponding first and second instruments, wherein the method further includes repeatedly generating instructions for replacing one of the first and second instruments with a third instrument, and repeatedly generating other instructions for manipulating the packaging material by means of the third instrument. This can be advantageous, for example, for situations where the manipulation involves a wider variety of instruments than a robotic arm. Thus, the first, second, and third instruments can relate to any device having functions for tooling and / or gripping. In an exemplary embodiment, the instruments relate to tools and / or clamps.

[0053] In an embodiment, at least one of the first and second robotic arms is a 6-axis robotic arm, preferably wherein each robotic arm is a 6-axis robotic arm. However, the robotic arms can be configured with any suitable number of axes. In an exemplary embodiment, at least the first and second robotic arms are 3-axis, 4-axis, 5-axis, or 7-axis robotic arms.

[0054] Reference Figures 1 to 7 Exemplary embodiments of the present invention are described below.

[0055] Example 1: An exemplary system according to the present invention

[0056] Figures 1 to 5 Different perspective views are shown in relation to the exemplary system 10 according to the present invention. Therefore, Figure 1 and Figure 2 The rear view and side view are shown respectively, while Figures 3 to 5 Three close-up views of the front are shown.

[0057] System 10 is equipped with four robotic arms (1-4), which are substantially identical, though they may differ in several variant instances (not shown). Each robotic arm is a 6-axis robotic arm.

[0058] The robotic arm (1-4) consists of different parts, such as Figure 5 The robotic arm 1 shown above has at least a shoulder blade 1a, an upper arm 1b, a lower arm 1c, and a wrist 1d. When a device (e.g., device 7) is positioned at the end, the end can be considered as a hand, and the portion connecting the upper arm 1b and the shoulder blade 1a can be considered as a shoulder. Therefore, the first robotic arm 1 is positioned on a surface portion located in a first non-horizontal plane 11, while the second, third, and fourth robotic arms 2-4 are positioned on surface portions in a common second plane 12, which is a substantially horizontal plane and also includes a substantially rectangular working area 5 extending within the second plane. Thus, the surface portion of the first robotic arm is positioned above the second plane, and the second plane and the first plane define an angle α between them, wherein the angle α is between 30 degrees and 60 degrees.

[0059] The system includes a PLC module (not shown) that uses corresponding PLC instructions to control the movement of each of the robot arms 1-4.

[0060] Figure 1 and Figure 2 The system is shown in non-operation mode, where no arm is involved in any manipulation and each arm is in a stationary position (the stationary position shown is only an exemplary stationary position). Figures 3 to 5 It provides views related to the operating mode. Specifically, Figures 3 to 5 Manipulation is shown using a device 7, which includes a clamp. Other elements 6, 8, and 9 are... Figure 3 and Figure 5 As shown in the image.

[0061] The system is configured to manipulate a physical structure (not shown) by means of each of the robotic arms, wherein the manipulation involves manipulating a first and a second portion of the physical structure. The physical structure comprises a physical object serving as an instrument tray and packaging material, and the manipulation involves manipulating the packaging material at its first and second portions, which are end portions for achieving a package with multiple folds (preferably a labyrinthine package). Specifically, the packaging involves wrapping the instrument tray within the packaging material to ensure the sterile supply of medical tools and instruments used during medical interventions.

[0062] In some embodiments, the maze-like packaging relates to a “modified packaging” method (Dutch: gemodificeerdepakket methode), one example of which is discussed in Example 2 (but not limited thereto). As will be clear, implementing packaging according to the modified packaging method involves many sequential operations, and the present invention can provide a significant acceleration beyond that disclosed, for example, in NL 2019198 B1, by allowing multiple arms to operate simultaneously within the same reference volume.

[0063] Specifically, the method of the present invention is used to measure the physical interaction between robot arms 1-4 by means of a feedback device. In this example, the feedback device is located in the PLC module and the robot arm.

[0064] Specifically, feedback devices and measured physical interactions are provided by enabling the PLC module to calculate the 3D path of each robot arm based on the desired position of the robot arm. The information repeatedly received by the feedback devices allows for correction of the robot arm's 3D path to prevent collisions or other undesirable behaviors. The feedback devices involve sensors dedicated to each robot arm to provide real-time information about the robot arm's position (i.e., coordinates).

[0065] The measured physical interactions are achieved by the PLC module receiving data from sensors on different robotic arms and measuring the physical interactions between the robotic arms based on this information. This can be advantageous because it allows two or more robotic arms to operate simultaneously within the same small reference volume, rather than having to operate in sequential steps. Furthermore, this can be advantageous because it allows packaging to be performed in a real-time corrective manner. For example, the measured physical interactions can indicate that the instrument tray is not properly aligned with the packaging material, thus clearly showing that the required overlap of the packaging portion cannot be achieved during folding. This can enable the generation of instructions to stop one or each of the robotic arms, and can also enable the generation of alarms, such as those alerting the operator. In other embodiments, the measured physical interactions can reveal that the correct folding pattern has not been achieved due to defects, such as, for example, the minimum number of folds required for the envisioned maze-like packaging cannot be achieved. This can similarly generate instructions to stop one or each of the robotic arms, and can also generate alarms, such as those alerting the operator.

[0066] In this embodiment, the system includes a weight sensor that measures the weight of any given instrument tray and sends it to the PLC module via one of its I / O points.

[0067] In this embodiment, the bounding box is automatically detected as a 3D bounding box surrounding the instrument tray. Detection involves two cameras connected to another device that performs bounding box detection via computer vision to send the detected bounding box dimensions to the I / O point of the PLC module. The PLC module then calculates a reference volume based on the bounding box to account for the space required to perform manipulation during folding. This allows the reference volume to be dynamically defined as a cuboid with a base (depth D and width W) and a height H of a portion of the working area 5, wherein each of D, W, and H is defined relative to the detected bounding box dimensions DB, WB, and HB. In this embodiment, the reference volume is centered such that the middle of the bounding box coincides with the base of the reference volume, where D = μ DB, W= μ WB, and H = μ HB, where p is 2 or 3 or 4 or 5 or 6 or 7 or 8 or 9 or 10, depending on the type and size of the instrument tray, the detected weight detected by the weight sensor, and the type of packaging.

[0068] In this embodiment, the system includes a supply device (not shown) for supplying packaging material and a supply device for supplying an instrument tray (not shown). The packaging material is supplied from a roll and must be cut to a predetermined length to form a substantially rectangular package. To achieve this, the packaging material is clamped by means of a clamp belonging to instrument 7, thereby allowing the packaging material to be aligned with the cutting device. The cutting device may involve, for example, a blade and / or an ultrasonic blade.

[0069] Device 7 relates to a single device disposed on only one of the robotic arms. In a variant of this example (not shown), manipulation at the first and second parts involves corresponding first and second devices, wherein the method further includes repeatedly generating instructions for replacing one of the first and second devices with a third device, and repeatedly generating other instructions for manipulating the packaging material by means of the third device.

[0070] Example 2: Different examples of maze-shaped packaging involving various aspects of the present invention

[0071] Figures 6 to 7 Two different exemplary types of labyrinthine packaging relating to various aspects of the present invention are shown. Therefore, Figure 6 A maze-like package is shown as an example of a modified packaging method with steps 61-66. On the other hand, Figure 7 A maze-like packaging is illustrated according to an example of the envelope-style packaging method with steps 71-79. As can be understood from the figures, some steps may be performed slightly differently or may be interchanged sequentially, thus generating numerous variant examples of both the improved packaging model and the envelope-style packaging method. It is also clear from these figures that this packaging is operationally intensive and requires many packaging steps. The invention advantageously allows at least some steps to be performed in partially overlapping time periods, rather than in a strict sequence, by means of measured physical interactions between the first and second robotic arms measured by a feedback device. Moreover, the invention advantageously enables the detection and correction of many unforeseen factors that may occur in such complex operations, which may involve, for example, material defects or imperfect alignment or problems related to an instrument tray with one or more protruding instruments.

[0072] (End of Example 2)

[0073] Several embodiments have been given above, wherein the physical structure involves physical objects and packaging materials, and manipulation involves manipulating the packaging material at its first and second portions, which are ends for realizing a package with multiple folded layers according to a maze-like packaging. However, the invention is not limited thereto. As will be apparent to those skilled in the art, the invention can be applied to all operations-intensive tasks, which may involve the repeated folding of sheet materials (also for applications other than packaging), or the collection and storage of multiple objects in containers (e.g., collecting and storing instruments in an instrument tray), or any other application involving the manipulation of physical structures at multiple portions by means of two or more robotic arms.

Claims

1. A method for generating first instructions and second instructions for manipulating a physical structure comprising a first part and a second part, comprising: Receive data about a reference volume, the reference volume including the first portion and the second portion of the physical structure; The first instructions for manipulating the physical structure at the first portion are repeatedly generated relative to the reference volume. A second instruction is repeatedly generated relative to the reference volume to cause the second robotic arm to manipulate the physical structure at the second part; Wherein, the first instruction and the second instruction include programmable logic controller (PLC) instructions; and The first instruction and the second instruction are generated at least based on the measured physical interaction between the first robot arm and the second robot arm, as measured by a feedback device.

2. The method according to claim 1, wherein, The first instruction and the second instruction are composed of PLC instructions.

3. The method according to claim 1 or 2, wherein, The measured physical interaction involves the detected physical proximity between the first robot arm and the second robot arm, preferably involving a distance between the first robot arm and the second robot arm that is less than a predetermined threshold.

4. The method according to claims 1 to 3, further comprising: A third instruction is repeatedly generated relative to the reference volume to cause the third robotic arm to manipulate the physical structure at a third portion contained in the reference volume. The third instruction is generated based at least on a measured interaction between the first and second robotic arms and the third robotic arm.

5. The method according to claim 4, further comprising: A fourth instruction is repeatedly generated relative to the reference volume to cause the fourth robotic arm to manipulate the physical structure at a fourth portion contained in the reference volume. The fourth instruction is generated based on at least one of the first, second, and third robot arms and a measured interaction between the fourth robot arm and the first robot arm.

6. The method according to claims 1 to 5, wherein, Receiving data about the reference volume involves automatically detecting a two-dimensional or three-dimensional bounding box comprising the first and second portions of the physical structure.

7. The method according to claim 6, wherein, Automatic detection of the two-dimensional or three-dimensional bounding box involves acquiring sensor data from at least two sensors and applying a bounding box detection algorithm to the sensor data.

8. The method according to claim 7, wherein, The at least two sensors include at least two cameras, wherein the sensor data relates to image data, and wherein the bounding box detection algorithm relates to a computer vision detection algorithm.

9. The method according to claims 1 to 8, preferably claims 2 to 8, wherein, The PLC instructions relate to a PLC module, which includes devices for controlling the first robotic arm to manipulate the first part and simultaneously controlling the second robotic arm to manipulate the second part. The control includes calculating a 3D path such that the robotic arms do not collide during the simultaneous manipulation.

10. The method according to claims 1 to 9, wherein, The physical structure involves physical objects and packaging materials, and the manipulation involves manipulating the packaging material at a first and a second portion, the first and second portions being end portions for realizing a package with multiple folds, preferably packaged according to a maze-like packaging.

11. The method according to claim 10, wherein, The manipulation at the first and second portions involves corresponding first and second instruments, and the method further includes repeatedly generating instructions to replace one of the first and second instruments with a third instrument, and repeatedly generating additional instructions to manipulate the packaging material via the third instrument.

12. The method according to claims 1 to 11, wherein, At least one of the first robot arm and the second robot arm is a 6-axis robot arm, preferably, each robot arm is a 6-axis robot arm.

13. An apparatus for generating first and second instructions, each of the first and second instructions comprising PLC instructions for manipulating a physical structure comprising a first portion and a second portion, the apparatus comprising a PLC module comprising a processor for executing the method according to claims 1 to 12.

14. A system for generating first instructions and second instructions for manipulating a physical structure comprising a first part and a second part, the system comprising: The apparatus, preferably the apparatus according to claim 13; A first robotic arm and a second robotic arm are connected to the device and are preferably positioned at different corresponding angles relative to the physical structure; A feedback device is used to measure the physical interaction between the first robot arm and the second robot arm and is connected to the device; The device is configured as follows: Receive data about a reference volume, the reference volume including the first portion and the second portion of the physical structure; The feedback device repeatedly receives measured physical interactions between the first robot arm and the second robot arm; Relative to the reference volume and based on the measured physical interaction, first instructions are repeatedly generated to cause the first robotic arm to manipulate the physical structure at the first portion; Relative to the reference volume and based on the measured physical interactions, second commands are repeatedly generated to cause the second robotic arm to manipulate the physical structure at the second portion; The first robotic arm and the second robotic arm are respectively configured as follows: The device repeatedly receives the first instruction and the second instruction, respectively. The physical structure is repeatedly manipulated at the first part and the second part respectively based on the first instruction and the second instruction; The feedback device is configured as follows: Repeatedly measure the physical interaction between the first robotic arm and the second robotic arm; and The measurement results of the physical interaction are repeatedly sent to the device; The instructions include programmable logic controller (PLC) instructions.

15. A computer program product comprising a medium for storing instructions for implementing the method according to claims 1 to 12.

Images