Method and device for finding objects

Abstract

The invention relates to a method and a device for finding objects (2, 3, 4, 5) in a filling and / or sealing and / or post-processing installation, in particular for the pharmaceutical industry, before a decontamination cycle, before start of production and / or after an end of production, wherein a camera system (10) is used to record an image of a region (6) of the filling and / or sealing and / or post-processing installation, and wherein, on the basis of the image, an artificial intelligence model (AI model) (120) trained to identify the objects (2, 3, 4, 5) is used to check whether an object (2, 3, 4, 5) is present in the region (6) of the filling and / or sealing and / or post-processing installation. The invention also relates to a computer program for finding objects (2, 3, 4, 5) in a filling and / or sealing and / or post-processing installation.

Description

[0001] Method and device for finding objects

[0002] FIELD OF APPLICATION AND STATE OF THE ART

[0003] The invention relates to a method and a device for locating objects in a filling and / or closing and / or post-processing system, particularly for the pharmaceutical industry, before a decontamination cycle, before the start of production, and / or after the end of production. The invention further relates to a computer program for locating objects in a filling and / or closing and / or post-processing system.

[0004] Filling and / or closing systems for the pharmaceutical industry are used to fill liquid or powdered pharmaceutical and biopharmaceutical products into containers such as vials, infusion bottles, cartridges, disposable syringes, or the like. The containers are sealed, if possible, immediately after filling and any subsequent inspection using a closure device such as a plug and / or cap. Post-processing systems are used, for example, for inspection, labeling, repackaging, or other handling of the filled and usually already sealed containers.

[0005] In the context of this application, containers and closure devices are referred to as primary packaging. It is known to provide pre-sterilized primary packaging in so-called nests, which in turn are inserted into tubs. Primary packaging, as well as tubs or nests containing these, or other outer packaging, are referred to as packaging in the context of this application.

[0006] Filling sensitive or hazardous products is typically carried out in an isolator or clean room. Systems without isolators are also known for filling or post-processing pharmaceuticals and / or other products.

[0007] For drug safety reasons, it is typically prescribed in the pharmaceutical industry that only one batch may be located in a filling and / or closing and / or post-processing facility (generally referred to as a facility in the application) at a time. Therefore, when a batch change occurs, all objects located in the facility must be located and removed. In the context of the application, "objects" refers to all primary packaging and other packaging materials used in the facility, such as containers, closure devices, nests, or tubs, as well as consumables used, such as pens, agar plates, or tools used, such as knives, tweezers, hooks, or wrenches.

[0008] The process of finding and removing the objects is called “line clearance”.

[0009] To remove all objects from the system after production has ended, it is common practice to first run the system until it is empty. This process, in which objects, especially primary packaging and other packaging materials, are removed from defined areas such as feeders, filling stations, closing stations, or transport devices, is also known as emptying. An operator must then manually remove all remaining objects.

[0010] Especially for systems that are entirely or partially housed in an isolator, depending on the application, for example, when processing highly potent substances, it is important to avoid opening the isolator doors to locate and remove objects. This limits visibility for locating objects, and all work must be performed using gloved hands, which severely restricts freedom of movement.

[0011] From US 2022 / 113708 a system for automated locating products for line release in a packaging plant is known, comprising a series of image capture devices controlled by a central control unit, wherein images taken by means of the image capture devices after an empty run are compared with control images taken before a start-up in order to determine whether a plant has been cleared.

[0012] TASK AND SOLUTION

[0013] The invention aims to provide a method, a device, and a computer program for improved detection of objects in a filling and / or closing and / or post-processing system, particularly for the pharmaceutical industry. These objectives are achieved by the subject matter having the features of claims 1, 6, and 10. Further advantageous embodiments are set forth in the subclaims.

[0014] According to a first aspect, a method is provided for locating objects in a filling and / or closing and / or post-processing plant, in particular for the pharmaceutical industry, before a decontamination cycle, before starting production and / or after the end of production, wherein an image of an area of ​​the filling and / or closing and / or post-processing plant is recorded by means of a camera system, and wherein the image is used to check whether an object is present in the area of ​​the filling and / or closing and / or post-processing plant using an artificial intelligence model (AI model) which is trained to recognize the objects.

[0015] The terms "a," "an," etc., are used in the context of this application as indefinite articles and not as counting words. In particular, embodiments provide for the acquisition and evaluation of a sequence of images, rather than a single image, for a continuous or time-triggered check to determine whether objects are present in the area. Furthermore, the method is not limited to checking whether a single object is present in the area; rather, it checks whether at least one object is present in the area.

[0016] If more than one object is present in the area, in embodiments of the method a number of the objects is recorded and transmitted to an operator.

[0017] Depending on the application, one or more camera systems are provided on the system, whereby the camera systems take images from different areas or from a common area from different angles, depending on the application.

[0018] In one embodiment, the Kl model is an artificial neural network. In particular, in some embodiments, the Kl model is an artificial neural network for "deep learning," commonly referred to as a "deep neural network."

[0019] Depending on the application, the AI ​​model can be trained to recognize all or individual objects selected from the group comprising primary packaging and other packaging materials, such as containers, closures, nests or tubs, consumables such as pens, agar plates, or tools used, such as knives, tweezers, hooks, or wrenches. This list is not exhaustive, and depending on the system, additional objects may be considered. Different types or kinds of objects within the group are referred to as object types in the context of the application.

[0020] Depending on the application, the areas are defined areas in which packaging materials or other objects are regularly present during production and / or setup of the system, such as areas including feeders, filling stations, closing stations or means of transport, or undefined areas in which objects may be present unexpectedly, such as table tops, floors of clean rooms, or the like.

[0021] By evaluating the image or images of a sequence of images using a Kl model, an automated check of the area for the presence of objects is possible, without the spatial and / or temporal occurrence of the objects in the area being known in advance and without the need for calibration images of the area without any objects present. This means that the method can also be used during initial commissioning to detect the presence of objects that may have undesirably remained in the system during setup. Furthermore, it is possible to locate objects for different machine states, possibly unknown in advance, in which the system comes to a standstill.

[0022] In some embodiments, the objects are removed manually. In other embodiments, they are removed at least from a selected area using a manipulator.

[0023] The check can be repeated continuously during the removal or after the removal of one or more objects until the check shows that no objects are present in the area or an operator confirms that the objects detected during the check can remain in the system.

[0024] In one embodiment, an audio signal or other signal perceivable by the operator is emitted if the check reveals that at least one object is present in the checked area of ​​the filling and / or closing and / or post-processing system. An operator is thus alerted to the presence, with further checking of the area of ​​the system and removal of the object or objects taking place manually. In one embodiment - if the check reveals that an object is present in the area of ​​the filling and / or closing and / or post-processing system - an object type of the present object is identified using the artificial intelligence model (AI model). The identification assists the operator and / or a manipulator in locating the object in the system and in removing the object.In embodiments, for example, it is provided that a removal tool suitable for removing the object of an identified object type is suggested to an operator.

[0025] Alternatively or additionally, in embodiments - if the inspection reveals that an object is present in the area of ​​the filling and / or closing and / or post-processing system - the image is supplemented with visually perceptible information about the present object, and the image supplemented with the visually perceptible information is displayed on a monitor. In embodiments, the monitor is a monitor of a control device of the system. In other embodiments, the monitor is a monitor of a separate desktop, notebook, or tablet computer and / or a viewing and / or display surface of an AR and / or VR terminal device (AR is short for "Augmented Reality"; VR is short for "Virtual Reality"), such as AR and / or VR glasses.

[0026] The visually perceptible information particularly supports the operator in locating the object in the system and in removing it. In some embodiments, the visually perceptible information is a marking of an existing object in the image. If an object type has been identified, the information can alternatively or additionally be a name of the identified object type. Depending on the embodiment, designations of the object types that can be specified during setup and / or by an operator can be displayed as the name. In one embodiment, an indication is also provided as to the probability with which the check has shown that an object is present in the area and / or the probability with which an object is a certain object type.

[0027] Alternatively or additionally, in embodiments - if the check shows that an object is present in the area of ​​the filling and / or closing and / or post-processing system - a position of the existing object in the area is recorded and an associated section of the filling and / or closing and / or post-processing system is identified. In this case, a section is referred to in particular but not exclusively as stations such as filling stations, closing stations, weighing stations, inspection stations, feeding devices, as well as transport devices and transport routes or a peripheral device such as a table top. In one embodiment, the visually perceptible information supplemented by the recorded image comprises an indication of the section in which an object is present or - if applicable - how many objects are present in a section.

[0028] In some embodiments, the results of the inspection are logged, in particular electronically logged, and / or communicated to a central machine control system. In one embodiment, logging only occurs if the inspection reveals that no objects are present in the area. In other embodiments, logging also occurs before all objects have been removed. In some embodiments, the results are communicated to a central machine control system. In some embodiments, a signal is sent to a machine control system of the system, which blocks the opening of an isolator or other housing, the commencement of production, and / or the execution of a decontamination cycle as long as the inspection reveals that at least one object is still present in the monitored area.In some embodiments, an operator can terminate a check of the area and, if necessary, release the system, even if the check using the AI ​​model reveals that at least one object is still present in the monitored area. In some embodiments, the presence of at least one object, which was determined using the AI ​​model, is logged. Depending on the embodiment, the location of an object detected using the AI ​​model and its type are also logged.

[0029] According to a second aspect, a device is provided for locating objects in a filling and / or closing and / or post-processing plant, in particular for the pharmaceutical industry, before a decontamination cycle, before starting production and / or after the end of production, the device comprising a camera system configured to record an image of an area of ​​the filling and / or closing and / or post-processing plant, and a computing unit with an artificial intelligence model (KI model) trained to recognize the objects, wherein the computing unit is configured to check, based on the image and using the artificial intelligence model (KI model), whether an object is present in the area of ​​the filling and / or closing and / or post-processing plant.

[0030] In particular, if the filling and / or closing and / or post-processing system is at least partially enclosed in an isolator, in some embodiments the camera system is also at least partially enclosed in the isolator. In some embodiments, the camera system is a camera system specifically designed for the purpose of locating the objects. In other embodiments, the camera system also serves at least partially to monitor the system during production.

[0031] In embodiments, the computing unit is part of a central machine control and / or communicates with it for at least a unidirectional exchange of data.

[0032] The computing unit is configured in embodiments to identify object types of existing objects using the artificial intelligence model (AI model).

[0033] Alternatively or additionally, the computing unit is configured to supplement the image with visually perceptible information about existing objects and to send the image supplemented with the visually perceptible information to a monitor for playback, wherein visually perceptible information comprises, in particular, markings of existing objects and / or - if identified - names of identified object types. In some embodiments, the monitor is a monitor of a control device of the system. In other embodiments, the monitor is a monitor of a separate desktop, notebook, or tablet computer and / or a viewing and / or display surface of an AR and / or VR terminal.

[0034] In some embodiments, the computing unit is additionally or alternatively configured to detect the positions of existing objects in the area and to identify an associated section of the filling and / or closing and / or post-processing system. In some embodiments, visually perceptible information about the identified area is added to the image. In other embodiments, the information about the identified area is transmitted in another way to a user or a manipulator for removing the objects.

[0035] According to a third aspect, a computer program comprising instructions which, when the program is executed by a computing unit, cause the computing unit to check, based on an image of an area of ​​a filling and / or closing and / or post-processing system, using an artificial intelligence model (AI model) which is trained to recognize objects, whether an object is present in the area of ​​the filling and / or closing and / or post-processing system.

[0036] In other words, the Kl model is trained to recognize objects from a group of objects and it is checked whether at least one object from this group of objects is present in the area. Depending on the application, the Kl model can be trained to recognize all or individual objects selected from the group comprising packaging materials such as containers, closures, nests or tubs, consumables such as pens, agar plates, or tools used such as knives, tweezers, hooks or wrenches. This list is not exhaustive and, depending on the system, other objects can be taken into account. Different types or kinds of objects within the group are referred to as object types in the context of the application.

[0037] In one embodiment, the computer program is installed on a control device of the filling and / or closing and / or post-processing system. In other embodiments, the computer program is installed on a separate desktop, notebook, or tablet computer or a cloud server, whereby a check of the area for objects present in the immediate vicinity of the filling and / or closing and / or post-processing system and / or remotely from it can be carried out.

[0038] In one embodiment, the computer program further comprises instructions which, when the program is executed by the computing unit, cause the computing unit to identify an object type of the existing object using the artificial intelligence model (AI model) if the check shows that an object is present in the area of ​​the filling and / or closing and / or post-processing system.

[0039] Alternatively or additionally, in one embodiment the computer program further comprises instructions which, when the program is executed by the computing unit, cause it - if the check shows that an object is present in the area of ​​the filling and / or closing and / or post-processing system - to supplement the image with visually perceptible information about the existing object and to send the image supplemented with visually perceptible information to a monitor for playback, wherein visually perceptible information includes in particular markings of existing objects and / or - if identified - names of identified object types.

[0040] Alternatively or additionally, in one embodiment, the computer program further comprises instructions which, when the program is executed by the computing unit, cause it - if the check reveals that an object is present in the area of ​​the filling and / or closing and / or post-processing system - to record a position of the existing object in the area and to identify an associated section of the filling and / or closing and / or post-processing system. In one embodiment, the computer program further comprises instructions which, when the program is executed by the computing unit, cause it to log the results of the check, in particular to log them electronically, and / or to send the results of the check for logging.

[0041] BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Further advantages and aspects of the invention emerge from the claims and from the description of exemplary embodiments of the invention, which are explained below with reference to the figures. In the figures:

[0043] Fig. 1: schematically shows a device for locating objects in an area of ​​a filling and / or closing and / or post-processing plant;

[0044] Fig. 2: schematically shows a monitor with a display of an image of the area of ​​the filling and / or closing and / or post-processing system according to Fig. 1, supplemented by visually perceptible information.

[0045] DETAILED DESCRIPTION OF THE EMBODIMENTS

[0046] Fig. 1 shows schematically a device 1 for locating objects 2, 3, 4, 5 in a filling and / or closing and / or post-processing plant, in particular for the pharmaceutical industry.

[0047] The objects 2, 3, 4, 5 shown as examples are containers in the form of bottles or vials 2, containers in the form of syringes 3, closure devices 4 and wrenches 5. However, the selection and representation of objects 2, 3, 4, 5 is merely exemplary.

[0048] Fig. 1 shows, by way of example, an area 6 of the filling and / or closing and / or post-processing system comprising a feed device 60 for feeding the objects 4 in the form of closure means, a manipulator 62 for transferring objects 2, 3 in the form of containers, and a tabletop 64. However, the representation of area 6 is also merely exemplary. Area 6 is shown after the end of production, with various objects 2, 3, 4, 5 still present in area 6, which must be removed before a decontamination cycle and / or before starting production of a subsequent batch.

[0049] For this purpose, the device 1 for locating objects comprises a camera system 10 which is configured to take an image of the area 6, as well as a computing unit 12 with an artificial intelligence model (AI model) 120 which is trained to recognize the objects 2, 3, 4, 5.

[0050] The computing unit 12 is configured to check, based on the image captured by the camera system 10 and using the Kl model 120, whether an object 2, 3, 4, 5 is present in the area 6.

[0051] In the state schematically shown in Fig. 1, the check shows that a total of ten objects 2, 3, 4, 5 are present.

[0052] In the illustrated embodiment, the computing unit 12 is further configured to assist an operator in removing the objects 2, 3, 4, 5 from the area 6 of the system by means of various measures. Alternatively, embodiments provide for at least some of the objects to be removed by means of a manipulator or a suitable controllable kinematic system.

[0053] Computing unit 12 communicates with another computing unit 7, which is configured as a desktop, notebook, or tablet computer. The other computing unit 7 includes a monitor 70 and a memory 72. In other embodiments, the monitor 70 and memory 72 are provided on different units, with the memory 72 being configured as a cloud server in some embodiments. In yet other embodiments, computing unit 12 and computing unit 7 are configured as a shared computing unit.

[0054] In some embodiments, the camera system 10 also serves to monitor the area 6 during production. For this purpose, in some embodiments, the camera system 10 sends images to an evaluation unit 8. The evaluation unit 8 communicates with the second computing unit 7 in order to display images or image sequences created by the evaluation unit 8 on the monitor 70.

[0055] The schematically illustrated computing unit 12 is configured to identify object types of the existing objects 2, 3, 4, 5 using the Kl model 120. This means that in the present case, the computing unit 12 is configured to detect at least vials 2, syringes 3, closure devices 4, and wrenches 5 and to identify them as object types.

[0056] The schematically illustrated computing unit 12 is configured to detect the positions of the existing objects 2, 3, 4, 5 in the area 6 and to identify an associated section of the filling and / or closing and / or post-processing system. This means that in the present case, the computing unit 12 is configured to determine that eight objects 2, 4, 5, including five vials 2, two syringes 3, and a wrench 5, are located on the tabletop 64, one object, namely a closure means 4, is located in the feed device 60, and one object, namely a syringe 3, is located on the manipulator 62.

[0057] To provide effective support to an operator when removing objects 2, 3, 4, 5 from area 6, the illustrated computing unit 12 is configured to supplement the image captured by the camera system with visually perceptible information on existing objects 2, 3, 4, 5 and to send the image supplemented with the visually perceptible information to the monitor 70 for playback.

[0058] Fig. 2 schematically shows the monitor 70 and the image of the area 6 supplemented by visually perceptible information in the state shown in Fig. 1 with 10 objects 2, 3, 4, 5 present in the area 6.

[0059] In the illustrated embodiment, the supplemented visually perceptible information includes markings 90 of detected existing objects as well as designations A, B, C, D of the identified object types. In the illustrated embodiment, the designations A, B, C, D are indicated directly on the markings 90.

[0060] Furthermore, the supplemented visually perceptible information in the illustrated embodiment comprises a text field 92. In the text field, for example, it can be displayed in which section of the filling and / or closing and / or post-processing system objects 2, 3, 4, 5 are located and, if applicable, how many objects 2, 3, 4, 5 are located in a section.

[0061] The schematic figures are for illustration purposes only and numerous variations are possible.

Claims

Patent claims 1. Method for locating objects (2, 3, 4, 5) in a filling and / or closing and / or post-processing system, in particular for the pharmaceutical industry, before a decontamination cycle, before starting production and / or after the end of production, wherein an image of an area (6) of the filling and / or closing and / or post-processing system is recorded by means of a camera system (10), and wherein, on the basis of the image and using an artificial intelligence model (AI model) (120) which is trained to recognize the objects (2, 3, 4, 5), it is checked whether an object (2, 3, 4, 5) is present in the area (6) of the filling and / or closing and / or post-processing system.

2. Method according to claim 1, characterized in that - if the check shows that an object (2, 3, 4, 5) is present in the area (6) of the filling and / or closing and / or post-processing system - an object type of the existing object (2, 3, 4, 5) is identified using the artificial intelligence model (AI model) (120).

3. Method according to claim 1 or 2, characterized in that - if the check shows that an object (2, 3, 4, 5) is present in the area (6) of the filling and / or closing and / or post-processing system - the image is supplemented by visually perceptible information about the existing object (2, 3, 4, 5) and the image supplemented by the visually perceptible information is displayed on a monitor (70), wherein visually perceptible information comprises in particular markings (90) of existing objects (2, 3, 4, 5) and / or - if identified - designations (A; B; C; D) of identified object types.

4. Method according to claim 1, 2 or 3, characterized in that - if the check shows that an object (2, 3, 4, 5) is present in the area (6) of the filling and / or closing and / or post-processing system - a position of the existing object (2, 3, 4, 5) in the area (6) is detected and an associated section of the filling and / or closing and / or post-processing system is identified.

5. Method according to one of claims 1 to 4, characterized in that results of the check are logged, in particular electronically logged, and / or communicated to a central machine control.

6. Device for locating objects (2, 3, 4, 5) in a filling and / or closing and / or post-processing system, in particular for the pharmaceutical industry, before a decontamination cycle, before starting production and / or after the end of production, comprising a camera system (10) which is set up to record an image of an area (6) of the filling and / or closing and / or post-processing system, and a computing unit (12) with an artificial intelligence model (Kl model) (120) which is trained to recognize the objects (2, 3, 4, 5), wherein the computing unit (12) is set up to check, based on the image and using the artificial intelligence model (Kl model) (120), whether an object (2, 3, 4, 5) is present in the area (6) of the filling and / or closing and / or post-processing system.

7. Device according to claim 6, characterized in that the computing unit (12) is configured to identify object types of existing objects (2, 3, 4, 5) using the artificial intelligence model (AI model) (120).

8. Device according to claim 6 or 7, characterized in that the computing unit (12) is set up to supplement the image with visually perceptible information on existing objects (2, 3, 4, 5) and to send the image supplemented by the visually perceptible information to a monitor (70) for reproduction, wherein visually perceptible information comprises in particular markings (90) of existing objects (2, 3, 4, 5) and / or - if identified - names (A, B, C, D) of identified object types.

9. Device according to claim 6, 7 or 8, characterized in that the computing unit (12) is configured to detect positions of existing objects (2, 3, 4, 5) in the area (6) and to identify an associated section of the filling and / or closing and / or post-processing system.

0. Computer program, comprising instructions which, when the program is executed by a computing unit (12), cause the latter to check, based on an image of an area (6) of a filling and / or closing and / or post-processing system and using an artificial intelligence model (AI model) (120) which is trained to recognize objects (2, 3, 4, 5), whether an object (2, 3, 4, 5) is present in the area (6) of the filling and / or closing and / or post-processing system.

11. Computer program according to claim 10, comprising instructions which, when the program is executed by the computing unit (12), cause the computing unit (12) to - if the check shows that an object (2, 3, 4, 5) is present in the area (6) of the filling and / or closing and / or post-processing system - to identify an object type of the existing object (2, 3, 4, 5) using the artificial intelligence model (KI model) (120).

12. Computer program according to claim 10 or 11, comprising instructions which, when the program is executed by the computing unit (12), cause the computing unit (12) to - if the check shows that an object (2, 3, 4, 5) is present in the area (6) of the filling and / or closing and / or post-processing system - supplement the image with visually perceptible information about the existing object (2, 3, 4, 5) and to send the image supplemented by visually perceptible information to a monitor (70) for reproduction, wherein visually perceptible information comprises in particular markings (90) of existing objects and / or - if identified - names (A; B, C, D) of identified object types.

13. Computer program according to claim 10, 11 or 12, comprising instructions which, when the program is executed by the computing unit (12), cause the computing unit (12) to - if the check shows that an object (2, 3, 4, 5) is present in the area (6) of the filling and / or closing and / or post-processing system - to detect a position of the existing object (2, 3, 4, 5) in the area and to identify an associated section of the filling and / or closing and / or post-processing system.

14. Computer program according to one of claims 10 to 13, comprising instructions which, when the program is executed by the computing unit (12), cause the latter to log the results of the check, in particular to log them electronically and / or to send the results of the check for logging.

Images