Method and system for monitoring setup for manufacturing of biopharmaceutical products

Abstract

This disclosure relates to systems and methods (10) for monitoring setup for manufacturing and / or for setting up for manufacturing and / or for disassembly after the manufacture of a biopharmaceutical product. The method includes: processing (S2, S3, S4) at least one image of a scenario including setup for the manufacture of a biopharmaceutical product. Processing the at least one image includes performing (S2) a first process on the at least one image for classifying a first object in the image, the first object being a device such as a clamp, pump, valve, and / or sensor and / or any other bioprocessing device. The first process includes identifying, locating, and classifying the first object in the image. Performing (S3) a second process on the at least one image for identifying and locating connections in the image. The second process includes: classifying each pixel using an associated second object classifier, the second object classifier classifying a second object selected from a group including the first object and connections; separating pixels associated with connections; and identifying connection mappings to the first object. The method further includes forming (S4) compilation information, the compilation information including information related to the identified connection mappings obtained from the second process and the first object as identified by the first process.

Description

Technical Field

[0001] This disclosure relates to an image / video frame-based method for monitoring the setup of the manufacture of biopharmaceutical products. Background Technology

[0002] The past decade has witnessed a significant shift in the nature of the products manufactured and marketed by the innovative biopharmaceutical industry. Today's global biopharmaceutical portfolio reflects the greater prevalence of large-molecule drugs, an expansion in the number of personalized or targeted products, and the rise of treatments for many orphan diseases. These trends have led to a demand for biopharmaceutical products with extremely limited production runs, highly specific manufacturing requirements, and genotype-specific characteristics. This ongoing transformation in the product mix necessitates continuous improvements in the efficiency and effectiveness of biopharmaceutical manufacturing.

[0003] Biopharmaceuticals (also known as biologics), such as therapeutic proteins, monoclonal antibodies, and vaccines, are complex molecules made through or from living cells. They typically require parenteral administration via infusion or injection, necessitating highly specialized manufacturing, specialized storage and handling, and closely controlled, high-quality manufacturing and distribution networks to ensure safety and efficacy. The development of orphan drugs (i.e., drugs for diseases affecting fewer than 200,000 people) has also been observed, with a steady increase over the past decade. Furthermore, manufacturers are increasingly focusing on more complex diseases for which there are few or no effective treatments. New treatments for these diseases are characterized by small-volume products.

[0004] Another important trend within biopharmaceuticals is the emergence of personalized medicine (i.e., products tailored to specific patient populations). Over time, the complexity of manufacturing and product supply is likely to increase with the introduction of patient-level personalized medicine. Generally, personalized medicine requires related biomanufacturing processes to be conducted on a smaller scale and closer to the patient.

[0005] Examples of applications in personalized medicine include gene and cell therapy treatments, such as CAR T-cell immunotherapy and regenerative medicine approaches.

[0006] Biopharmaceutical products are intended for use, for example, in treatment or diagnosis. To ensure adequate product quality and well-controlled, auditable manufacturing conditions, cGMP (Current Good Manufacturing Practice) and QMS (Quality Management System) are typically required and applied. GMP processing environments are designed to comply with guidelines recommended by specialized agencies authorized and licensed to control the manufacture and sale of pharmaceutical products (such as the Food and Drug Administration, FDA).

[0007] These drug combination trends have led to an increase in the quantity and complexity of products manufactured and commercially marketed. Furthermore, the aforementioned trends and rapid market growth in biotherapies also suggest a dramatic increase in the number of small production batches during drug development before products reach the market, particularly for clinical trial materials. Since the drug development process is generally characterized as a 'development funnel,' a significantly larger number of drug candidates undergo clinical trials compared to the number of successful and ultimately approved drugs. The production of active pharmaceutical ingredients (APIs) during clinical trials and regular manufacturing requires high safety and throughput provided by the manufacturing system. However, API production for clinical trials generally requires even greater flexibility in manufacturing methods and systems to accommodate different requirements and production protocols while still adhering to stringent cGMP and QMS requirements. Given the cost pressures faced by all biopharmaceutical products, cost reduction during clinical manufacturing and drug development is equally important for the regular manufacturing of approved drugs. Therefore, cost-effective and safe manufacturing systems that provide high throughput need to accommodate frequent process changes, process and equipment reconfigurations, and other modifications. Additionally, some new pharmaceuticals have increased the need for more complex manufacturing processes and more advanced equipment. Continuous and interconnected processing systems are becoming a complement to or alternative to traditional batch manufacturing methods, and can offer advantages in terms of overall product and / or process quality, efficiency, production volume, or cost.

[0008] Overall, these drug combination trends indicate a need for improved manufacturing flexibility that does not compromise quality, while simultaneously creating operational efficiencies that help reduce costs. A recent technological development aimed at reducing production costs, increasing production volume and quality, and mitigating safety concerns is the use of single-use technology (SUT) for processing. With SUT equipment, wetted components (e.g., fluid storage containers, piping, separation devices, etc.) that come into contact with process fluids and drug products during processing are provided as consumables. These consumables are installed and used only in a specific process and are subsequently disposed of. SUT consumables are typically generated, configured, and packaged in a cleanroom environment and are pre-sterilized (e.g., by gamma irradiation) before use in the biomanufacturing process. In contrast to traditional and fixed installations using stainless steel piping, reactors, and containers, SUT equipment and consumables offer greater flexibility to adapt to different process scenarios and configurations simply by rearranging (mobile) equipment within the process and installing and replacing different consumables. For example, tank bins can be equipped with clean and / or sterilized SUT fluid bags to provide a clean and enclosed enclosure for the fluid and its processing. The main advantage of using single-use technology (SUT) fluid handling equipment is that it eliminates cross-contamination between production batches and campaigns when the SUT equipment is used for a single pharmaceutical product. The SUT equipment is disposed of after use, which can occur after a single run, batch, or campaign comprising multiple runs and batches. When pre-sterilized or bioload-controlled SUT equipment is provided, initial cleaning and sterilization (e.g., by contacting the flow path with sodium hydroxide solution) or sterilization can be avoided. Post-use cleaning can even be omitted when the SUT is used for a single run or batch. Through these features, SUT equipment offers improved efficiency, safety, and convenience.

[0009] Today, SUT equipment can be used in most types of equipment and / or unit operations, including bioreactors for cell culture or fermentation, buffer bags for liquid storage, piping and pumps for liquid transfer and filling operations, filters for separation, chromatography columns and related systems. However, the adaptation to single-use technology also implies higher production volumes and flow rates of materials (i.e., SUT consumables) in production processes and facilities compared to conventional manufacturing with fixed stainless steel installations.

[0010] Furthermore, numerous additional operational steps and operator interactions are required to control the material flow, install and remove single-use consumables before and after processing, and document the materials, material flow, and usage during processing. Frequent changes associated with SUT consumables imply that new (fresh) installations in the processing pipeline will need to be used and documented for each drug run, batch, or stage of production in the manufacturing process. The biopharmaceutical industry has rapidly adopted SUTs for the aforementioned reasons, but this adaptation is also characterized by at least some of the following challenges:

[0011] Frequent installation of the required complete fluid passage

[0012] The large amount of material to be handled by operators and managed in planning, logistics, and documentation.

[0013] Materials (i.e., consumables and goods) are subject to process variations due to procurement variability and / or the lack of standardization for single-use consumables.

[0014] Many manual interactive steps. Summary of the Invention

[0015] The purpose of this disclosure is to provide solutions that mitigate, alleviate or eliminate one or more of the aforementioned defects in the art, and to provide improvements in biopharmaceutical manufacturing.

[0016] According to this disclosure, the objective is achieved by a method for monitoring the setup for manufacturing and / or for setting up manufacturing and / or for tearing down after the manufacture of a biopharmaceutical product. The method includes processing at least one image of a scene comprising the setup for manufacturing a biopharmaceutical product, wherein processing of the at least one image includes performing a first process on the at least one image for classifying a first object in the image, the first object being a device such as a clamp, pump, valve, and / or sensor and / or single-use bag and / or any other bioprocessing device, the first process including identifying, localizing, and classifying the first object in the image; performing a second process on the at least one image for identifying connections in the image, the second process including classifying each pixel using an associated second object classifier, the second object classifier being a classifier for selecting second objects from a group comprising the first object and connections; segmenting out pixels associated with connections; and identifying connection mappings to the first object. The method further includes forming compilation information, the compilation information including information related to the identified connection mappings obtained from the second process and the first objects as identified by the first process.

[0017] The term "manufacturing" is intended to be understood in a broad sense in this article, and also includes process development.

[0018] The term "image" is intended to also include video frames from a video sequence.

[0019] The term biopharmaceutical products encompasses any type of biopharmaceutical product, including personalized medicine. Examples of personalized medicine include gene and cell therapy treatments, such as CAR T-cell immunotherapy and regenerative medicine approaches.

[0020] Using two different processes for identifying and locating objects within an image allows for flexibility without compromising quality, while also creating operational efficiency that helps reduce costs. The first process focuses on identifying, locating, and classifying a first object and, if possible, its state, while the second process focuses on identifying the connection mapping between the first objects.

[0021] Another advantage of this monitoring method is that it is completely non-invasive.

[0022] Another advantage of the accompanying monitoring method is that the monitoring system is inexpensive.

[0023] Connections may include flexible connections. Connections may include flexible and / or rigid fluid connections. Flexible connections may include pipes.

[0024] In various embodiments of the invention, the first process of identifying, locating, and classifying a first object in an image includes running a deep learning / AI classifier on the image, wherein the method further includes the step of using a boundary to enclose the identified first object.

[0025] In different embodiments of the invention, the step of classifying each pixel using an associated second object classifier includes running the image through a deep learning / AI pixel classifier to identify and locate the second object at the pixel level.

[0026] Therefore, the processing is field-programmable and improves itself over time.

[0027] Furthermore, this method is highly scalable for many types of workflows.

[0028] Setting up flexible workflows adds high value to existing infrastructure and also prevents errors.

[0029] Furthermore, the same algorithm can be used for many customized requirements.

[0030] In different embodiments, the compiled information is compared with the correct workflow scheme or a predefined workflow process scheme to detect any deviations.

[0031] The use of predefined workflow process protocols associated with a specific selected biopharmaceutical product, and comparison with at least a portion of the interactions with the traced user operators of the manufacturing system and / or the results of at least a portion of the interactions with the traced operators of the manufacturing system, aims to improve manufacturing flexibility without compromising quality, while simultaneously creating operational efficiencies that can help reduce costs. Quality, as used herein, applies to multiple or all parts of the process, such as pre-installation inspection (i.e., so-called inspection of the Bill of Materials (BOM), including the materials required to achieve the installation setup), installation of the installation setup, installation procedures, approval / inspection of the installation setup and installation procedures, the handling itself, and disassembly of the installation setup.

[0032] All of the above components can be included in the predefined workflow process scheme described above. Verification and documentation are available. Therefore, the predefined workflow process scheme includes at least data related to traditional batch protocols, but it may further include other processes, i.e., extended batch protocols as evident above. The predefined workflow process scheme may also include batch record data, thus including data related to the approval and verification of batch protocols / extended batch protocols.

[0033] Therefore, the terms 'batch record' (BR, eBR—electronic batch record) and batch protocol are equivalent to "predefined workflow process plan". The protocol is the instruction, and the record is the complete protocol as documented proof and result of the performed process.

[0034] In this embodiment, the monitoring method may support (and / or refine) electronic protocols and records, thereby allowing the processor to communicate and being effective in terms of its flexibility, mobility, adaptability, and learning capabilities.

[0035] Deviations from the installation setup as indicated by the predetermined plan can be detected. Installation setup may include pre-installation inspection (so-called bill of materials inspection) and / or installation and / or disassembly of the installation setup. Deviations from the installation setup that adversely affect the final biopharmaceutical product can lead to a determination that the manufacturing system is not correctly set up.

[0036] Furthermore, deviations from the installation process as indicated by the predetermined plan can be detected. Deviations from the installation procedures that adversely affect the final biopharmaceutical product can lead to a determination that the manufacturing system is not correctly configured.

[0037] Furthermore, deviations from the installation setup and / or installation procedures known to adversely affect the final biopharmaceutical product, as well as other deviations, can then be recorded. The characteristics of the final biopharmaceutical product thus manufactured can be recorded in association with the recorded deviations. A detailed analysis of the impact of different procedural deviations can then be obtained, which can be used to further improve the planned installation procedures. Accordingly, deviations in the resulting final biopharmaceutical product may even decrease over time.

[0038] The monitoring system and the way it works with electronic work instructions also allow for a degree of flexibility in using different external devices (e.g., consumables) that produce the same functionality as the final assembled system. The monitoring system allows operators to be guided through the process according to electronic batch protocols. This flexibility addresses the variability in material procurement; the system is self-learning and improves upon instructions and autonomy in electronic inspections during changes in configuration and reuse.

[0039] In different embodiments, continuous monitoring is performed to track any deviations (e.g., single-use workflow leaks, etc.). Interruption wait).

[0040] Further embodiments of the invention are defined in the dependent claims.

[0041] In addition to the advantages described above (which also apply to system and computer program product embodiments), this disclosure provides the advantage of improved process robustness and the possibility of accelerating vertical scaling to commercial production of biopharmaceutical products. Additional advantages include increased flexibility and reduced lead times.

[0042] This disclosure further relates to a computer program product including at least one non-transitory computer-readable storage medium having computer-executable program code instructions stored therein, wherein the computer-executable program code instructions include program code instructions configured to perform the methods described above when executed.

[0043] This disclosure further relates to a system for monitoring the manufacture of a biopharmaceutical product and / or for setting up the manufacture of the biopharmaceutical product and / or for disassembling the biopharmaceutical product after its manufacture, the system comprising:

[0044] At least one image capturing device (image device) is arranged to capture a scene including a manufacturing system capable of manufacturing biopharmaceutical products, and

[0045] A processor, connected to the at least one image capture device and arranged to process images captured by the at least one image capture device, to track the state of the manufacturing system.

[0046] The processor is configured to execute the method described above. Attached Figure Description

[0047] Figure 1 This is a flowchart illustrating examples of methods for monitoring the setup of a manufacturing process and / or for setting up a manufacturing process and / or for disassembling a biopharmaceutical product after its manufacture.

[0048] Figure 2 This is a schematic diagram. Figure 1 A flowchart illustrating the first process of the method.

[0049] Figure 3 This is a schematic diagram. Figure 1 A flowchart illustrating the second process of the method.

[0050] Figure 4 This is an example of the compilation information generated by the diagram.

[0051] Figure 5 This is a schematic block diagram illustrating an example of a monitoring system used for monitoring a manufacturing system.

[0052] Figure 6 The illustration shows an example of a system that includes a manufacturing system and a monitoring system.

[0053] Figure 7 The illustration shows another example of a manufacturing system and the compiled expanded image provided by the example of a monitoring system.

[0054] Figure 8 The illustrated scheme provides different levels of manufacturing support in accordance with the ISA95 standard.

[0055] Figure 9 The diagram illustrates a timeline of advanced workflow operations and activities, along with examples of the instructions associated with them. Detailed Implementation

[0056] Figure 1 Examples of methods 10 for monitoring the setup of manufacturing and / or for setting up manufacturing and / or for disassembling after the manufacture of a biopharmaceutical product are disclosed.

[0057] The method includes the steps of obtaining one or more images of S1 for use in monitoring the setup of manufacturing and / or setting up manufacturing and / or disassembling after the manufacture of a biopharmaceutical product.

[0058] The resulting images may include images captured at predetermined time intervals or at predetermined time intervals. The resulting images may include video frames.

[0059] The resulting images may include images captured within the field of view. The resulting images may include thermal images. The resulting images may include three-dimensional images.

[0060] When initiating the manufacture of biopharmaceutical products, the process setup regarding the type, configuration, and installation of the manufacturing system, as well as the detailed handling system for operating parameters, is critical to the final biopharmaceutical product. Therefore, the installation procedures used to set up the manufacturing process can impact the final biopharmaceutical product, as incorrect or inadequate installation can cause fluid leaks, malfunctions in handling procedures, or alterations in their results. For example, regulatory and / or legal requirements for the manufacture of biopharmaceuticals approved by the FDA (Food and Drug Administration) necessitate stringent controls and documentation regarding the setup, installation, and use of equipment, such as those related to operator interaction and automated process control. Operating procedures (e.g., batch agreements) and records (e.g., batch records) are fundamental concepts in the development and manufacture of biopharmaceuticals, including approvals and monitoring from regulatory agencies. The resulting images are also useful in monitoring the setup of the manufacturing installation and the procedures used to set up the manufacturing installation.

[0061] The method includes the step of processing the resulting images of scenes S2, S3, and S4, which include settings for the manufacture of biopharmaceutical products.

[0062] The processing of the resulting images includes performing a first process S2 on at least one image to identify, locate, and classify a first object in the image. The first object is a device such as a clamp, pump, valve, and / or sensor, and / or single-use bag, and / or any other biological processing device. The first process identifies and classifies the first object in the resulting image based on the content of the image.

[0063] The first process may include running a first deep learning / AI classifier on the image. The first deep learning / AI classifier may be a trained classifier that has been trained to recognize, locate, and classify a first object in the image. In one example, the trained classifier is an artificial neural network such as a convolutional neural network (CNN).

[0064] The processing of the resulting image includes performing a second process S3 on at least one image for identifying and locating connections in the image. The second process involves classifying each pixel in the corresponding resulting image and associating second object classifications with those pixels identified as components forming the second object in the image. The second object includes connections as well as, as well as, the first object identified by the first process. Therefore, the second process classifies the second object. The second object classifications associated with the corresponding pixels are selected from groups including the first object and connections. Furthermore, the second process identifies connection mappings to the first object.

[0065] At least some of the connections interconnect the first objects and / or connect them to other devices. Connections may include flexible connections. Connections may include flexible and / or rigid fluid connections. Flexible connections may include pipes.

[0066] The second process may include running a second deep learning / AI classifier on the image to classify corresponding pixels in the image. The second classifier may be a trained classifier that has been trained to recognize, locate, and classify a second object in the image. In one example, the trained classifier is an artificial neural network such as a convolutional neural network (CNN).

[0067] The method further includes the step of forming S4 compilation information, which includes information related to the identified link maps obtained from the second process and information related to a first object as identified by the first process. The step of forming the S4 compilation information may include integrating the link maps obtained from the second process and the first object as identified by the first process into the same image to obtain an illustration of the state of the manufacturing setup.

[0068] In one example, the generated compilation information includes a first object in the form of a display. The generated compilation information may also include data related to sensors / devices that were not originally monitored. Therefore, sensor / device data can be obtained using the display readings present in the generated compilation information. Display readings can be obtained continuously, and the compilation information updated accordingly. Alternatively or supplementarily, the display readings can be used by the system for, for example, manufacturing system control.

[0069] The method may further include a step of comparing compilation information with the correct workflow S5 to determine whether the installation settings are correct.

[0070] In detail, comparing the compiled information with the correct workflow may include comparing the compiled information with a predefined workflow process plan involving the selected biopharmaceutical product, and determining whether at least one preset criterion set by the workflow process plan is met based on the comparison. A flag can then be triggered in the event of any error.

[0071] The predetermined workflow process plan can define the correct connection mapping with the first object and the preset criteria that will be met by the corresponding first object, and may define the connection for correct installation.

[0072] In particular, compilation information, including displayed readings, can be compared with a correct workflow to determine whether the installation settings are correct. Therefore, the displayed readings described above can also be used to determine whether at least one preset criterion set by the workflow process scheme is met based on this comparison.

[0073] The method may further include the step of providing S6 as follows: verifying the process for those pipe mappings and / or installations of the first object determined to meet at least one preset criterion set by the workflow process scheme. Verification may be performed using a second source of information (e.g., visual inspection, sensor data from other sensors, etc.).

[0074] The resulting compilation information can be stored along with the verification. Images used in the process can also be stored along with the verification, which is part of the process documentation.

[0075] Images for which the comparison results cannot be verified can be saved for future training of the classifier.

[0076] The monitoring method is suitable for monitoring the setup of a manufacturing process and / or for setting up a manufacturing process and / or for disassembly following the manufacture of various biopharmaceutical products. The monitoring method is also suitable for monitoring small-scale operations that require frequent changes and may necessitate equipment reconfiguration and updates.

[0077] Furthermore, as mentioned above, this method is suitable for monitoring manufacturing processes that involve increased complexity and / or where more advanced equipment can be used.

[0078] Using two distinct processes for recognizing and locating objects within an image allows for flexibility without compromising quality, while simultaneously creating operational efficiency that helps reduce costs. The first process focuses on recognizing, locating, and classifying the first object and possibly its state, while the second process focuses on the connection mapping between recognizing and locating the first object.

[0079] As is evident from the above, it is also possible to automatically integrate images as part of the process documentation for future reference and study, for example, when manual errors are not detected during the process.

[0080] As is evident from the above, errors in classification can be used as training data points for future improvements in expanded scenarios. The process allows users to train on their specific devices and use them in their workflow.

[0081] exist Figure 2 The illustration shows the execution of S2 of a first process on at least one image for classifying a first object in the image. The first process forms components of a method for monitoring the setup for manufacturing and / or for setting up manufacturing and / or for disassembling after the manufacturing of a biopharmaceutical product, having, relative to... Figure 1 At least some of the characteristics that have been disclosed.

[0082] The first object is a device such as a clamp and / or pump and / or valve and / or sensor and / or single-use bag and / or any other bioprocessing equipment belonging to a biopharmaceutical manufacturing system for the manufacture of biopharmaceutical products.

[0083] The first process includes rescaling the image to a lower resolution (S22) before identifying, locating, and classifying the first object. Rescaling can be performed before or after the extraction of regions of the image.

[0084] The first process includes identifying, locating, and classifying a first object in images S23, S24, and S27. In the first process, the steps of identifying, locating, and classifying one or more first objects in images S23 and S27 may include running a deep learning / AI classifier on the images.

[0085] Identifying, locating, and classifying the first object in the images S23, S24, and S27 may include the step of using a boundary to surround the first object identified in S24.

[0086] After the identification, location, and classification of the first object, it can be determined that the identified and classified objects need further classification.

[0087] The first object of interest in the S25 image can then be cropped based on the enclosed boundary.

[0088] The method may then include the following steps: increasing the resolution of the first object of interest in S26; and running a classifier on the high-resolution image in S27 to extract additional details.

[0089] Whether the first identified and classified object needs further classification can be determined based on any of the following:

[0090] The probability of correctly identifying and / or classifying the first object.

[0091] It is possible to determine the state of the first object based on the portion of the image it surrounds.

[0092] It is possible to determine the probability of whether a preset standard set by the workflow process scheme is met based on the portion of the image that is surrounded.

[0093] The first object requiring more details includes, for example, a fixture, where the state of the fixture can be requested.

[0094] The determination of whether the first identified and classified object needs further classification and subsequent enhancement can be performed in multiple steps. Therefore, multiple enhancement levels can be obtained.

[0095] For example, an upgrade can be performed whereby the level of a first object, which can be read in the form of a display, is increased. This allows data related to sensors / devices that were not previously monitored. Therefore, sensor / device data can be obtained using the display readings. Display readings can be obtained continuously.

[0096] Images that are incorrectly identified and classified, or images with a low probability of being correctly identified and classified, can be used for self-learning, for example. Self-learning can involve monitoring and / or continuously improving the setup of a manufacturing process and / or disassembly after the manufacture of a biopharmaceutical product. For example, self-learning can be fed with more data about the view of a photographic apparatus and / or the resolution and / or type of the photographic apparatus.

[0097] In addition to a single image capture infrastructure, supplemental / more image capture infrastructure can be deployed to make the monitoring system (e.g., 3D imaging devices, thermal imaging devices, etc.) more robust. These features can be built into an imaging device or can be mounted as a stand-alone imaging device.

[0098] For images in which the first identified and classified object needs further classification, those images can be saved for future training of the classifier.

[0099] For images that have already been identified and classified as the first object, or even for improved image portions that require further classification, those images can be saved separately for use in future training of the classifier and / or for improving how the images are obtained.

[0100] exist Figure 3 The illustration shows the execution of S3 of the second process on at least one image for classifying a second object in the image. The second process forms components of a method for monitoring the setup of manufacturing and / or for setting up manufacturing and / or for disassembling after the manufacture of a biopharmaceutical product, having, relative to... Figure 1 At least some of the characteristics that have been disclosed.

[0101] The second object includes, for example, through, as targeted Figure 2 The first object identified in the first process. The second object further includes a connector.

[0102] The second process S3 may include the following steps: before classifying each pixel of the image using the associated second object classifier, the image is rescaled S32 to a low resolution.

[0103] The second process includes classifying each pixel of S33 using an associated second object classifier. The step of classifying each pixel of S33 using an associated second object classifier may include running the image through a deep learning / AI pixel classifier to identify and locate the second object at the pixel level.

[0104] The second process may further include separating S34 from the pixels associated with the connection.

[0105] The second process S3 may further include forming a boundary S36 around the first object.

[0106] The second process S3 may further include determining whether the first object classified by the second process in S37 corresponds to the first object classified by the first process (e.g., for...). Figure 2 (as described above). When forming a boundary around the first object in the first and / or second process S36, the determination S37 regarding whether the first object classified by the second process corresponds to the first object classified by the first process can be based on the image portion within the boundary.

[0107] When it is determined that at least one of the first objects classified by the second process does not correspond to the classification performed by the first process, the following steps may be performed:

[0108] Based on probability, make a classification decision S38a, and / or

[0109] Notify S38b users to intervene, and / or

[0110] Store the S38c image for use in training, for example, a classifier that will come from learning and / or, for example, using AI.

[0111] exist Figure 4 The diagram illustrates an example of the compiled information generated. The generated compiled information includes information related to the identified linker map 5 obtained from the second process and, as well as the first object 2 identified by the first process.

[0112] The compiled information is formed based on the first enlarged image 7 of the manufacturing system. Based on, for example, regarding... Figure 2 The disclosed first process is used to form a first enlarged image 7. The first enlarged image 7 includes a first object 2 and a connection 3. The first image is enlarged using a boundary 1' surrounding the first object 2.

[0113] The second enlarged image 8 based on the manufacturing system further forms the compiled information. Based on, for example, regarding... Figure 3The disclosed second process is used to form a second enlarged image. The second enlarged image includes the identified connection map 5 obtained from the second process and an indication 4 of the first object as identified and classified in the second process. The second enlarged image is not necessarily an enlarged actual captured image. The second enlarged image can be any representation of the identified connection map 5 and the indication 4 of the first object.

[0114] exist Figure 4 In the example, the resulting compilation information is represented as an enlarged compilation image 6. In the enlarged compilation image 6, the first object is exposed and marked by boundary 1. The boundary can be compared with, for example, for... Figure 2 The boundary 1' formed in the first process disclosed coincides.

[0115] In the compiled enlarged image 6, the indicator 4 of the second enlarged image 8 and the first object 2 of the first enlarged image 7 are matched, and the connection map 5 is correctly positioned relative to the first object 2 / boundary 1.

[0116] In the illustrated example, boundary 1 of the compiled enlarged image 6 is represented by a solid line for those first objects 2 correctly installed according to the predetermined workflow process. The predetermined workflow process can define the correct connection mapping with the first objects and the preset criteria to be satisfied by the corresponding first objects, and may define the connections used for correct installation. In the illustrated example, boundary 1 is represented by a dashed line for those first objects 2 not yet correctly installed according to the predetermined workflow process. Each boundary can be further associated with a probability value. For example, if the first object is a valve that should be closed according to the predetermined workflow process, a probability value indicating that the valve is closed can be provided based on a comparison of the image with the predetermined workflow process when it has been determined that the valve is closed. This probability value can be shown in the compiled enlarged image. The probability value can be shown at the position of the valve or at the position of the valve boundary. Similarly, if the valve has been determined to be open (even if it should be closed according to the predetermined workflow process), this probability value indicating that the valve is open can be provided in the same manner. In addition, a probability value related to the probability classification of the first object can also be provided when the valve is correct.

[0117] exist Figure 5 The diagram illustrates a system 300, which includes a monitoring system 100 for monitoring the manufacture of a biopharmaceutical product and / or for setting up the manufacture of the biopharmaceutical product and / or for disassembling the biopharmaceutical product after its manufacture. The monitoring system 100 may be arranged to monitor a manufacturing system 200 capable of manufacturing the biopharmaceutical product.

[0118] The monitoring system 100 includes at least one image capture device 140, which is arranged to capture a scene including the manufacturing system 200. At least one image capture device 120 is at least one photographic device. The at least one photographic device may be arranged to record images within a field of view. The at least one photographic device may include a thermal photographic device. The at least one photographic device may include a three-dimensional photographic device. The at least one photographic device may include a camera, which is, for example, arranged to capture images within a field of view, IR, or NIR, and / or arranged to capture three-dimensional images.

[0119] The monitoring system 100 further includes one or more processors 120 and a memory 110. The processors 120 and / or the memory are connected to at least one photographic device. The processors are arranged to process images captured by the photographic device to track the state of the manufacturing system 200.

[0120] A processor is configured to process at least one image of a scene, the scene including a setup or manufacturing system for manufacturing a biopharmaceutical product. Processing of the at least one image includes performing a first process on the at least one image for classifying a first object in the image, the first object being a device such as a clamp, pump, valve, and / or sensor and / or single-use bag and / or any other bioprocessing device, the first process including identifying, locating, and classifying the first object in the image. Processing of the at least one image includes performing a second process on the at least one image (or a copy thereof) for identifying and locating connections in the image. The second process includes: classifying each pixel using an associated second object classifier, the second object classifier being a classifier for selecting second objects from a group including the first object and pipes; separating pixels associated with connections; and identifying connection mappings to the first object. The process further includes forming compilation information, the compilation information including information related to the identified connection mappings obtained from the second process and, as well as the first object as identified by the first process.

[0121] The processor can be configured to perform actions such as those targeting Figure 1 , Figure 2 and Figure 3 The processing is illustrated in the description.

[0122] Figure 6 The illustration shows an example of a system 300 including a manufacturing system 200 and a monitoring system 100. The monitoring system 100 includes an image capture device 140, a processor 120, and a memory 110. The monitoring system has a field of view covering a portion of the manufacturing system having a first part and connections.

[0123] The monitoring system 100 can be small in size and therefore easy to install into an environment containing the manufacturing system 200. In the illustrated example, the monitoring system 100 is formed in a single unit. This is just an example. Different parts of the monitoring system 100 can be formed in different locations. For example, the image capture device can be placed in one location, while the processor and memory are placed in another. The different devices can communicate wirelessly or via wired connections.

[0124] Figure 7 The illustration shows an example of using a monitoring system for monitoring a more flexible manufacturing system 200. The top image illustrates the manufacturing system. The bottom image shows an enlarged view 6 compiled by the monitoring system.

[0125] In the illustrated example, the compiled enlarged image 6 includes information related to the identified connection map 5 obtained from the second process and the first object 2 as identified by the first process. The information related to the corresponding first object 2 as identified by the first process includes a mark-up of the first object in the illustrated example. The mark-up can be enclosed by a boundary 1. The mark-up / boundary 1 can be located at or adjacent to the corresponding object 2. The mark-up / boundary should be positioned relative to the associated first object 2 such that it is clear which first object the mark-up / boundary 1 is associated with. As illustrated, an arrow can point from the mark-up / boundary to the associated first object, or vice versa. The mark-up may include written information indicating the classification of the first object (e.g., pump, workflow connection, display, pressure sensor, temperature sensor, fixture). The mark-up may further include the status associated with the classified first object, such as fixture closure.

[0126] In the illustrated example, the first object 2 is identified by a mark 2', which at least partially surrounds the first object 2.

[0127] Therefore, the monitoring system can support monitoring the status of manufacturing systems that are very flexible and not even constructed in this way.

[0128] This illustration shows how monitoring, as defined in this article, can potentially utilize artificial intelligence in error-proofing workflows, sensor scalability / addition, flexibility, process optimization, and / or continuous monitoring.

[0129] Figure 8The illustrated scheme provides different levels of manufacturing support conforming to the ISA95 standard. Level 0 illustrates the physical production process. In Level 1, the components of the physical production process (e.g., sensors, pumps, etc.) are defined. Sensors are used to sense the physical process, and the production process can be manipulated. Level 2 involves automated systems. Level 2 can be defined as the plant or unit operation level. In Level 2, process monitoring and supervisory control, as well as automatic control, are achieved. Manufacturing control is realized by implementing Levels 1 and 2. Manufacturing control involves basic control, supervisory control, process sensing, and process manipulation.

[0130] Level 3 is the Manufacturing Execution System (MES) level. In the illustrated example, at Level 3, batch records are controlled to produce the intended final product. Batch records are intended for workflow and / or recipe control. Maintaining batch records and / or optimizing the production process is possible. The use of predetermined workflow process schemes disclosed herein allows for manufacturing support primarily at this level, namely, the control, execution, and documentation of batch records. To fully utilize the monitoring systems described herein, batch records are preferably managed electronically, and workflow instructions, as well as inputs and feedback to electronic batch records (eBRs), such as by sensors, are transmitted electronically. This Level 3 can represent the Manufacturing Operations Management level. This level covers production assignment and / or detailed production scheduling and / or reliability assurance.

[0131] However, at least a portion of the workflow can be defined at level 2. This will be relative to... Figure 9 Further discussion.

[0132] Level 4 is Enterprise Resource Planning (ERP) level. Level 4 is the business planning and logistics level. This level involves factory production scheduling and business management.

[0133] Figure 9 The illustration shows an example of a scheme for manufacturing a pre-ordered biopharmaceutical product.

[0134] The illustrated example includes an advanced workflow 80 for the manufacture of a predetermined biopharmaceutical product. In the illustrated example, the advanced workflow 80 begins with material transfer and / or BOM inspection 81. Following this, an installation step 83 for installing the manufacturing system is performed. Afterward, the installation 83 is inspected. Then, an automated process 84, which may involve manual interaction, is performed. Following this, a sampling step 85 for manual sampling activities is performed. Afterward, the manufacturing environment 86 is cleaned. In this step, a single-use product is disposed of. Steps can be added and / or removed from this advanced workflow 80.

[0135] Furthermore, the scheme may include Directives 90 and 100 for the manufacture of pre-determined biopharmaceutical products. These directives include, for example, Standard Operating Procedures (SOPs) and / or Electronic Batch Records (eBRs). The directives may belong to Level 2 or Level 3, or a combination of different levels providing manufacturing support in accordance with the (International Association for Automation) ISA 95 standard, as relative to… Figure 7 As stated above.

[0136] In the illustrated example, Level 3 instruction 90 includes instructions for pipeline cleaning 91. This instruction predates, or can be understood as, the initialization of material transfer and / or BOM check 81.

[0137] In the illustrated example, Level 3 directives include additional directive 92 for consumables and / or equipment and / or fluids and / or etiquette. This directive is characteristically related to material transfer and / or BOM inspection 81 and / or installation 82.

[0138] In the illustrated example, Level 3 instructions include an additional instruction 93 for bag installation and / or bag filling. This instruction is characteristically related to installation 82 and / or inspection 83.

[0139] In the illustrated example, Level 3 instructions include additional instructions 94 for final installation prior to fluid handling. These instructions are characteristically related to inspection 83 and / or automated processing 84.

[0140] In the illustrated example, Level 3 instructions include additional instructions 95 for the collection of fluid samples. These instructions are characterized as automated processing 84 and / or manual sampling activities 85.

[0141] In the illustrated example, Level 3 instructions include additional instructions 96 for product manipulation. These instructions are characteristically related to sampling manual activity 85.

[0142] In the illustrated example, Level 3 instructions include additional instruction 97 for the disposal of consumables. Consumables may be of the SUT type (Single-Use Technology). This instruction falls under the category of Cleaning 86.

[0143] Furthermore, in the illustrated example, Level 2 instruction 100 includes instruction 101 for flow path installation (e.g., a chromatography system). This instruction is characteristically related to material transfer and / or BOM inspection 81 and / or installation 82, and can be invoked or referenced by Level 3 eBR as a standard operating procedure (SOP), for example, defined, executed, documented, and maintained by the chromatography system and its control system or different workflow management systems.

[0144] Furthermore, in the illustrated example, Level 2 instruction 100 includes instruction 102 for the connection of the bag to the chromatography system. This instruction is characteristically related to installation 82 and / or inspection 83. This instruction, its execution, and documentation can again be invoked or referenced by Level 3 eBR as a Standard Operating Procedure (SOP).

[0145] Furthermore, in the illustrated example, level 2 instruction 100 includes instruction 103 for the execution of process automation and / or data logging. This instruction is characteristically related to automatic processing 84.

[0146] Furthermore, in the illustrated example, Level 2 instruction 100 includes instruction 104 for document verification and / or data management. This instruction may include manual sampling activity 85 and / or cleaning activity 86 or as part thereof.

[0147] This approach for manufacturing pre-selected biopharmaceutical products is evident from the above example. Advanced workflows 80 and / or instructions 90, 100 can be added or removed. Instructions can be further appropriately selected to belong to Level 2 or Level 3. Level 2 activities can be understood as SOPs (Standard Operating Procedures), which can be managed by a separate electronic workflow system. This Level 2 workflow management system provides the aforementioned features of instructing, guiding, and correcting operators, documenting results, providing intelligent sensing, learning, and workflow improvement capabilities. The Level 2 workflow management system can be provided by an instrument or system (e.g., a chromatography system and its control software) or by a separate, independent system and software.

[0148] The monitoring systems and methods described herein can be integrated into, for example, systems targeting... Figure 8 and Figure 9 Within the publicly available infrastructure, integrating monitoring systems / methods would be straightforward.

[0149] As described in this article, classifiers and pixel classifiers are, in essence, mathematical functions performed on images.

Claims

1. A method (10) for manufacturing setup and / or post-manufacturing disassembly of a biopharmaceutical product, the method comprising: Process at least one image of scenario (S2, S3, S4), said scenario including the manufacturing setup of the biopharmaceutical product. The processing of the at least one image includes: Perform a first process (S2) on the at least one image to classify a first object in the image, the first object being a clamp, pump, valve, and / or sensor, and / or single-use bag, and / or any other bioprocessing device. The first process includes identifying, locating, and classifying (S23, S24) the first object in the image. Perform a second process (S3) on the at least one image to identify and locate connections in the image, the second process including: Each pixel is classified using an associated second object classifier (S33), the second object classifier being a classifier for selecting second objects from a group including the first object and the connection. Separate (S34) the pixels associated with the connection, and Identify (S35) the connection mapping with the first object, and Form (S4) compilation information, which includes information related to the identified link mapping obtained from the second process and the first object identified by the first process.

2. The method as described in claim 1, wherein, The connection includes at least one flexible connection such as a pipe and / or at least one non-flexible connection.

3. The method as described in claim 1, wherein, In the first process, the step of identifying, locating, and classifying (S23) the first object in the image includes running a deep learning / AI classifier on the image, wherein the method further includes the step of using local boundaries to surround (S24) the identified first object.

4. The method of claim 3, wherein, Performing a first process (S2) on the at least one image to classify a first object in the image includes: Extract (S21) a region of the image having at least a predetermined probability of including a first object, wherein the step of running (S23) the classifier is performed only on the extracted region to identify, locate and classify the first object in the extracted region.

5. The method as described in any of the preceding claims, wherein, The execution of the first process (S2) for classifying objects in the image includes: Before identifying, locating, and classifying the first object, the image is rescaled (S22) to a low resolution, and After the first object is identified and classified, it is determined that the identified and classified object needs further classification. The first object of interest in the image is cropped based on the enclosed boundary (S25). Increase the resolution as described in (S26), and The classifier described in (S27) is run on the high-resolution image to extract additional details.

6. The method of claim 1, wherein, The step of classifying each pixel using an associated second object classifier (S33) includes running the image through a deep learning / AI pixel classifier to identify and locate the second object at the pixel level.

7. The method of claim 1, wherein, The second process (S3) further includes determining (S37) whether the first object classified by the second process corresponds to the first object classified by the first process.

8. The method of claim 7, wherein, The second process (S3) further includes forming a boundary around the first object (S36) before determining (S37) whether the first object classified by the second process corresponds to the first object classified by the first process.

9. The method of claim 7 or 8, wherein, When it is determined that at least one of the first objects classified by the second process does not correspond to the classification performed by the first process, the following steps are performed: Make a classification decision based on probability (S38a), and / or Notify (S38b) the user to intervene, and / or The image is stored (S38c) for use in learning.

10. The method of claim 1, wherein, The second process (S3) further includes the following step: before classifying each pixel of the image using the associated second object classifier, the image is rescaled (S32) to a low resolution.

11. The method of claim 1, wherein, The compiled information is compared with the correct workflow scheme (S5) to detect any deviations.

12. The method of claim 1, wherein, The step of forming (S4) compilation information includes integrating the connector mapping obtained from the second process and the first object as identified by the first process in the same image to obtain a display illustrating the state of the manufactured setup.

13. The method of claim 1, wherein, Continuous monitoring is performed to track any deviations.

14. The method of claim 13, wherein, The deviations include single-use workflow leaks and interruptions.

15. The method of claim 1, wherein, The method is used to monitor the setup for manufacturing biopharmaceutical products and / or to set up the manufacturing of biopharmaceutical products and / or to disassemble the biopharmaceutical products after manufacturing.

16. A computer program product comprising at least one non-transitory computer-readable storage medium having computer-executable program code instructions stored therein, wherein the computer-executable program code instructions include program code instructions configured to perform the method of any one of claims 1-15 when executed.

17. A system (100) for biopharmaceutical manufacturing, the system comprising: At least one image capture device (140) is arranged to capture a scene, the scene including a manufacturing system capable of manufacturing biopharmaceutical products, and A processor (120) is connected to the at least one image capture device (140) and arranged to process images captured by the at least one image capture device to track the state of the manufacturing system. The processor (120) is configured to perform the method as described in any one of claims 1-15.

Images