Systems and methods for performing line clearance and monitoring

Abstract

Systems and methods for performing line clearance and monitoring are disclosed herein. An example method includes receiving first and second sets of images of a production line during runtime operation of the production line, the first set of images representing a first field of view (FOV) oriented to capture objects while they are falling from the production line, and the second set of images representing a second FOV different from the first FOV and oriented to capture objects positioned below the production line. The example method further includes analyzing the first set of images to identify a falling object and analyzing the second set of images to identify a stationary object. The example method further includes, in response to identifying the falling object or the stationary object, causing a display to present a notification including an image of the falling object or the stationary object.

Description

[Technical Field]

[0001] This application relates generally to the use of imaging systems and image analysis algorithms to identify unexpected items on or near manufacturing lines. More particularly, this application relates to systems and methods for performing line clearance and monitoring in biopharmaceutical processes and applications. [Background technology]

[0002] Many manufacturing processes exist that require reconciliation between process input materials and process output materials. The procedures for achieving this reconciliation are commonly known as line clearance. Line clearance is a prominent issue, particularly in biopharmaceutical manufacturing lines, and traditionally involves standardized procedures to ensure that equipment and work areas are free of product, documentation, and materials from the previous process (e.g., manufacturing line run). Broadly speaking, line clearance procedures help prepare operators for the next scheduled process and avoid labeling errors and cross-contamination of the final product.

[0003] However, conventional line clearance procedures suffer from many drawbacks. Specifically, they involve operators manually clearing packaging lines after each lot and manually inspecting entire areas of the production line to ensure no components or materials remain at the end of the process. This conventional procedure is time-consuming, can require two people to check, and generally poses safety and ergonomic concerns for the human operators involved. Moreover, because both the physical inspection tasks and the documentation completion of conventional procedures are almost entirely manual, these conventional procedures often introduce a significant amount of human error. As a result, these conventional line clearance procedures inevitably slow subsequent production operations, place operators in compromised / unsafe positions within the production line to perform manual inspections, result in mislabeled or cross-contaminated products due to manual errors, and / or result in hazardous conditions if the manual inspection overlooks missed or otherwise stray objects within the production line.

[0004] Therefore, a need exists for a line clearance system and method for performing line clearance and monitoring in biopharmaceutical processes and applications that allows operators to easily, efficiently, and safely monitor and clean the manufacturing line during and after operations. Summary of the Invention [Means for solving the problem]

[0005] In general, the disclosed systems and methods may include a camera system that can be mounted on a benchtop or on a production line and remotely view and record video and images of the production line and surrounding area over a network. The camera may run motion detection or machine learning (ML) algorithms to record and store video if an unusual event (e.g., a falling or stationary object) occurs outside of an expected area, and can notify a user / operator in real time. For example, the disclosed systems and methods may provide immediate notification of a dropped or detached product moving through the production line and provide video / image evidence of the dropped or detached product's final location, thereby reducing downtime and improving overall line clearance quality. Live / real-time and recorded video may be accessed within an enterprise network, a production network, and / or a private cloud server, and recorded video may be stored for historical reference / recording. Furthermore, the disclosed systems and methods are modular, such that any number of devices may be used on a single production line, and these devices may be coordinated using on-premise or remote computer systems. The systems and methods of the present disclosure may include multiple cameras installed at selected locations within, near, around, and / or otherwise proximate to the production line to provide a wide field of view (FOV) corresponding to the production line. The systems and methods of the present disclosure may also allow a user / operator to view the processes and line clearance operations associated with the production line in real time through a live camera feed.

[0006] Overall, the systems and methods of the present disclosure may provide significant advantages over conventional techniques, including at least the following: (1) a significant (e.g., about 60%) reduction in time spent performing line clearances, monitoring, and adjustments; (2) an increase in production time / uptime of the manufacturing line (e.g., about 20 days per year); (3) improved safety and a corresponding reduction in ergonomic issues due to the elimination of manual inspection of hard-to-reach and / or otherwise dangerous areas in the manufacturing line; and (4) a reduction in issues / deviations in manual clearance and documentation operations due to a general reduction in human error.

[0007] In particular, aspects of the present disclosure provide a computer-implemented method for performing line clearance and monitoring, the computer-implemented method including: receiving, by one or more processors, a first set of images of a production line during runtime operation of the production line, the first set of images representing a first field of view (FOV) oriented to capture objects while they are falling from the production line; receiving, by the one or more processors, a second set of images of the production line during runtime operation of the production line, the second set of images representing a second FOV different from the first FOV and oriented to capture objects positioned below the production line; analyzing, by the one or more processors, the first set of images and the second set of images to identify (i) a falling object within the first FOV or (ii) a stationary object within the second FOV; and in response to identifying the falling object or the stationary object, causing, by the one or more processors, to present a notification on a display, the notification including an image of the falling object or the stationary object.

[0008] In some aspects, generating the notification further includes, in response to identifying the falling object or the stationary object, triggering, by the one or more processors, recording of a plurality of images from either the first set of images or the second set of images, each image of the plurality of images depicting the falling object or the stationary object, and causing, by the one or more processors, a display to present a notification, the notification including the recording.

[0009] In certain aspects, the computer-implemented method further includes masking a portion of the first image set or the second image set before analyzing the first image set or the second image set, the portion of the first image set or the second image set corresponding to one or more moving components of the manufacturing line.

[0010] In some aspects, generating a notification further includes generating a notification in substantially real time for display on a user computing device in response to identifying the falling or stationary object, the notification comprising at least one of: (i) an email message; (ii) a text message; or (iii) a line monitoring application alert.

[0011] In certain aspects, analyzing the first set of images and the second set of images may further include, by one or more processors, analyzing the first set of images by applying a first algorithm and analyzing the second set of images by applying a second algorithm to identify (i) a falling object within the first FOV or (ii) a stationary object within the second FOV. In these aspects, both the first algorithm and the second algorithm may be, for example, a motion detection algorithm or an ML algorithm / model. Additionally, in these aspects, the first algorithm may be (i) a motion detection algorithm or (ii) a machine learning (ML) algorithm trained with a plurality of training data comprising a plurality of training images representing a production line, where the ML algorithm is configured to receive image data of the production line as input and output an anomaly score corresponding to a confidence level associated with detecting a falling object or a stationary object, and the second algorithm is (i) a motion detection algorithm or (ii) an ML algorithm. Additionally, in these aspects, the computer-implemented method may further include training the ML model using a plurality of training images representing the manufacturing line, the plurality of training images representing the manufacturing line operating with (i) no falling objects within a first FOV and (ii) no stationary objects within a second FOV. Moreover, in these aspects, the ML algorithm may be at least one of (i) an anomaly detection algorithm, (ii) an image classification algorithm, or (iii) an object detection algorithm.

[0012] Another aspect of the present disclosure provides a computer system for performing line clearance and monitoring, the computer system including one or more processors and a program memory coupled to the one or more processors and storing executable instructions that, when executed by the one or more processors, cause the computer system to perform the method of any one of the preceding aspects.

[0013] A further aspect of the present disclosure provides a tangible, non-transitory computer-readable medium storing executable instructions for performing line clearance and monitoring, the executable instructions, when executed by one or more processors of a computer system, causing the computer system to perform the method of any one of the preceding aspects.

[0014] In accordance with the above and the disclosures herein, the present disclosure includes computer improvements or improvements to other technologies, at least because the present disclosure describes, for example, that line clearance and monitoring systems and their various associated components may be improved or enhanced by the disclosed methods, computer systems, and tangible, non-transitory computer-readable medium to provide more accurate, efficient, and safe performance of line clearance and monitoring procedures. That is, the present disclosure describes improvements in the functionality of the line clearance and monitoring system itself or “any other technology or technical field” (e.g., the field of line clearance and monitoring) because the disclosed methods, computer systems, and tangible, non-transitory computer-readable medium improve and enhance the operation of line clearance and monitoring systems by introducing imaging devices incorporating multiple algorithms specifically configured to monitor / analyze active manufacturing line operations, thereby eliminating typical errors and inefficiencies experienced over time by line clearance and monitoring systems lacking such methods, computer systems, and tangible, non-transitory computer-readable medium. This is an improvement over the prior art at least because such conventional systems are prone to error due to their lack of ability to accurately, consistently, or efficiently analyze line clearance and perform line monitoring.

[0015] Additionally, the present disclosure includes the application of various features and functions as described herein by or through the use of particular machines, such as imaging devices, computing systems, and / or other hardware components as described herein.

[0016] Additionally, the present disclosure includes converting or reducing specific articles to a different state or state as a result of accurate line clearance and monitoring based on real-time video and / or image analysis by multiple algorithms specifically configured to analyze specific areas of the production line, for example, converting or reducing error rates and / or production line downtime from a non-optimal or error state to an optimal state.

[0017] Further, the present disclosure includes certain features that add unconventional steps outside of well-understood, routine, and conventional activities in the art, or that, in various embodiments, demonstrate particular useful applications, such as analyzing a first set of images by applying a first algorithm to identify falling objects within a first FOV, analyzing a second set of images by applying a second algorithm to identify stationary objects within the second FOV, and generating a notification for display on a user computing device, the notification including an image of the falling or stationary objects.

[0018] Additional advantages of the techniques of the present disclosure over conventional approaches to line clearance and monitoring will be appreciated by those skilled in the art through this disclosure. The various concepts and techniques introduced above and discussed in more detail below may be implemented in any of numerous ways, and the concepts described above are not limited to any particular implementation scheme. Examples of implementations are provided below for illustrative purposes.

[0019] Those skilled in the art will appreciate that the figures described herein are included for illustrative purposes and are not intended to limit the disclosure. The figures are not necessarily to scale, emphasis instead being placed upon illustrating the principles of the present disclosure. It should be understood that in some instances, various aspects of the illustrated embodiments may be shown exaggerated or enlarged to facilitate understanding of the illustrated embodiments. In the figures, like primary characters throughout the various views generally refer to functionally similar or structurally similar components. [Brief explanation of the drawings]

[0020] [Figure 1A] FIG. 1 is a simplified block diagram of an example system for performing line clearance and monitoring in biopharmaceutical processes and applications according to various aspects disclosed herein. [Figure 1B] FIG. 1 is a simplified block diagram of an example system for performing line clearance and monitoring in biopharmaceutical processes and applications according to various aspects disclosed herein. [Figure 2A] 1 illustrates an example implementation of an imaging device / system in a manufacturing line for performing line clearance and monitoring in accordance with various aspects disclosed herein. [Figure 2B] 1 illustrates an example implementation of an imaging device / system in a manufacturing line for performing line clearance and monitoring in accordance with various aspects disclosed herein. [Figure 3A] 1 illustrates example line clearance and monitoring analysis operations performed as part of execution of a line monitoring application in accordance with various aspects disclosed herein. [Figure 3B] 1 illustrates example line clearance and monitoring analysis operations performed as part of execution of a line monitoring application in accordance with various aspects disclosed herein. [Figure 3C]1 illustrates example line clearance and monitoring analysis operations performed as part of execution of a line monitoring application in accordance with various aspects disclosed herein. [Figure 3D] 1 illustrates example line clearance and monitoring analysis operations performed as part of execution of a line monitoring application in accordance with various aspects disclosed herein. [Figure 4] 1 illustrates an example user interface presented by a line monitoring application including notifications to a user in accordance with various aspects disclosed herein. [Figure 5] FIG. 1 is a flow diagram illustrating an example method for performing line clearance and monitoring in biopharmaceutical processes and applications according to various aspects disclosed herein. DETAILED DESCRIPTION OF THE INVENTION

[0021] Exemplary System 1A is a schematic block diagram of an exemplary system 100A for performing line clearance and monitoring in a biomanufacturing process machine 160, for example, which may produce pharmaceutical products. In some embodiments, system 100A comprises a standalone device, while in other embodiments, system 100A is integrated into other devices. At a high level, system 100A includes the following components: a computing device 110, one or more training image data sources 150, a biomanufacturing process machine 160, and one or more imaging devices 162. In FIG. 1A, computing device 110, biomanufacturing process machine 160, and training image data sources 150 are communicatively coupled via a network 170, which may be or include a proprietary network, the secure public Internet, a virtual private network, and / or any other type of suitable wired or wireless network (e.g., a dedicated access line, a satellite link, a cellular data network, a combination thereof, etc.). In embodiments in which network 170 comprises the Internet, data communications may occur over network 170 via Internet communications protocols. In some embodiments, more or fewer instances of the various components of system 100A than those shown in FIG. 1A may be included in system 100A (e.g., one instance of computing device 110, ten instances of biomanufacturing process machine 160, ten instances of imaging device 162, two instances of training image data source 150, etc.).

[0022] It is worth noting that while system 100A is shown as including biomanufacturing process machine 160, one skilled in the art would understand that the techniques and components of system 100A may be applied to performing line clearance and monitoring in other processes or fields. For example, instead of biomanufacturing process machine 160, the techniques and components of system 100A may be applied to manufacturing in the food / beverage, automotive, electronics, chemical, and / or other industries.

[0023] Biomanufacturing process machine 160 may include a single biomanufacturing process machine or multiple biomanufacturing process machines, either co-located or remote from one another, suitable for producing biological products such as pharmaceutical products. Biomanufacturing process machine 160 may generally include physical devices configured for use in producing (e.g., manufacturing) biological products (e.g., pharmaceutical products), such as filling devices, stirring devices, star wheels, or other container handling equipment.

[0024] In some embodiments, biomanufacturing process machine 160 may be connected to computing device 110 either via network 170 or directly, allowing at least some of the functionality of biomanufacturing process machine 160 to be controlled by computing device 110. In some embodiments, biomanufacturing process machine 160 may be capable of receiving instructions directly from a user (e.g., biomanufacturing process machine 160 may be manually configurable). For example, in some embodiments, biomanufacturing process machine 160 may receive instructions directly from a user to control its operation (e.g., to start or stop an operation).

[0025] The imaging device 162 may be included in the biomanufacturing process machine 160 (e.g., integrated into the biomanufacturing process machine 160) or may be an external device connected to and / or otherwise located in proximity to the biomanufacturing process machine 160. The imaging device 162 may be used to collect video / image data of the interior, exterior, and / or surroundings of the biomanufacturing process machine 160. The imaging device 162 may provide the video / image data to the computing device 110 (e.g., via the network 170), for example. The video / image data may be of any suitable data type, such as real-time video data of a production line included as part of the biomanufacturing process machine 160, single image frames of the production line, and / or any other suitable data type, or a combination thereof. The video / image data may be collected or provided automatically or on demand. For example, a user of the computing device 110 may desire to monitor a production line in the biomanufacturing process machine 160 over a period of time. Accordingly, one or more imaging devices 162 may collect and provide video / image data of the production line over a period of time to computing device 110 and / or may transmit a live video stream of the production line over a period of time or a portion thereof to computing device 110. In some embodiments, imaging device 162 may collect audio data in response to the operation of biomanufacturing process machine 160. For example, imaging device 162 may begin collecting video / image data when biomanufacturing process machine 160 is powered on / begins operation and may continue collecting video / image data until biomanufacturing process machine 160 is powered off / ends operation.

[0026] Biomanufacturing process machine 160 may include one or more devices (not shown) used in the production of biologics (e.g., pharmaceuticals, as discussed in the Background section). Biomanufacturing process machine 160 may be configured to be controllable via manual or automatic input. In some embodiments, biomanufacturing process machine 160 may be configured to receive such control input locally, such as via a user input device local to biomanufacturing process machine 160. In some embodiments, biomanufacturing process machine 160 is configured to receive control input remotely, such as from computing device 110 (e.g., via network 170). Control input may include operational instructions, such as commanding biomanufacturing process machine 160 to power on / initiate operation. In some embodiments, biomanufacturing process machine 160 may terminate operation in response to one or more of the following: That is, (i) the biomanufacturing process machine 160 completes production of a biological product (e.g., an entire batch of pharmaceutical product is finished), (ii) receives an instruction from the line monitoring application 130 regarding line clearance and monitoring, or (iii) receives a manual instruction to end operation.

[0027] Training image data source 150 generally includes training video / image data that may correspond to (e.g., collected during) one or more biomanufacturing processes for producing one or more biological products using biomanufacturing process machine 160. The training video / image data may represent (i) manufacturing line components, (ii) manufacturing line floor areas, (iii) the manufacturing line interior (e.g., gaps between components, etc.), and / or other suitable areas or portions of areas related to the manufacturing line. Additionally, the training video / image data may be collected using imaging device 162 or other similar sensors (by computing device 110 or another device / system). In some aspects, the training video / image data includes image data corresponding to each component and area of ​​the manufacturing line. Accordingly, imaging device 162 may have a collective field of view (FOV) that includes each component and / or area of ​​the manufacturing line such that the models and algorithms described herein may be trained and / or otherwise configured to analyze subsequent runtime image data of any component or area of ​​the manufacturing line based on the training video / image data. In some embodiments, system 100A may omit training image data source 150 and instead receive training video / image data locally, such as via user input at computing device 110 (e.g., a user providing training video / image data on a portable memory drive). In some examples, the training video / image data includes video / image data that does not include any unexpected and / or otherwise unauthorized items (e.g., product containers) that are concatenated (or mixed) with second video / image data that includes the items. In these examples, training image data source 150 or computing device 110 may augment (or concatenate) the first video / image data and the second video / image data using techniques such as Poisson image blending, seamless cloning, etc.

[0028] Computing device 110 may include a single computing device or multiple computing devices collocated or remote from each other. Computing device 110 is generally configured to input video / image data spanning a period of interest into at least one model / algorithm (e.g., trained using training video / image data) to analyze the video / image data and identify falling objects in a first FOV and / or stationary objects in a second FOV. Components of computing device 110 may be interconnected via an address / data bus or other means. Components included in computing device 110 may include a processing unit 120, a network interface 122, a display 124, a user input device 126, and a memory 128, which are discussed in more detail below.

[0029] Processing unit 120 includes one or more processors, each of which may be a programmable microprocessor that executes software instructions stored in memory 128 to perform some or all of the functions of computing device 110 as described herein. Alternatively, one or more of the processors in processing unit 120 may be other types of processors (e.g., application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), etc.).

[0030] Network interface 122 may include any suitable hardware (e.g., front-end transceiver hardware), firmware, or software configured to communicate with external devices or systems (e.g., imaging device 162, biomanufacturing process machine 160, training image data source 150, etc.) over network 170 using one or more communication protocols. For example, network interface 122 may be or include an Ethernet interface.

[0031] Display 124 may use any suitable display technology (e.g., LED, OLED, LCD, etc.) to present information to a user, and user input device 126 may be a keyboard or other suitable input device. In some aspects, display 124 and user input device 126 are integrated into a single device (e.g., a touchscreen display). In general, display 124 and user input device 126 may be combined to allow a user to interact with a graphical user interface (GUI) or other (e.g., text) user interface provided by computing device 110 (e.g., for purposes such as informing a user of line clearance / monitoring glass implementation).

[0032] Memory 128 may include one or more physical memory devices or units, including volatile or non-volatile memory, and may or may not include memory located in various computing devices of computing device 110. Any suitable memory type or types may be used, such as read-only memory (ROM), solid-state drive (SSD), hard disk drive (HDD), etc. Memory 128 may store instructions for one or more software applications included in line monitoring application 130 that may be executed by processing unit 120. In example system 100A, line monitoring application 130 includes a data collection unit 132, a model training unit 134, a user interface unit 136, an unexpected item detection unit 138, and a notification unit 140. Units 132-140 may be separate software components or modules of line monitoring application 130 or may simply represent functionality of line monitoring application 130 that is not necessarily divided among different components / modules. For example, in some embodiments, data collection unit 132 and user interface unit 136 are included in a single software module. Moreover, in some embodiments, units 132-140 may be distributed among multiple copies of line monitoring application 130 (e.g., running on different components of computing device 110) or among different types of applications stored and running on one or more of computing device 110.

[0033] Data collection unit 132 is generally configured to receive data (e.g., video / image data, operator instructions, etc.). In some embodiments, data collection unit 132 receives training video / image data of a biomanufacturing process for producing a biological product (e.g., including historical video / image data of multiple instances of the biomanufacturing process and corresponding historical video / image data). Data collection unit 132 may receive the training video / image data via, for example, training image data source 150, user input received via user interface unit 136 via user input device 126, or other suitable means. In some embodiments, data collection unit 132 may receive the video / image data via, for example, imaging device 162, user input received via user interface unit 136 via user input device 126, or other suitable means. In some embodiments, computing device 110 may receive an indication, e.g., at data collection unit 132, that a biomanufacturing process has begun, and one or more components of computing device 110 may begin monitoring video / image data provided by imaging device 162, for example. In some aspects, data collection unit 132 may apply pre-processing, e.g., resizing, reorientation, color balancing, etc., to the received video / image data, where the training video / image signal and the video / image signal correspond to the training video / image data and the video / image data, respectively.

[0034] Model training unit 134 is generally configured to generate, train, or apply a model. The model may be any suitable model for analyzing video / image data to identify falling or stationary objects. In some embodiments, and as described further below, the model may be trained using at least some of system 100A, or in some embodiments, the model may be pre-trained (i.e., trained before being acquired by computing device 110). The model may be trained using training video / image data representing (i) production line components, (ii) production line floor areas, (iii) production line interiors (e.g., gaps between components, etc.), and / or other suitable areas or portions of areas related to the production line. In some aspects, the model may include a statistical model, a rule-based model, or other suitable model, or a combination thereof, to analyze images captured by imaging device 162, for example, by performing motion detection on the video / image data. Thus, in these aspects, the model may include any suitable image processing algorithm, such as a motion detection algorithm, a dithering algorithm, a feature detection algorithm, a seam carving algorithm, a segmentation algorithm, and / or any other suitable image processing algorithm, or combinations thereof.

[0035] In other embodiments, the model includes a machine learning model. For example, the model may employ a neural network such as a convolutional neural network or a deep learning neural network. Other examples of machine learning models in the model include models that use support vector machine (SVM) analysis, K-nearest neighbor analysis, naive Bayes analysis, clustering, reinforcement learning, or other machine learning algorithms or techniques. The machine learning model included in the model may identify and recognize patterns in training data to facilitate making predictions for new data. The model training unit 134 may train the model using training video / image data, which may be received from the training image data source 150. Of course, generally speaking, the line monitoring application 130 may include any ML model, statistical model, rule-based model, and / or any other suitable model / algorithm in any suitable combination for identifying falling and / or stationary objects in the video / image data.

[0036] In particular, if at least one of the models included as part of the line monitoring application 130 is a machine learning model, the model may be universal (i.e., applicable to all situations) or more specific (i.e., different models for different situations). The machine learning model may be trained using a supervised or unsupervised machine learning program or algorithm. The machine learning program or algorithm may employ a neural network, which may be a convolutional neural network (CNN), a deep learning neural network, or a hybrid learning model or program that learns on two or more features or feature datasets in a particular region of interest. The machine learning program or algorithm may also include regression analysis, support vector machine (SVM) analysis, decision tree analysis, random forest analysis, K-nearest neighbor analysis, naive Bayes analysis, clustering, reinforcement learning, and / or other machine learning algorithms or techniques, or a combination thereof. In some embodiments, due to the processing power requirements of training a machine learning model, the selected model may be trained using additional computing resources (e.g., cloud computing resources) based on data provided by an external source (e.g., training image data source 150). The training data may be unlabeled, or the training data may be labeled, such as by a human. Training of the machine learning models may continue until at least one model of the machine learning models meets selection criteria to be validated and used as a predictive model for identifying falling and / or stationary objects in video / image data. In one embodiment, the machine learning models may be validated using a second subset of the training data to determine the accuracy and robustness of the algorithm. Such validation may include applying the machine learning model to the second subset of the training data to identify falling and / or stationary objects in the video / image data in the second subset of the training data. The machine learning models may then be evaluated to determine whether the machine learning model performance is sufficient based on the validation phase predictions.The sufficiency criteria applied may vary depending on the size of the training data available for training, the performance of previous iterations of the machine learning model, or user-specified performance requirements.

[0037] To be most effective, the ML model may be computationally inexpensive to enable real-time or near-real-time identification of falling and / or stationary objects in the video / image data (e.g., capable of processing and classifying live video / image data at the edge, i.e., by the device itself, or capable of transmitting the video / image data to the cloud for processing in real time). It is generally preferred that the ML model maximize its predictive power within computational constraints driven by device specifications. Because falling and / or stationary objects are rare and the impact of detection is substantial (e.g., stopping / slowing the biomanufacturing process), the ML model may preferably have extremely strong predictive power to avoid false positive events (incorrectly identifying falling and / or stationary objects in the video / image data).

[0038] In general, CNNs are well suited for machine vision applications due to their pattern recognition capabilities. As should be appreciated, CNNs differ from standard multilayer perceptrons (MLPs) by using convolutional layers, in which a matrix of numbers, commonly referred to as filters, is convolved with an input image to generate a tensor representing a new image with any number of channels. This new tensor can then be convolved with a new set of filters in another convolutional layer to generate yet another tensor. This process is repeated for each layer defined in the CNN. In a typical classification task, the final output of the CNN is a set of vectors representing the predicted likelihood of each class. CNN filters can be trained and selected based on recognizing distinct patterns, such as edges, corners, or shapes. Thus, in certain aspects, the ML model included as part of the line monitoring application 130 may be or include a CNN configured to identify falling and / or stationary objects in video / image data.

[0039] In any event, the user interface unit 136 is generally configured to receive user input. In one embodiment, the user interface unit 136 generates a user interface for presentation via the display 124 and may receive, via the user interface and user input device 126, user input training video / image data to be used by the model training unit 134 when training a model. In another embodiment, the user interface unit 136 may receive, via the user interface and user input device 126, input to initiate operation of the biomanufacturing process machine 160 or the imaging device 162. The user interface unit 136 may also be used to display information. For example, the user interface unit 136 may be used to display a representation of a falling or stationary object depicted in the video / image data.

[0040] The unexpected item detection unit 138 may also apply or access a model and / or another model / algorithm (e.g., a motion detection algorithm) trained by the model training unit 134 (or otherwise obtained by the computing device 110 as a pre-trained model) when analyzing the video / image data to identify fallen and / or stationary objects. In some embodiments, the unexpected item detection unit 138 begins analyzing the video / image data in response to the data collection unit 132 receiving the video / image data. The unexpected item detection unit 138 may monitor the video / image data as it is collected by the data collection unit 132 in real time, near real time (i.e., with some buffer), or asynchronously (i.e., after the video / image data has been fully collected over the period of interest). It should be understood that when the unexpected item detection unit 138 is referred to as identifying fallen and / or stationary objects, this also includes detecting that an object has fallen from the production line and / or that an object is stationary on the floor or other surface proximate to the production line (as the unexpected item detection unit 138 may monitor in real time).

[0041] The notification unit 140 is generally configured to notify a user when a falling and / or stationary object is identified and / or to notify a user of where the falling and / or stationary object is located relative to the production line. The notification unit 140 may cooperate with the user interface unit 136 to display the notification. The notification unit 140 may send an electronic message (e.g., email, text, etc.) along with the notification to a user of the computing device 110 or an external computing device. In some embodiments, the notification unit 140 may send a control signal to stop operation of the biomanufacturing process machine 160 when a falling and / or stationary object is detected by the unexpected item detection unit 138. In some embodiments, the notification may be stored (e.g., in memory 128) along with other data (e.g., operational data) regarding the biomanufacturing process machine 160 that may be useful in diagnosing the cause of the falling and / or stationary object, in some cases.

[0042] In some aspects, some or all of the functionality of the line monitoring application 130 may be provided by a third party (i.e., not on the computing device 110). For example, the machine learning model and / or other algorithms / models may be hosted by a third party, and the line monitoring application 130 may access the machine learning model and / or other algorithms / models remotely by transmitting data (e.g., video / image data) and receiving data (e.g., identification of falling objects and / or stationary objects). In such an embodiment, the functionality of the unexpected item detection unit 138 may be hosted by a third party. Turning to a different embodiment, the machine learning model may be trained by a third party, and the line monitoring application 130 may receive the machine learning model remotely from the third party (e.g., by the computing device 110 receiving one or more elements of the machine learning model, such as weights or architecture). In such an embodiment, the functionality of the model training unit 134 may be hosted by a third party. In other embodiments, one or more instances of the functionality of any of the units 132-140 may be hosted by a third party, for example, on a remote server accessible via the network 170.

[0043] FIG. 1B illustrates various exemplary system configurations 100B for performing line clearance and monitoring in biopharmaceutical processes and applications according to various embodiments disclosed herein. In general, the exemplary system configurations 100B may correspond to various configurations of some components included in the exemplary system 100A and / or may include fewer or additional components as described herein. Each of these configurations 100B may enable the actions described herein for performing line clearance and monitoring such that some components of each of the configurations 100B may be located in proximity to the production line (e.g., imaging device 170A1), while other components may not need to be located in proximity to the production line (e.g., operator workstation 170A5). In particular, the exemplary system configuration 100B may include three separate configurations: an Internet of Things (IoT) configuration 170A, a fully cloud-based configuration 170B, and a local computing configuration 170C.

[0044] In IoT configuration 170A, the system may generally include a set of imaging devices 170A1, a set of computing devices 170A2, a network switch 170A3, a cloud-based platform 170A4, and an operator workstation 170A5. Set of imaging devices 170A1 may include any suitable type and / or number of imaging devices configured to capture video / image data corresponding to a manufacturing line or surrounding area (e.g., a floor area proximate to a manufacturing line) before, during, and / or after operation. Due to differences in lighting, available space, and other imaging parameters at various locations near and around the manufacturing line, these various locations may require different imaging devices to capture video / image data useful for subsequent analysis described herein to perform line clearance and monitoring. Accordingly, set of imaging devices 170A1 may include various camera and lens hardware specifically configured to monitor specific portions / areas of the manufacturing line and capture video / image data.

[0045] These sets of imaging devices 170A1 may include, without limitation, standard FOV cameras with variable or fixed zoom, dedicated wide-angle (e.g., 180°+) lenses / cameras, gyroscope-style cameras, and / or any other suitable imaging device type, or combinations thereof. In general, standard FOV cameras may be configured to capture video / image data corresponding to a general observation of various stations / equipment in a manufacturing line. Dedicated wide-angle / viewing area cameras may be specifically configured to observe and capture video / image data corresponding to areas of larger physical volume, such as the floor space under a manufacturing line and associated equipment. Gyroscope-style cameras may be configured to capture video / image data that may correspond to observation areas in tighter spaces within and between equipment that other imaging devices cannot adequately capture. In addition to the cameras and lenses themselves, there may be a local camera / image processor that performs calculations and analyses on the images or video.

[0046] Set of computing devices 170A2 may be or include one or more IoT devices and may be communicatively coupled to one or more other devices (e.g., set of imaging devices 170A1). For example, set of computing devices 170A2 may include an interface for connecting to an imaging device (which may be one or more of set of imaging devices 170A1) and may connect to and / or otherwise interact with network switch 170A3 configured to transmit data between set of computing devices 170A2 and cloud-based platform 170A4. Set of computing devices 170A2 may be selected for any suitable reason, such as for technical specifications that enable recording and storage of live video / image data while providing remote access and a simple user interface.

[0047] In general, cloud-based platform 170A4 may be or include any suitable cloud-based computing platform, such as, for example, Amazon Web Services (AWS). Cloud-based platform 170A4 may also include multiple web-based services 171A1-A4, which may perform various services corresponding to video / image data and / or notifications resulting therefrom. For example, in an aspect where cloud-based platform 170A4 is AWS, multiple web-based services 171A1-A4 may include, but are not limited to, AWS IoT Core 171A1, Amazon CloudWatch 171A2, Storage Service (S3) 171A3, and Amazon Cognito 171A4. In some aspects, cloud-based platform 170A4 may receive video / image data from set of computing devices 170A2 via network switch 170A3, and platform 170A4 may apply various algorithms / models to the video / image data to identify falling and / or stationary objects in the video / image data.

[0048] Further in these aspects, if cloud-based platform 170A4 successfully identifies a falling and / or stationary object in the video / image data, platform 170A4 may generate and / or cause a notification to be displayed to the user / operator, including an image from the video / image data of the falling and / or stationary object. For example, cloud-based platform 170A4 may generate the notification by aggregating images from the video / image data of the falling and / or stationary object and sending the aggregated image to operator workstation 170A5 for display to the user / operator. Additionally or alternatively, cloud-based platform 170A4 may simply cause a display (e.g., display 124, a display of operator workstation 170A5) to present a notification including an image from the video / image data of the falling and / or stationary object.

[0049] In particular, cloud-based platform 170A4 may generate and / or display notifications at operator workstation 170A5 for review by a user / operator. Operator workstation 170A5 may more generally be a computing device / system (e.g., a supervisory control and data acquisition (SCADA) system) that may be communicatively coupled to and / or otherwise configured to control the operation of one or more components of the manufacturing line monitored by set of imaging devices 170A1 and IoT configuration 170A. In particular, operator workstation 170A5 may be configured to communicate and coordinate activities and / or processes between IoT configuration 170A and manufacturing line equipment with respect to operations such as timing, equipment operations, start / stop / hold / resume commands, and / or any other suitable commands, or combinations thereof.

[0050] Broadly speaking, the fully cloud-based configuration 170B includes many components similar to the IoT configuration 170A, but with some differences. That is, the fully cloud-based configuration 170B utilizes additional cloud-based services 171B1 and 171B2 relative to the IoT configuration 170A to account for the lack of the set of computing devices 170A2 included as part of the IoT configuration 170A. In particular, and in an aspect where the cloud-based platform 170B3 is AWS, the plurality of web-based services 171A1-A4 and 171B1-B2 may include, but are not limited to, AWS IoT Core 171A1, Amazon CloudWatch 171A2, Storage Service (S3) 171A3, Amazon Cognito 171A4, Amazon Kinesis 171B1, and Amazon EC2 171B2. The Amazon Kinesis 171B1 and Amazon EC2 171B2 web services may generally host applications configured to perform video / image data processing (e.g., line monitoring application 130) and / or independently perform video / image data processing that would otherwise be performed by set of computing devices 170A2 in IoT configuration 170A. An otherwise complete cloud-based configuration 170B may include a similar or identical set of imaging devices 170A, a network switch 170A3 that directly connects the set of imaging devices 170A to a cloud-based platform 170A4, and an operator workstation 170A5.

[0051] Similarly, local computing configuration 170C may include similar components to both IoT configuration 170A and entirely cloud-based configuration 170B, but with some differences. More particularly, local computing configuration 170C includes local computing device 170C1, which may be configured to perform some / all of the video / image data aggregation, processing, and notification generation / transmission performed by some combination of set of computing devices 170A2, network switch 170A3, and / or cloud-based platform 170A4 in IoT configuration 170A and entirely cloud-based configuration 170B. Thus, the local computing device 170C1 may receive live / real-time streaming video / image data from the set of imaging devices 170A1, analyze the video / image data in accordance with the various line clearance and monitoring operations / actions described herein, generate notifications corresponding to the video / image data analysis, transmit the notifications and / or video / image data to the operator workstation 170A5 for display to a user / operator, and / or cause a display on the operator workstation 170A5 to present a notification to the user / operator including the video / image data of the falling and / or stationary objects.

[0052] As a result of these various exemplary system configurations 100B, it should be understood that some / all of the processing steps / actions performed as part of the line clearance and monitoring operations described herein may be performed remotely (e.g., in a configuration similar to or identical to IoT configuration 170A or fully cloud-based configuration 170B) and / or locally (e.g., in a configuration similar to or identical to local computing configuration 170C).

[0053] FIG. 1C illustrates another exemplary system configuration 100C for implementing line clearance and monitoring in biopharmaceutical processes and applications in accordance with various embodiments disclosed herein. Generally speaking, exemplary system configuration 100C may correspond to any of the various exemplary system configurations 100B illustrated in FIG. 1B, and more specifically, to IoT configuration 170A. In particular, exemplary system configuration 100C generally illustrates how an IoT-based architecture may function as a complementary system to a manufacturing line (e.g., manufacturing lines 181A2 and 181B2). The data flows illustrated in FIG. 1C broadly include data / notifications / commands corresponding to user management, event logging, transmission and recording of video / image data, email and text message dispatch, and system operation commands. The exemplary system configuration 100C includes two manufacturing observation areas 180A1 and 180A2, an IoT upload point 180A3, a cloud-based notification / command storage service 180A4, a local computing network 180A5, and a user / operator account 180A6.

[0054] More specifically, two manufacturing observation areas 180A1 and 180A2 may each include multiple production / manufacturing components (e.g., included as part of manufacturing lines 181A2 and 181B2) and components configured to monitor the production / manufacturing components. These monitoring components include sets of imaging devices 181A1 and 181B1, sets of IoT topics 181A3 and 181B3, cloud-based video / image data storage services 181A4 and 181B4, and real-time operator alerts 181A5 and 181B5 that may be generated as a result of video / image data captured by sets of imaging devices 181A1 and 181B1. Sets of imaging devices 181A1 and 181B1 may be similar to or identical to the imaging devices described herein (e.g., imaging device 162, set of imaging devices 170A1).

[0055] IoT topics 181A3 and 181B3 may generally include operational commands related to production lines 181A2 and 181B2, such as stop commands, start commands, resume commands, hold commands, etc. IoT topics 181A3 and 181B3 may also include and / or otherwise generate / send notifications corresponding to motion alerts (e.g., associated with a falling object), device status (e.g., a device being stopped / paused), and / or other suitable notifications, or combinations thereof. Moreover, IoT topics 181A3 and 181B3 may generally organize sets of commands and notifications by particular production line (e.g., production lines 181A2 and 181B2), batch identifier (e.g., a particular batch of a product and / or a particular product), imaging device identifier (e.g., a particular imaging device in set of imaging devices 181A1 and 181B1), timestamp, and / or any other suitable identifier / metric, or combination thereof.

[0056] The cloud-based video / image data storage services 181A4 and 181B4 may generally receive video / image data from the sets of imaging devices 181A1 and 181B1. More specifically, the cloud-based video / image data storage services 181A4 and 181B4 may receive video / image data representing motion events and / or other events occurring on or near the manufacturing lines 181A2 and 181B2. Thus, when the IoT upload point 180A3 receives a notification from the IoT topic 181A3 and 181B3 corresponding to a motion alert, the IoT upload point 180A3 may also retrieve and / or otherwise receive the video / image data from the cloud-based video / image data storage services 181A4 and 181B4 for any further processing, storage, and / or transmission of the video / image data to the user / operator account 180A6 via the local computing network 180A5. Of course, in certain aspects, the notification received from the IoT topic 181A3 and 181B3 may include the video / image data.

[0057] As part of the notifications generated / sent as a result of IoT topics 181A3 and 181B3, user / operator account 180A6 may also receive real-time operator alerts 181A5 and 181B5 as a result of video / image data captured by sets of imaging devices 181A1 and 181B1. These real-time operator alerts 181A5 and 181B5 may be or include text messages, email messages, line monitoring application messages (e.g., messages received via line monitoring application 130 as running on a user / operator computing device), and / or any other suitable type of message, or combination thereof.

[0058] As described above, commands, notifications, video / image data, and / or other data generated and / or stored in the two manufacturing observation areas 180A1 and 180A2 may be sent to IoT upload point 180A3 for further processing, storage, and / or transmission / routing to associated components. For example, IoT upload point 180A3 may send notifications and commands received from IoT topics 181A3 and 181B3 to cloud-based notification / command storage service 180A4 for storage. IoT upload point 180A3 may also forward notifications, commands, video / image data, and / or any other data to local computing network 180A5 for further processing, storage, or user / operator interaction. Once local computing network 180A5 receives the data from IoT upload point 180A3, local computing network 180A5 may route the data to the appropriate user / operator account 180A6. Through user / operator account 180A6, the associated user / operator may view any / all notifications, commands, video / image data received from local computing network 180A5 and / or directly as real-time operator alerts 181A5 and 181B5.

[0059] More specifically, user / operator account 180A6 may enable an associated user / operator to analyze and generally respond to the data. In certain aspects, components configured to monitor production / manufacturing components (e.g., sets of imaging devices 181A1 and 181B1, etc.) may be directly integrated with manufacturing lines 181A2 and 181B2 such that these components may communicate with and coordinate with the manufacturing equipment of manufacturing lines 181A2 and 181B2 regarding operations such as timing, equipment operation, start / stop / hold / resume commands, etc. Thus, in these aspects, the monitoring components may directly affect and / or otherwise control the operation of the manufacturing equipment of manufacturing lines 181A2 and 181B2, and the user / operator may see the commands executed by the monitoring components as data uploaded to user / operator account 180A6.

[0060] Alternatively, in some embodiments, and as shown in FIG. 1C , components configured to monitor production / manufacturing components (e.g., sets of imaging devices 181A1 and 181B1, IoT topics 181A3 and 181B3, etc.) may be configured in an add-on style architecture that does not communicate directly with manufacturing equipment on production lines 181A2 and 181B2. In these configurations, the monitoring components may require additional operator input via user / operator account 180A6 to execute control commands (e.g., timing, equipment operation, start / stop / hold / resume commands). Thus, the configuration shown in FIG. 1C may require significantly shorter integration time than the direct integration configuration described above, and may be substantially more modular such that the configuration shown in FIG. 1C may be easily applied to various production lines (e.g., production lines 181A2 and 181B2).

[0061] Exemplary Imaging Device / System Implementation 2A illustrates an example implementation 200 of an imaging device 202 disposed within a manufacturing line to perform line clearance and monitoring in accordance with various aspects disclosed herein. Generally, imaging device 202 may be configured to capture real-time video / image data of a floor area 204 within FOV 206 of imaging device 202. More generally, imaging device 202 may be a portable device that is placed as needed within the system and / or may be integrated into and / or attached to manufacturing line equipment 208. In this manner, imaging device 202 may be positioned, oriented, and configured to capture video / image data that may include, for example, stationary objects that have fallen onto floor area 204 from overhanging manufacturing line equipment (collectively referred to herein as 208). Imaging device 202 may continuously capture video / image data of floor area 204, which may be streamed or periodically uploaded to a processing device (e.g., computing device 110) for analysis. If an object falls from manufacturing line equipment 208 (or elsewhere) onto floor area 204 such that the object is within FOV 206, the live stream video data and / or real-time image data may capture the moment the object lands on floor area 204. Accordingly, systems and methods of the present disclosure may identify when and where the object lands on floor area 204 and may then take sufficient action to remove the object from floor area 204, if necessary.

[0062] Imaging devices the same and / or similar to imaging device 202 of FIG. 2A may be positioned at multiple locations throughout a manufacturing line to capture video / image data corresponding to any relevant area of ​​the line. For example, FIG. 2B illustrates an implementation 220 of multiple imaging devices 224A-F distributed throughout manufacturing line equipment 222 to perform line clearance and monitoring in accordance with various aspects disclosed herein. As shown in FIG. 2B, manufacturing line equipment 222 may include multiple stations 222A-F, where manufacturing components are positioned and configured to perform operations / processes that result in the production of a particular product. Each station 222A-F may perform a particular operation that contributes to a portion of the overall manufacturing process, such that an unfinished product enters station 222A and is completed incrementally at each station 222A-F until a finished product exits station 222F.

[0063] At each station 222A-F, a corresponding imaging device 224A-F may capture video / image data corresponding to a particular component of production line equipment 222 located at the respective station 222A-F. For example, imaging device 224B may capture video / image data corresponding to a particular component located at station 222B. Moreover, while the example implementation 220 shown in FIG. 2A depicts a single imaging device (e.g., imaging devices 224A-F) at each station 222A-F, it should be understood that multiple imaging devices 224A-F may be present at each station 222A-F. In this manner, the multiple imaging devices 224A-F may capture video / image data corresponding to multiple different FOVs, thereby providing a more complete perspective of the production line equipment 222 and surrounding areas to more effectively perform line clearance and monitoring. For example, a first imaging device (e.g., imaging device 224C) positioned at station 222C may be positioned / oriented to include within its FOV an equipment / manufacturing component that is part of production line equipment 222 such that the first imaging device captures video / image data corresponding to the equipment / manufacturing component. A second imaging device positioned at station 222C may be positioned / oriented to include within its FOV an equipment / manufacturing component that is part of production line equipment 222 such that the second imaging device captures video / image data corresponding to the floor area surrounding the equipment / manufacturing component.

[0064] Exemplary Line Clearance and Monitoring Analysis 3A illustrates example line clearance and monitoring analysis actions 300 performed as part of execution of a line monitoring application (e.g., line monitoring application 130) in accordance with various aspects disclosed herein. Example line clearance and monitoring analysis actions 300 generally include line monitoring application 130 receiving an initial image 302 of the area surrounding the production line and a subsequent image 304 of the surrounding area including a stationary object 304A. For example, initial image 302 may represent the area surrounding the production line at a first time instance, with no stationary objects present on the floor or the general area surrounding the production line. At a second time instance, an unexpected object 304A may fall off the production line and / or otherwise fall through the area surrounding the production line and land on the floor. Thus, the subsequent image 304 may characterize the stationary object 304A, and the line monitoring application 130 may record and / or otherwise store the initial image 302 and the subsequent image 304 along with timestamps corresponding to the first and second time instances during which the initial image 302 and the subsequent image 304 were captured.

[0065] Once the line monitoring application 130 receives the initial image 302 and the subsequent image 304, the application 130 may run one or more algorithms on the initial image 302 and the subsequent image 304 to identify stationary objects 304A represented in the subsequent image 304. In particular, the line monitoring application 130 may run a motion detection algorithm on the initial image 302 and the subsequent image 304 to identify stationary objects 304A represented in the subsequent image 304. The motion detection algorithm may include subtracting the initial image 302 from the subsequent image 304, thereby generating a difference image 306. Generally, if there are multiple pixels that exceed a specified threshold, an event is recorded by the line monitoring application 130, and the image data (e.g., the initial image 302, the subsequent image 304, and / or the difference image 306) may be saved for operator review. In certain aspects, performing the motion detection algorithm may also include applying an additional filter to either the initial image 302 and / or the subsequent image 304 to generate a difference image 306 .

[0066] The line monitoring application 130 may also execute a motion detection algorithm to identify falling objects captured by an imaging device with a high frame capture rate. Generally, executing a motion detection algorithm with a high frame capture rate imaging device may result in a higher probability that the line monitoring application 130 will detect fast-moving objects (e.g., falling objects). In certain embodiments, the motion detection algorithm may also include a masking function that allows the motion detection algorithm to ignore areas of the captured image that may contain certain motion (e.g., a moving conveyor belt), while still allowing the motion detection algorithm to detect objects that move out of the area of ​​the conveyor belt.

[0067] As mentioned above, in addition to using motion detection algorithms, line clearance and monitoring operations may be performed using other algorithms, such as algorithms / models that utilize machine learning (ML) and artificial intelligence (AI). In certain aspects, these AI and ML models may be trained individually for each imaging device and may be specifically tailored to the particular viewpoint and FOV that the imaging device has on the production line. Of course, captured images may be processed using these AI / ML models on a local processor (e.g., in local computing configuration 170C), on a centralized server located on-premise, and / or in a cloud-based server environment (e.g., IoT configuration 170A or fully cloud-based configuration 170B), but training these AI / ML algorithms has traditionally been difficult as a result of requiring images that display line clearance issues (e.g., stray containers, vials, or syringes) within the FOV. While it is straightforward / easy to acquire images of manufacturing line equipment during normal operation, introducing line clearance issues (e.g., stray containers) during normal operation with conventional techniques is substantially more difficult because it deviates from the standard operating procedures described for many manufacturing processes (e.g., FDA-approved manufacturing processes). To overcome this difficulty experienced with conventional systems, the present technology may apply image enhancement techniques that enhance images of stray containers against images of manufacturing line equipment taken during normal production.

[0068] 3B shows example line clearance and monitoring analysis actions 320 performed as part of execution of a line monitoring application (e.g., line monitoring application 130) according to various aspects disclosed herein. Example line clearance and monitoring analysis actions 320 generally include line monitoring application 130 receiving an input image 322 of a component on a manufacturing line and augmenting input image 322 with an unexpected item 324A (e.g., a vial) to generate an augmented image 324. Generally, line monitoring application 130 may augment input image 322 with an image of unexpected item 324A using image enhancement techniques, including, for example, but not limited to, Poisson image blending, seamless cloning, and / or other suitable image enhancement techniques, or a combination thereof. When the line monitoring application 130 generates the augmented image 324, the line monitoring application 130 may then use the augmented image 324 to train an AI / ML model to identify falling objects (e.g., unexpected item 324A) and / or stationary objects (e.g., stationary object 304A).

[0069] More specifically, FIG. 3C illustrates example line clearance and monitoring analysis actions 340 performed as part of execution of line monitoring application 130, according to various aspects disclosed herein. Example line clearance and monitoring analysis actions 340 generally represent training inputs and training outputs for training an AI / ML model, as trained and executed by line monitoring application 130. The training input 324 illustrated in FIG. 3C is the augmented image 324 of FIG. 3B, and as such, training input 324 also includes unexpected item 324A that was augmented into training input 324 by line monitoring application 130. Thus, line monitoring application 130 may input training input 324 into the AI / ML model to train the AI / ML model to identify falling objects (e.g., unexpected item 324A) and / or stationary objects (e.g., stationary object 304A). In certain aspects, the AI / ML models that may be trained by the line monitoring application 130 may be or may include an anomaly detection model, an image classification model, an object detection model, and / or any other suitable model / algorithm, or combination thereof.

[0070] As a result of inputting the training inputs 324 into the AI / ML model, the model may output training outputs, which may be presented as part of the line monitoring graphical user interface (GUI) 342. For example, the line monitoring GUI 342 includes a training output 342A indicating that the AI / ML model correctly identified a falling object in the training input 324. In addition, the AI / ML model may output a score for the identification of the falling object in the training input 324. This score may be an anomaly score that generally reflects the confidence with which the AI / ML model identified the anomaly (e.g., the falling object) in the training input 324 as the training output 342A. The AI / ML model may condition its identification of falling and / or stationary objects in the training data (e.g., the training input 324) and / or live data (e.g., data captured during normal operation of the production line) on a discrimination threshold stored in the line monitoring application 130. In certain aspects, the discrimination threshold may be adjusted / set by a user / operator during training and / or prior to execution of the AI / ML model during normal operation of the production line.

[0071] 3D illustrates yet another example line clearance and monitoring analysis action 360 performed as part of execution of line monitoring application 130, in accordance with various aspects disclosed herein. FIG. 3D illustrates heat map portions 364A1-A2, 364B1, and 364C1-C2 in grayscale shading and / or patterning, with heat map portions 364A1-A2, 364B1, and 364C1-C2 being represented using color coding in some embodiments. In any case, in the example line clearance and monitoring analysis action 360 of FIG. 3D, line monitoring application 130 may receive a first image 362 featuring a portion of a manufacturing line in normal operation with an identified unexpected object 362A. In this example line clearance and monitoring analysis action 360, line monitoring application 130 may run a motion detection algorithm and / or an AI / ML model on first image 362 to identify unexpected object 362A.

[0072] In any case, upon identification of unexpected object 362A, line monitoring application 130 may access video / image data from some / all of the other imaging devices captured at the same or similar timestamp as first image 362 and may run motion detection algorithms and / or AI / ML models on the video / image data to identify any additional unexpected objects. In one particular aspect, as shown in example line monitoring GUI 364 of FIG. 3D , line monitoring application 130 may acquire and analyze data from three additional imaging devices to generate multiple heat maps corresponding to the identified unexpected objects in and / or around the production line. First GUI image 364A may correspond to first image 362 when line monitoring algorithm 130 applies an algorithm configured to generate a heat map graphical overlay on first image 362 to generate heat map portions 364A1 and 364A2. These heatmap portions 364A1 and 364A2 may correspond to portions of the first GUI image 364A that may include an unexpected object, and unexpected object 352A may be represented in first heatmap portion 364A1. Similarly, second and third GUI images 364B and 364C may include multiple heatmap portions 364B1, 364C1, and 364C2 that also correspond to portions of GUI images 364B and 364C that may include an unexpected object. Line monitoring application 130 may analyze fourth GUI image 364D and may not detect any unexpected objects, such that fourth GUI image 364D may not include a heatmap graphical overlay.

[0073] In certain aspects, the heat map graphical overlay may also indicate historical regions of the respective FOVs represented by the images of the example line monitoring GUI 364 that contain identified unexpected objects. Accordingly, the first image 362 may affect the historical heat map graphical overlay represented by the first GUI image 364A by causing the line monitoring application 130 to update the location and / or depth of the color / patterning, etc., representing the heat map portions 364A1 and 364A2 based on the identified unexpected object 362A in the first image 362. Moreover, the heat map graphical overlays included as part of the first GUI image 364A, the second GUI image 364B, and the third GUI image 364C may indicate areas in the production line that may have been the source of an unexpected object (e.g., unexpected object 362A) in the image (e.g., first image 362).

[0074] Example Notifications 4 illustrates an example user interface 402 that may be presented by a line monitoring application (e.g., line monitoring application 130) via a display (e.g., display 124) including notifications 402A-E to a user in accordance with various aspects disclosed herein. Generally, the line monitoring application 130 may send and cause the display 124 to display a notification to a user / operator in response to one of the applied algorithms / models identifying an unexpected object (e.g., a falling object and / or a stationary object) in video / image data representing the production line and / or the area surrounding the production line. For example, the line monitoring application 130 may send one or more notifications 402A-E including a link to real-time video / image data including the unexpected object displayed as part of the user interface 402. When the user interacts (e.g., clicks, taps, swipes, etc.) with the link included in the notification 402A-E, the line monitoring application 130 may cause the display 124 to display the video / image data including the unexpected object.

[0075] In certain aspects, each of notifications 402A-E may correspond to a different event in which an unexpected object is identified within video / image data of the production line and / or the area surrounding the production line. While shown as text messages in FIG. 4 , some or all of notifications 402A-E may additionally or alternatively be sent by the line monitoring application 130 as email messages and / or as messages within the line monitoring application 130 that a user may access by launching the line monitoring application 130. In this manner, a user / operator may receive a notification that enables the user to access and analyze real-time video / image data corresponding to the identification of the unexpected object and take corrective action, such as executing and / or approving control commands to stop / pause operation of the production line to perform line clearance operations.

[0076] Exemplary Flow Diagram 5 is a flow diagram illustrating an example method 500 for performing line clearance and monitoring in biopharmaceutical processes and applications according to various embodiments disclosed herein. Method 500 may be implemented by one or more components of systems 100A-C, such as processing unit 120, when executing instructions of line monitoring application 130 and possibly biomanufacturing process machine 160 (which may be operating a biomanufacturing process). Method 500 may be or include analyses that are the same as or similar to the example line clearance and monitoring analysis actions performed in FIGS. 3A-3C. Example method 500 may generally include the following elements: (1) receiving a first set of images (block 502); (2) receiving a second set of images (block 504); (3) analyzing the first set of images by applying a first algorithm to identify falling objects within a first FOV (block 506); (4) analyzing the second set of images by applying a second algorithm to identify stationary objects within a second FOV (block 508); and (5) generating a notification for display at a user computing device (block 510).

[0077] Method 500 may include receiving a first set of images of the production line during runtime operation of the production line (block 502). The first set of images may represent a first FOV oriented to capture objects while falling from the production line. Method 500 may also include receiving a second set of images of the production line during runtime operation of the production line (block 504). The second set of images may represent a second FOV different from the first FOV and oriented to capture objects positioned below the production line. In some aspects, method 500 may further include capturing the first set of images and the second set of images by at least one of: (i) a variable zoom imaging device, (ii) a fixed zoom imaging device, (iii) a wide-angle imaging device, and / or (iv) a gyroscopic imaging device.

[0078] Method 500 may further include analyzing the first set of images and the second set of images to identify (i) a falling object in the first FOV or (ii) a stationary object in the second FOV (block 506). Method 500 may further include, in response to identifying the falling object or the stationary object, causing a display to present a notification, where the notification includes an image of the falling object or the stationary object (block 508). In certain aspects, the falling object and the stationary object are the same object, such that the falling object in the first FOV is the same object as the stationary object in the second FOV. Furthermore, in some aspects, a processor performing one or more of the actions included in blocks 502-508 may be a cloud-based processor (e.g., hosted on cloud-based platform 170A4).

[0079] In some aspects, generating the notification further includes, in response to identifying the falling object or the stationary object, triggering, by the one or more processors, recording of a plurality of images from either the first set of images or the second set of images, each image of the plurality of images depicting the falling object or the stationary object, and causing, by the one or more processors, a display to present a notification, the notification including the recording.

[0080] In certain aspects, the method 500 further includes masking a portion of the first image set or the second image set before analyzing the first image set or the second image set, the portion of the first image set or the second image set corresponding to one or more moving components of the manufacturing line.

[0081] In some aspects, generating a notification further includes generating a notification in substantially real time for display on a user computing device in response to identifying the falling or stationary object, the notification comprising at least one of: (i) an email message; (ii) a text message; or (iii) a line monitoring application alert.

[0082] In certain aspects, analyzing the first set of images and the second set of images may further include, by one or more processors, analyzing the first set of images by applying a first algorithm and analyzing the second set of images by applying a second algorithm to identify (i) a falling object within the first FOV or (ii) a stationary object within the second FOV. In these aspects, both the first algorithm and the second algorithm may be, for example, a motion detection algorithm or an ML algorithm / model. Additionally, in these aspects, the first algorithm may be (i) a motion detection algorithm or (ii) a machine learning (ML) algorithm trained with a plurality of training data comprising a plurality of training images representing a production line, where the ML algorithm is configured to receive image data of the production line as input and output an anomaly score corresponding to a confidence level associated with detecting a falling object or a stationary object, and the second algorithm is (i) a motion detection algorithm or (ii) an ML algorithm. Additionally, in these aspects, method 500 may further include training the ML model with a plurality of training images representing the production line, where the plurality of training images represent the production line operating with (i) no falling objects within a first FOV and (ii) no stationary objects within a second FOV. Moreover, in these aspects, the ML algorithm may be at least one of (i) an anomaly detection algorithm, (ii) an image classification algorithm, or (iii) an object detection algorithm.

[0083] In some aspects, the first algorithm and the second algorithm may be included as part of a line monitoring application (e.g., line monitoring application 130), and analyzing the first set of images and the second set of images may be performed by an unexpected item detection unit (e.g., unexpected item detection unit 138) executing instructions comprising the first algorithm and the second algorithm.

[0084] In certain aspects, a notification (e.g., notifications 402A-E) may include a heatmap image with heatmap portions superimposed on an image of a falling or stationary object. The heatmap portions may be positioned over the falling or stationary object in the image. In certain examples, a notification may include multiple images, and the heatmap image may be multiple heatmap images. In these examples, multiple heatmap portions may be superimposed on multiple images, such that a notification may include multiple images from the first image dataset and / or the second image dataset, as well as multiple heatmap images.

[0085] In some embodiments, method 500 may be performed entirely by automation, for example, by one or more processors (e.g., CPUs or GPUs) executing instructions stored on one or more non-transitory computer-readable storage media (e.g., volatile or non-volatile memory, read-only memory, random access memory, flash memory, electronically erasable programmable read-only memory, and / or one or more other types of memory). More generally, method 500 may employ any of the components, processes, or techniques of one or more of FIGS. 1-4.

[0086] Additional considerations Some of the drawings described herein show example block diagrams having one or more functional components. It will be understood that such block diagrams are for illustrative purposes, and that the devices described and shown may have more, fewer, or alternative components than those shown. Also, in various aspects, the components (and the functionality provided by each component) may be associated with or otherwise integrated as part of any suitable component.

[0087] Some aspects of the present disclosure relate to non-transitory computer-readable storage media having instructions / computer-readable storage media for performing various computer-implemented operations. The term “instruction / computer-readable storage medium” is used herein to include any medium capable of storing or encoding a set of instructions or computer code for performing the operations, methods, and techniques described herein. The media and computer code may be those specially designed and constructed for the purposes of the present disclosure, or may be of the kind known and available to those skilled in the computer software arts. Examples of computer-readable storage media include, but are not limited to, magnetic media such as hard disks, floppy disks, and magnetic tape; optical media such as CD-ROMs and holographic devices; magneto-optical media such as optical disks; and hardware devices specially configured to store and execute program code, such as ASICs, programmable logic devices (“PLDs”), and ROM and RAM devices.

[0088] Examples of computer code include machine code, such as that produced by a compiler, and files containing higher-level code that is executed by a computer using an interpreter or compiler. For example, an aspect of the present disclosure may be implemented using Java, C++, or other object-oriented programming language and development tools. Additional examples of computer code include encryption and compression code. Furthermore, aspects of the present disclosure may be downloaded as a computer program product or transferred over a transmission channel from a remote computer (e.g., a server computer) to a requesting computer (e.g., a computer or a different server computer). Other aspects of the present disclosure may be implemented in hardwired circuitry instead of, or in combination with, machine-executable software instructions.

[0089] As used herein, the singular terms "a," "an," and "the" may include plural referents unless the context clearly dictates otherwise. This specification and the claims that follow should be read to include one or at least one, and the singular also includes the plural unless expressly stated or clearly meant otherwise. As used herein, the terms "comprise," "including," "includes," "including," "has," "having," or any other variation thereof, are intended to cover non-exclusive inclusions. For example, a process, method, article, or apparatus that includes a list of elements is not necessarily limited to only those elements but may include other elements not expressly listed or inherent in such process, method, article, or apparatus. Furthermore, unless expressly contrary, "or" refers to an inclusive "or," not an exclusive "or." For example, condition A or B is satisfied by any of the following: A is true (or exists) and B is false (or does not exist), A is false (or does not exist) and B is true (or exists), and A and B are both true (or exist).

[0090] As used herein, the terms "substantially," "substantial," "roughly," and "about" are used to describe and explain slight differences. When used in conjunction with an event or circumstance, these terms may refer to instances in which the event or circumstance occurs exactly, as well as instances in which the event or circumstance occurs approximately. For example, when used in conjunction with a numerical value, the terms may refer to a variation range of ±10% or less of the numerical value, e.g., ±5% or less, ±4% or less, ±3% or less, ±2% or less, ±1% or less, ±0.5% or less, ±0.1% or less, or ±0.05% or less. For example, two numerical values ​​can be considered "substantially" identical if the difference is ±10% or less of the mean of the numerical values, e.g., ±5% or less, ±4% or less, ±3% or less, ±2% or less, ±1% or less, ±0.5% or less, ±0.1% or less, or ±0.05% or less.

[0091] Additionally, amounts, ratios, and other numerical values ​​may be presented herein in a range format. It should be understood that such range format is used for convenience and brevity and should be understood flexibly to include not only the numerical values ​​explicitly stated as the limits of a range, but also to include all individual numerical values ​​or subranges subsumed within that range as if each numerical value and subrange were expressly stated.

[0092] While the techniques disclosed herein have been described primarily in terms of particular operations being performed in a particular order, it will be understood that these operations may be combined, divided into parts, or reordered to form equivalent techniques without departing from the teachings of this disclosure. Accordingly, unless specifically indicated herein, the order and grouping of operations is not a limitation of this disclosure.

Claims

1. 1. A computer-implemented method for performing line clearance and monitoring, comprising: receiving, by one or more processors, a first set of images of a manufacturing line during runtime operation of the manufacturing line, the first set of images representing a first field of view (FOV) oriented to capture an object while it is falling from the manufacturing line; receiving, by the one or more processors, a second set of images of the production line during the runtime operation of the production line, the second set of images representing a second FOV different from the first FOV and oriented to capture objects positioned below the production line; analyzing, by the one or more processors, the first set of images and the second set of images to identify (i) falling objects within the first FOV, or (ii) stationary objects within the second FOV; and in response to identifying the falling object or the stationary object, causing, by the one or more processors, a notification to be presented on a display, the notification including an image of the falling object or the stationary object. Computer-implemented methods.

2. generating the notification triggering, by the one or more processors, in response to identifying the falling object or the stationary object, the recording of a plurality of images from either the first image set or the second image set, each image of the plurality of images depicting the falling object or the stationary object; causing, by the one or more processors, a display to present the notification, the notification including the recording. The computer-implemented method of claim 1 .

3. 3. The computer-implemented method of claim 1, further comprising masking a portion of the first set of images or the second set of images before analyzing the first set of images or the second set of images, the portion of the first set of images or the second set of images corresponding to one or more moving components of the manufacturing line.

4. generating the notification generating the notification in substantially real time for display on a user computing device in response to identifying the falling object or the stationary object, the notification comprising at least one of: (i) an email message; (ii) a text message; or (iii) a line monitoring application alert. A computer-implemented method according to any one of claims 1 to 3.

5. Analyzing the first set of images and the second set of images includes: analyzing, by the one or more processors, the first set of images by applying a first algorithm and the second set of images by applying a second algorithm to identify (i) the falling object within the first FOV, or (ii) the stationary object within the second FOV. A computer-implemented method according to any one of claims 1 to 4.

6. the first algorithm is (i) a motion detection algorithm or (ii) a machine learning (ML) algorithm trained with a plurality of training data comprising a plurality of training images representing the production line, the ML algorithm being configured to receive as input image data of the production line and output an anomaly score corresponding to a confidence level associated with detecting the falling object or the stationary object; The computer-implemented method of claim 5 , wherein the second algorithm is (i) the motion detection algorithm or (ii) the ML algorithm.

7. The computer-implemented method of claim 6 , wherein the ML algorithm is at least one of: (i) an anomaly detection algorithm; (ii) an image classification algorithm; or (iii) an object detection algorithm.

8. 7. The computer-implemented method of claim 6, further comprising: training the ML model using the plurality of training images representing the manufacturing line, the plurality of training images representing the manufacturing line operating with (i) no falling objects within the first FOV, and (ii) no stationary objects within the second FOV.

9. 9. The computer-implemented method of claim 1, wherein the notification includes a heat map image having a heat map portion superimposed on the image of the falling object or the stationary object, the heat map portion being positioned over the falling object or the stationary object in the image.

10. The computer-implemented method of any one of claims 1 to 9, wherein the falling object and the stationary object are the same object.

11. The computer-implemented method of any one of claims 1 to 10, wherein the one or more processors include one or more cloud-based processors.

12. 12. The computer-implemented method of claim 1, further comprising capturing the first set of images and the second set of images with at least one of: (i) a variable zoom imaging device, (ii) a fixed zoom imaging device, (iii) a wide-angle imaging device, and (iv) a gyroscope imaging device.

13. 1. A computer system for performing line clearance and monitoring, comprising: one or more processors; a program memory coupled to the one or more processors, the program memory, when executed by the one or more processors, providing to the computer system: receiving a first set of images of the manufacturing line during runtime operation of the manufacturing line, the first set of images representing a first field of view (FOV) oriented to capture an object while it is falling from the manufacturing line; receiving a second set of images of the production line during the runtime operation of the production line, the second set of images representing a second FOV different from the first FOV and oriented to capture objects positioned below the production line; analyzing the first set of images and the second set of images to identify (i) a falling object within the first FOV or (ii) a stationary object within the second FOV; in response to identifying the falling object or the stationary object, causing a display to present a notification, the notification including an image of the falling object or the stationary object. a program memory storing executable instructions.

14. The instructions, when executed, further cause the one or more processors to: triggering, in response to identifying the falling object or the stationary object, the recording of a plurality of images from either the first image set or the second image set, each image of the plurality of images depicting the falling object or the stationary object; 14. The computer system of claim 13, wherein the notification is generated by causing a display to present the notification, the notification including the record.

15. The instructions, when executed, further cause the one or more processors to:

15. The computer system of claim 13 or 14, further comprising: masking a portion of the first image set or the second image set prior to analyzing the first image set or the second image set, the portion of the first image set or the second image set corresponding to one or more moving components of the manufacturing line.

16. The instructions, when executed, further cause the one or more processors to:

16. The computer system of claim 13, wherein in response to identifying a falling or stationary object, the computer system generates a notification in substantially real time for display on a user computing device, the notification comprising at least one of: (i) an email message; (ii) a text message; or (iii) a line monitoring application alert.

17. The instructions, when executed, further cause the one or more processors to: analyzing the first set of images by applying a first algorithm and analyzing the second set of images by applying a second algorithm to identify (i) the falling object within the first FOV or (ii) the stationary object within the second FOV; the first algorithm is (i) a motion detection algorithm or (ii) a machine learning (ML) algorithm trained with a plurality of training data comprising a plurality of training images representing the manufacturing line; the ML algorithm is configured to receive as input image data of the production line and output an anomaly score corresponding to a confidence associated with detecting the falling object or the stationary object; the second algorithm is (i) the motion detection algorithm or (ii) the ML algorithm; A computer system according to any one of claims 13 to 16.

18. The instructions, when executed, further cause the one or more processors to:

14. The computer system of claim 13, wherein the ML model is trained using the plurality of training images representing the manufacturing line, the plurality of training images representing the manufacturing line operating with (i) no falling objects within the first FOV and (ii) no stationary objects within the second FOV.

19. 1. A tangible, non-transitory computer-readable medium storing executable instructions for implementing line clearance and monitoring, which when executed by one or more processors of a computer system, provides the computer system with: receiving a first set of images of the manufacturing line during runtime operation of the manufacturing line, the first set of images representing a first field of view (FOV) oriented to capture an object while it is falling from the manufacturing line; receiving a second set of images of the production line during the runtime operation of the production line, the second set of images representing a second FOV different from the first FOV and oriented to capture objects positioned below the production line; analyzing the first set of images and the second set of images to identify (i) a falling object within the first FOV or (ii) a stationary object within the second FOV; a tangible, non-transitory computer-readable medium that, in response to identifying the falling object or the stationary object, causes a display to present a notification, the notification including an image of the falling object or the stationary object.

20. Analyzing the first set of images and the second set of images includes: analyzing, by the one or more processors, the first set of images by applying a first algorithm and the second set of images by applying a second algorithm to identify (i) the falling object within the first FOV, or (ii) the stationary object within the second FOV; the first algorithm is (i) a motion detection algorithm or (ii) a machine learning (ML) algorithm trained with a plurality of training data comprising a plurality of training images representing the manufacturing line; the ML algorithm is configured to receive as input image data of the production line and output an anomaly score corresponding to a confidence associated with detecting the falling object or the stationary object; 20. The tangible, non-transitory computer-readable medium of claim 19, wherein the second algorithm is (i) the motion detection algorithm or (ii) the ML algorithm.

Images