Method and apparatus for detecting anomalies when filling a container with fluid - Patents.com

Abstract

The technology described herein relates to computerized methods and apparatus for detecting anomalies in a container using one or more images of a video of a filling process in which the container is filled with a liquid. The technology described herein further relates to computerized methods and apparatus for detecting anomalies in real time while the container is being filled with a liquid, and, upon detection of at least one anomaly, outputting an alert or altering one or more parameters of a filling process that the filling machine is configured to perform.

Description

[Background technology]

[0001] Automated techniques can be used to fill containers with solutions. For example, in an assembly line for manufacturing liquid drug products, glass bottles can be filled using a needle that injects the drug product into the bottle. In some implementations, the needle is inserted almost all the way into the bottle and the liquid drug is pumped into the bottle. Once the bottle is completely filled, the needle is withdrawn from the bottle. Different types of "bad" fills can occur during the filling process. For example, as the liquid drug is pumped out of the needle, it may splash high enough to leave traces of drug product on the needle. In this case, the bottle being filled is underdosed. These traces can later drip into other bottles being filled, causing the other bottles to be overdosed. These defects can be detected by a quality control evaluation process that occurs after the filling process. At that point, the source of the traces from anomalies that occurred during the filling process cannot be identified in a timely manner, causing the filled bottle to be discarded. Summary of the Invention

[0002] The present disclosure relates to techniques for detecting anomalies when filling a container with a fluid. The technique provides a computerized method for detecting anomalies in a filling process in which a container is filled with a fluid using a filling component. The method can include capturing video of the container while the container is being filled with fluid through the filling component, analyzing images of the video using one or more processors to detect anomalies including one or more of fill component submersion, splashing in the container, dripping in the container, or unsafe fill component-fluid distance in the container, and generating an alert indicating the detected anomaly.

[0003] The method may further include adjusting the filling machine that filled the container with the fluid by modifying one or more parameters of a filling process that the filling machine is configured to perform based on the detected anomaly. The method may also detect anomalies in the filling process at a video level. For example, the method may include analyzing additional images of the video and determining that the filling process has a fill component immersion anomaly if any of the images or the additional images are determined to show a fill component immersion anomaly, determining that the filling process has a splash anomaly if any of the images or the additional images are determined to show a splash anomaly, or determining that the filling process has an unsafe fill component-fluid distance if (1) none of the images or the additional images are determined to show fill component immersion and (2) at least one of the images or the additional images is determined to show a minimum vertical distance between the fill component and the fluid is less than a threshold distance.

[0004] Various anomalies may be detected during the filling process. In one embodiment, the technology provides a computerized method for using images of the filling process to detect either a risk of fill component immersion, splashing in the container, or unsafe fill component-fluid distance anomalies. The method may include identifying the fill component in the image, identifying a fluid in the container in the image, and detecting the anomaly based on a relative position of the fill component and the fluid and / or a comparison of a width of the fill component and a width of the fluid.

[0005] In one embodiment, the technology provides a computerized method for detecting splashing in a container based on a fluid characteristic near a fill component, where the fluid characteristic is an elevated surface of the fluid. In one embodiment, the technology provides a computerized method for detecting a risk of an unsafe fill component-fluid distance when the fill process is nearing completion. When the image being analyzed corresponds to a time when the fill process is nearing completion, the method can detect one or more anomalies by identifying the fill component in the image, identifying a fluid in the container in the image, determining a distance between the fill component and the fluid, and detecting an unsafe fill component-fluid distance when the distance between the fill component and the fluid is determined to be below a threshold distance.

[0006] In one embodiment, the technology provides a computerized method for detecting dripping using images / videos of a filling process using a machine learning model. The method includes providing either the image or additional images in the video to a trained machine learning model and using the trained machine learning model to determine one of a plurality of classifications, where at least one of the plurality of classifications is indicative of dripping in either the image or the additional images. The method may also include training the machine learning model by obtaining a plurality of training videos and training a neural network model using the plurality of training videos. Each of the plurality of training videos may include a plurality of training images capturing an associated fluid filling process and each labeled with one of the plurality of classifications. [Brief description of the drawings]

[0007] Additional embodiments of the present disclosure, as well as its features and advantages, will become more apparent by reference to the description herein taken in conjunction with the accompanying drawings, in which components are not necessarily drawn to scale. Moreover, in the drawings, like reference numerals designate corresponding parts throughout the different views. [Figure 1] 1 is a schematic diagram of a system for detecting anomalies while filling a container with liquid, according to some embodiments. [Figure 2A] 1A-1C illustrate examples of splashes that may occur during the filling process, according to some embodiments. [Figure 2B] 1A-1C illustrate examples of splashes that may occur during the filling process, according to some embodiments. [Figure 3A] 1A-1C show examples of splashes that may occur during the filling process. [Figure 3B] 3B illustrates an example of the tilt of the surface in detecting splashing of FIG. 3A according to some embodiments. [Figure 4A] 1A-1C are diagrams illustrating examples of bounding boxes when detecting splashes, according to some embodiments. [Figure 4B] FIG. 4B illustrates the example bounding box of FIG. 4A superimposed on an image of the filling process in which the bounding box of FIG. 4A is detected, according to some embodiments. [Figure 5A] 13 is an example image of a filling process showing no dripping of liquid according to some embodiments. [Figure 5B] 11A-11C show example images of the filling process showing dripping in the container, according to some embodiments. [Figure 5C] 11A-11C show example images of the filling process showing dripping in the container, according to some embodiments. [Figure 5D] 1 is an exemplary saliency map of an image of a filling process, where samples in the saliency map are shown with higher intensity around the drop, according to some embodiments. [Figure 6A] 1 is an example image of a filling process with dripping, according to some embodiments. [Figure 6B] FIG. 6B illustrates the interface region between the filling component and the fluid when detecting a drop from the image of FIG. 6A. [Figure 7A] 13 is an exemplary image of a filling process showing filled component immersion, according to some embodiments. [Figure 7B] 7B shows the fill components and fluid detected from the image of FIG. 7A, respectively. [Figure 7C] 7B shows the fill components and fluid detected from the image of FIG. 7A, respectively. [Figure 8A] 13A-13C are exemplary images of a filling process having various unsafe filling component-fluid distances according to some embodiments. [Figure 8B] 13A-13C are exemplary images of a filling process having various unsafe filling component-fluid distances according to some embodiments. [Figure 9] 1 is a diagram illustrating an exemplary convolutional neural network that may be implemented to detect drips using images of the filling process, according to some embodiments. [Figure 10] 1 is a flowchart illustrating an exemplary computerized method for using video of the filling process to detect anomalies and adjust the filling machine based on the detected anomalies, according to some embodiments. [Figure 11] 1 is a flowchart illustrating an exemplary computerized method for detecting various types of anomalies using images of the filling process, according to some embodiments. [Figure 12] 1 is a flowchart illustrating an exemplary computerized method for detecting splash anomalies using images of a filling process, according to some embodiments. [Figure 13] 1 is a flow chart illustrating an exemplary computerized method for detecting safe / unsafe filling component-fluid distances using video of the filling process, according to some embodiments. [Figure 14] 1 is a flowchart illustrating an exemplary computerized method for detecting drips using images of the filling process, according to some embodiments. [Figure 15A]1 is a flow chart illustrating an exemplary computerized method for detecting fill components in an image of a filling process, according to some embodiments. [Figure 15B] 1 is a flowchart illustrating an exemplary computerized method for detecting fluid in an image of a filling process, according to some embodiments. [Figure 16] FIG. 1 illustrates an exemplary implementation of a computer system that may be used to implement any of the aspects of the techniques and embodiments disclosed herein, according to some implementations. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] For the purposes of promoting an understanding of the principles of the present disclosure, reference will now be made to the embodiments illustrated in the drawings and specific language will be used to describe the same, it being understood nevertheless that no limitation of the scope of the invention is intended.

[0009] During a filling process for filling a container with liquid, a filling component (e.g., a needle or other component) is inserted into a container, such as a glass bottle, to fill the container with a liquid product, such as a medical fluid and / or other type of fluid. For example, a filling machine can be configured to insert the filling component into the container (e.g., into the majority of the container) and pump liquid into the container. Once the container is filled fully with liquid, the filling component is withdrawn out of the container accordingly (e.g., when the fluid level rises in the container). Once filled, a plunger can be inserted into the container to seal the container.

[0010] Various types of anomalies can occur during the filling process. For example, when liquid is pumped from the filling component into the container, the liquid may splash high enough to leave traces of the liquid product on the filling component. In another example, if the filling component is immersed in the fluid being filled, traces of the liquid may also be left on the filling component. If this is not detected and corrective action is not taken in a timely manner, it may result in the container filled using the filling process having to be subsequently discarded. In a drug situation where a needle is used to fill a bottle with fluid, for example, traces of fluid left on the needle may cause the bottle being filled to be under-dosed. These traces may later drip into other bottles being filled, causing the other bottles to be over-dosed. Under quality control protocols, containers with such defects may have to be discarded.

[0011] Another example of a filling anomaly is when liquid is pumped out of the filling component, it may splash high enough to leave droplets on the side of the container. When the plunger or stopper is subsequently inserted into the filled container, this droplet may become trapped between the side of the plunger / stopper and the container. If this anomaly is not detected in time, it may result in the container being discarded at a later time. For example, under quality control inspection, if droplets of drug product are spotted between the side of the plunger / stopper and the container, the container may need to be discarded because the manufacturer may not be able to determine if the droplets occurred due to splashing during filling, or because the plunger / stopper did not seal properly and is therefore defective (and therefore at risk of product spoilage).

[0012] The present inventors recognize and understand that conventional filling systems are unable to detect anomalies in real time while the containers are being filled. In such systems, containers on a filling assembly line may undergo quality control inspection after they are filled with liquid and moved out of the filling station. At that point, the filling process is complete and therefore the filling components are withdrawn from the containers and the containers are often sealed with a plunger or stopcock. As previously mentioned, anomalies that occur during the filling process (e.g., liquid splashing onto the containers and / or filling components) often result in the disposal of those containers because artifacts left from the anomalies in the filling process cannot be properly identified after the filling process is completed. If anomalies that occur during the filling process are not detected in a timely manner, the filling process continues uncorrected until inspection, which may occur at a much later stage, which may result in the waste and / or disposal of an entire batch of containers that have been filled over a long period of time.

[0013] In order to solve the various technical problems discussed in this disclosure, the inventors recognize and understand that detecting anomalies occurring inside a container can be useful not only in detecting anomalies in a laboratory environment, but also in detecting anomalies in a production environment.

[0014] The inventors recognize and understand that conventional container filling systems also never provide a way to automatically adjust the filling process in response to a filling anomaly. Instead, human monitoring and control are often used to determine and develop a filling profile. A filling profile can include a set of parameters that govern how the filling machine fills the container in the filling process. Typical parameters include the speed at which the fluid is pumped, the movement and / or speed of the filling components, the movement and / or speed of the container on the assembly line, the viscosity of the fluid, and / or the amount of fluid filled. Automated monitoring and control can provide greater consistency when developing a filling profile. Once these filling profiles are developed in the laboratory, the recommended parameters are sent to production. In this way, any errors that cause anomalies in the container are not detected or corrected in a timely manner. For example, errors in the filling process that systematically result in a faulty fill may not be corrected or even detected before the contents are discarded. To solve these technical problems, the inventors have realized and understood that detecting anomalies occurring in a container can be useful not only in detecting anomalies early in the process, but also in automatically adjusting the filling process to avoid further anomalies. For example, the position of the filling component relative to the surface of the fluid in the container can give an indication as to whether an anomaly may occur. If the filling component is too close to the liquid being filled in, it can be an indication that immersion of the filling component in the liquid may occur. If the filling component is too far from the liquid being filled in, it can be an indication that splashing in the container may occur. In another example, if dripping in the container is detected, it can be an indication that the speed of filling is too slow. All of these anomalies occurring in the container can be used to automatically adjust the filling machine.

[0015] Accordingly, the present inventors have developed a new technique for detecting anomalies in a filling process in real time using a video of the filling process while the container is being filled with liquid. Described herein are various techniques, including systems, computerized methods, and non-transient instructions, that process high speed video of a filling process performed by a liquid filling station or assembly line. The system can analyze one or more images of the video to monitor and / or detect anomalies during the filling process. In response to detecting an anomaly, the system can output a signal or warning message, or modify one or more parameters of the filling process that the filling machine is configured to perform.

[0016] In some embodiments, a system may be provided to analyze one or more image frames of a video of a filling process to detect one or more anomalies in the filling process. In some examples used to illustrate the technology herein, the system may include a filling machine configured to perform a filling process in which a container is filled with a fluid using a filling component. The container may be transparent, such as a glass bottle. The fluid may be, for example, a flowable drug product. The system may also include an image capture device, such as a high-speed camera, configured to capture a video of the container while it is being filled with the fluid. The system may further include at least one processor configured to execute programming instructions to process the captured video by analyzing one or more images of the video to detect one or more anomalies in the filling process. Examples of anomalies to be detected may include filling component immersion when the filling component is immersed in the liquid being filled right. Anomalies may also include splashing in the container, dripping in the container, or an unsafe filling component-fluid distance in the container. When an anomaly is detected, the system may generate an alarm indicating the detected anomaly. The system may also adjust the filling machine by modifying one or more parameters of the filling process that the filling machine is configured to perform to reach an optimal filling profile based on the detected anomaly. For example, the system can predict optimal machine parameters based on the detected anomaly and automatically transmit the predicted machine parameters to the filling machine.

[0017] The inventors have also developed various embodiments in which the system may be configured to detect various anomalies in the container while it is being filled. In some embodiments, the system may analyze one or more images in a video of the filling process to determine whether the filling process has an anomaly. For example, the system may determine that the filling process has a fill component immersion anomaly if any of the images in the video are determined to have a fill component immersion anomaly. In another example, the system may determine that the filling process has a splash anomaly if any of the images in the video are determined to show a splash anomaly or if the number of images in the video having a splash anomaly is greater than a threshold number. In another example, the system may determine that the filling process has an unsafe fill component-fluid distance if at least one of the images in the video is determined to show an unsafe fill component-fluid distance. In another example, the system may determine that the filling process has a drip anomaly if any of the images in the video are determined to show a drip anomaly. Embodiments in detecting various types of anomalies are further described in this disclosure.

[0018] In some embodiments, the system may identify a fill component in the image and identify a fluid in the container in the image. The identified fill component and fluid may each be represented by a respective bounding box. The system may detect the anomaly based on the relative positions of the fill component and the fluid. For example, the system may detect a fill component immersion anomaly when a bounding box of the fill component overlaps with a bounding box of the fluid and a width of the bounding box of the fluid is greater than a width of the bounding box of the fill component by a threshold width difference. Additionally and / or alternatively, the system may detect a fill component immersion anomaly in response to determining that filling has begun, e.g., the elapsed time in the filling process is greater than a threshold fill time.

[0019] In some embodiments, the system can detect splashing in the container when it is determined that the bounding box of the fill component overlaps with the bounding box of the fluid and the width of the bounding box of the fluid is not greater than the width of the bounding box of the fill component by a threshold width difference. In some embodiments, the system can detect splashing in the container when it is determined that (i) the bounding box of the fill component does not overlap with the bounding box of the fluid, (ii) the vertical distance between the fill component and the fluid is less than a threshold distance, and (iii) the width of the bounding box of the fluid is not greater than the width of the bounding box of the fill component by a threshold width difference. Additionally and / or alternatively, the system can detect the splashing anomaly in response to determining that filling has just started, e.g., the elapsed time in the filling process is less than a threshold fill time.

[0020] In some embodiments, the system can detect splashing in the container by inspecting fluid properties in a region around the filling component. In some examples, the region around the filling component can include a first region on one side of the filling component and a second region on the other side of the filling component. The first region and the second region can be defined to be positioned vertically between the filling component and the fluid. The width of the first region or the second region can be a fixed width. Alternatively, the width of the first region or the second region can be proportional to the width of the filling component. Similarly, the height of the first region or the second region can be a fixed height or can be proportional to the vertical distance between the filling component and the fluid. Thus, the system can detect a splash anomaly if at least one of the first region or the second region exhibits splashing. In some embodiments, the fluid properties can include a slope (or rise) in the surface of the fluid. Thus, the system can detect splashing in a region if the slope in the surface of the fluid in the region is determined to exceed a threshold value. In some examples, the system may determine a slope at the surface of the fluid in the region based on surface edge pixels in the region, which may be determined by applying an edge detector, for example a Canny operator, to regions in the image where fill components and fluids are identified.

[0021] In some embodiments, the system can determine a region around the fill component in response to determining that the fill component is fully inserted into the container in the filling process. For example, the system can determine that the fill component is fully inserted into the container if the variation in the fill component width in several images in a video of the filling process is below a threshold variation. In some embodiments, the system can determine that the fill component is fully inserted into the container in response to determining that filling has begun, e.g., fluid has been detected in the images being analyzed.

[0022] In some embodiments, the system can detect an unsafe fill component-fluid distance if it is determined that (i) the bounding box of the fill component does not overlap with the bounding box of the fluid, (ii) the vertical distance between the fill component and the fluid is less than a threshold distance, and (iii) the width of the bounding box of the fluid is greater than the width of the bounding box of the fill component by a threshold width. In other variations, the system can determine that the anomaly is an unsafe fill component-fluid distance at a point when the filling process is nearing completion. In some examples, the system can determine the variation in the height of the surface of the fluid in several consecutive images in the video. If the variation in the height of the fluid in several consecutive images is below a threshold, the system can determine that the filling process is nearing completion. Once it is determined that the filling process is nearing completion, the system can determine the distance between the fill component and the fluid. If the distance between the fill component and the fluid is below a lower threshold distance, the system can determine an unsafe fill component-fluid distance, indicating that the distance between the fill component and the fluid is too small. If the distance between the fill component and the fluid exceeds an upper threshold distance, the system can determine an unsafe fill component-fluid distance, indicating that the distance between the fill component and the fluid is too large. If the distance between the fill component and the fluid is between a lower threshold distance and an upper threshold distance, the system can determine a safe fill component-fluid distance.

[0023] In some embodiments, the system can detect dripping in the container using a machine learning model. In some examples, the machine learning model can be configured to obtain as input an image in a video of a filling process, where the image includes the container and the filling component in the filling process. The machine learning model can be used to determine one of a plurality of classifications, where at least one of the plurality of classifications indicates dripping in the image. In some examples, the machine learning model can include a neural network model that can be trained over a training dataset. In some embodiments, the training process can obtain a plurality of training videos, where each training video captures an associated fluid filling process. Each training video can include a plurality of training images (e.g., frame images) each labeled with one of a plurality of classifications, where one of the classifications indicates dripping in the training image. The training process can train the machine learning model using the training images from the training videos and the labels associated with each training image. The trained machine learning model can include trained weights, e.g., trained weights of a neural network model. An example of a neural network model can include a convolutional neural network (CNN) or any variation thereof. The training process may use any training algorithm, such as, for example, gradient descent or any other suitable algorithm.

[0024] In some examples, the training process can provide a portion of each of the training images instead of the entire image to train the machine learning model. Similarly, a portion of the image can be provided to the trained machine learning model to detect anomalies. In this way, an image having a reduced size is provided to the machine learning model, resulting in less calculations and shorter processing time to detect anomalies. In some embodiments, the portion of the image to provide to the machine learning model can be determined by reducing the field of view of the image. In some examples, the portion of the image to be provided to the machine learning model can be determined based on the intensity of the image. In various embodiments, determining the portion of the image can be performed on an image in a video of the filling process. In other embodiments, the portion of the image for the machine learning model can be determined by an image capture sensor, which is configured to determine an appropriate field of view and generate the portion of the image based on the determined field of view. For example, based on the intensity of the image, the image sensor can determine a central field of view of the image and generate an image including the central field of view.

[0025] In various embodiments described herein, the system may process the image to identify the filling component and / or fluid, where the identified filling component and / or fluid may be used by other processes to detect anomalies in the filling process. In identifying the filling component and / or fluid in the image, the system may identify edge pixels in the image and determine one or more boundary regions based on the edge pixels. In some examples, pixel values ​​for the filling component and the fluid in the image may be different. For example, the fluid may be shown in a light color and the filling component may be shown in a dark color. Thus, in identifying the filling component and / or fluid, the system may perform different operations to identify edge pixels in the image. For example, in identifying the filling component, the system may subtract the image from a reference image in the video of the filling process. In a non-limiting example, the reference image in the video may be the first image in the filling process. In identifying the fluid, the system may subtract the reference image in the video from the current image. Optionally, the system may apply a filter to remove noise from the subtracted image while preserving the edge pixels. Optionally, the system may binarize the subtracted image or the filtered subtracted image before identifying edge pixels.

[0026] In some embodiments, the system can further process the edge pixels to generate one or more boundary regions, where each of the boundary regions includes connected edge pixels, such as, for example, 4-connected or 8-connected edge pixels. Various tracking algorithms can be used to find the connected edge pixels. For each of the boundary regions, the system can determine whether the size of the boundary region exceeds a respective threshold for a fill component or a fluid. If the size of the boundary region exceeds a respective threshold, the system can select the boundary region as a fill component or a fluid. In some embodiments, for a selected boundary region, the system can determine a bounding box that is the smallest rectangular box that encompasses the selected boundary region. In some examples, the threshold for selecting a fill component and the threshold for selecting a fluid may be different.

[0027] In some embodiments, variations in identifying the fill component and / or fluid may be possible. For example, when subtracting the images described above, the system may binarize the images before generating the subtracted image. In other variations, the system may identify the fill component or fluid by selecting a boundary region that has the largest area among multiple boundary regions, rather than selecting a boundary region that has an area above a threshold area.

[0028] The techniques described herein can provide advantages over conventional filling systems in improving the performance of filling containers, e.g., glass bottles, by detecting anomalies in real-time while the container is being filled with liquid. For example, by using the machine vision and / or machine learning techniques disclosed in the present invention, the system can detect various types of anomalies in high-speed video. These anomalies can include splashing in the container, dripping in the container, fill component immersion, or unsafe fill component-liquid distances. Detecting these anomalies can trigger the system to generate an alert to a user (e.g., an operator) so that the operator can take corrective action in a timely manner. For example, the operator can adjust the flow of fluid from the fill component to reduce splashing and / or dripping. The operator can also adjust the height of the fill component or the timing of inserting / withdrawing the fill component to avoid unsafe fill component-liquid distances.

[0029] Alternatively and / or additionally, the system may automatically modify one or more parameters of the filling process based on the detected anomaly. For example, in response to detecting splashing in the filling process, the system may modify one or more parameters of the filling process to cause the filling station to adjust the flow of liquid from the filling component, the height of the filling component, or the timing of inserting / withdrawing the filling component, or a combination thereof. Similarly, in response to detecting dripping, filling component immersion, and / or an unsafe filling component-fluid distance in the filling process, the system may modify one or more parameters of the filling process to cause the filling station to adjust the flow of liquid from the filling component, the height of the filling component, or the timing of inserting / withdrawing the filling component, or a combination thereof. Thus, various embodiments described herein may enable the filling system to detect and correct errors in the filling process in time before the container can be discarded as in conventional systems at a later stage during inspection.

[0030] While various embodiments have been described, it will be apparent to those skilled in the art that many more embodiments and implementations are possible. Thus, the embodiments described herein are examples and are not the only possible embodiments and implementations. Furthermore, the advantages described above are not necessarily the only advantages, and it is not necessarily expected that all of the advantages described will be achieved in each embodiment.

[0031] FIG. 1 is a schematic diagram of a system for detecting anomalies while filling a container with a liquid, according to some embodiments. In some embodiments, the system 100 can include a filling machine 102 configured to perform a filling process in which a container 104 is filled with a fluid using a filling component 108. The container can be any holder capable of holding a fluid or any substance inside in a liquid state. The container can be transparent. Examples of containers can include transparent drug product containers, glass vials, bottles, jars, and / or any suitable holder. The filling component can be any component capable of filling a container with a liquid. For example, the filling component can be a filling tube, needle, pipe, nozzle, funnel, or any other component that can be used to direct a fluid into the container. The fluid can be a fluid drug product, e.g., a pharmaceutical fluid. Other examples of fluids can include water, beverages, oils, foods, chemicals, solutions, compounds, and / or any substance in a liquid state. In one example, the container 104 can be a glass vial for holding a fluid drug product. The filling component 108 can be a needle. For example, in a filling process, fluid may be injected into the container 104 via a needle 108 and a filling pump 106 that pumps the fluid from a reservoir (not shown) through the filling component 108 and into the container 104. As shown in Figure 1, the needle may be inserted into the container 104 while the filling process is performed. Once the filling process is completed, the needle may be removed from the container 104.

[0032] The system 100 may also include an image capture device 112, such as a camera, configured to capture a video of the filling process. Thus, the video may include a series of images, each of which may include the filling component and the container being filled with the fluid. The system may further include a server 110 having at least a processor, a memory, a storage medium, and / or other components. The at least one processor may be configured to execute programming instructions stored in the memory to process the captured video and detect one or more anomalies in the filling process by analyzing one or more images of the video. In some examples, the filling process from start to finish may be completed in a short period of time, such as a few seconds. Thus, in some examples, the image capture device 112 may be a high-speed camera that may capture a video of the filling process.

[0033] The server 110 can be configured to process the captured video during the filling process in real time. In a non-limiting configuration, the container 104 can be placed on the filling platform 103. For example, the filling platform 103 can be a conveyor belt on an assembly line, a nest and tub, or a turntable on which multiple containers to be filled are placed, where the filling platform can be controlled to move the containers. Each time the filling platform moves a container to a location for filling, the container on the filling platform 103 is engaged with the filling component 108 such that the filling component 108 is inserted into the container to fill it with fluid. Once the filling process is completed, the filling component is withdrawn from the container and the filling platform is moved to move the next container into position and the filling process for the next container is started, and so on. The server 110 can be configured to process the captured video during such a filling process. Although the example of FIG. 1 shows only one filling component 108, it should be understood that multiple filling components can be configured to fill multiple containers, if desired.

[0034] The server 110 may be configured to process the captured video to monitor for potential anomalies that may occur during the filling process. Examples of anomalies may include fill component immersion, which may occur when the fill component is immersed in liquid during filling. Anomalies may also include splashing in the container, dripping in the container, or unsafe fill component-fluid distance in the container (e.g., as the filling process nears completion). A splash anomaly may include splashing of fluid in the container, for example, on the neck of the container between the stopcock / plunger and the glass. This splash anomaly may result in bubbling or drying at the tip of the needle. A drip anomaly may include dripping of fluid in the container, for example, dripping onto the fill component. An unsafe fill component-fluid distance anomaly may include a scenario in which the fill component comes within an unsafe distance of the surface of the fluid in the container. This unsafe distance may be outside of a safe distance range, for example, the safe distance range may be between a low threshold distance and a high threshold distance. Other anomalies can include high velocity fluid splashing out of the vial, droplets landing on the filling components, and dripping into the next vial, causing under-dose / over-dose scenarios.

[0035] In some non-limiting examples, various types of anomalies are shown. For example, FIG. 2A shows an example of a glass vial 202 and a filling component 204 (e.g., a needle) and minimal splashing of fluid 206 that may occur during the filling process. FIG. 2B shows an example of more severe splashing of fluid 206 compared to that shown in FIG. 2A, where the fluid 206 splashes above the filling component 204, and FIG. 3A shows an example of moderate splashing of fluid 306 onto the side of the vial 302 as the fluid is being filled by the filling component 304 (e.g., a needle). FIG. 5A shows an example image of the filling process for a glass vial 502 and a filling component 504 (e.g., a needle) that does not show any dripping of fluid 506. For comparison, FIG. 5B and FIG. 5C both show dripping of fluid 508 and 510, respectively. 7A shows an example of fill component immersion when a fill component 704 (e.g., needle) is immersed in fluid 706 in a vial 702. Figures 8A and 8B show example images of a fill process with various unsafe fill component-fluid distances. For example, the distance between fill component 804 (e.g., needle) and fluid 806 shown in Figure 8A is too large, which causes dripping and / or splashing 808, whereas the distance between fill component 814 (e.g., needle) and fluid 816 shown in Figure 8B is too small at the point where the fill component is touching or about to touch the fluid, which may be caused by one or more parameters in the fill profile.

[0036] Returning to FIG. 1 , when a potential occurrence of an anomaly is detected, the system can generate an alert indicating the detected anomaly. The alert can include any suitable form and can be communicated to the user device over any suitable communication protocol to indicate to the user that an anomaly has been detected. For example, the server 110 can send a notification, warning message, or report to the user device 114 over the communication line. The notification or report can include information about the anomaly and / or the type of anomaly. In some examples, the system can also adjust the filling machine by modifying one or more adjustment parameters of the filling process, which the filling machine is configured to implement to reach an optimal filling profile, based on the detected anomaly. For example, the server 110 can predict optimal machine parameters based on the detected anomaly and automatically send the predicted machine parameters to the filling machine 102.

[0037] In some embodiments, the system 100 can be configured such that detection and / or reporting of anomalies can be performed in real-time. For example, in response to detecting an anomaly while a container is being filled with fluid, the system can generate an alert to a user (e.g., an operator) by sending the alert to the user device 114 so that the operator can take corrective action in a timely manner. For example, the operator can adjust the flow of fluid from the filling component to reduce splashing and / or dripping. The operator can also adjust the height of the filling component or the timing of inserting / withdrawing the filling component to avoid unsafe filling component-liquid distances.

[0038] In another example, the server 110 can be configured to adjust one or more parameters of the filling station 102 during the filling process to achieve an optimal set of machine parameters that mitigate the occurrence of a possible anomaly (e.g., such that an anomaly may occur during filling one container, but by adjusting parameters in real time during the filling process, the anomaly can be mitigated during the filling process of the next and / or subsequent containers). For example, in response to detecting splashing in the filling process, the server can modify one or more parameters of the filling process to cause the filling station to adjust the flow of liquid from the filling component, the height of the filling component, or the timing of inserting / withdrawing the filling component, or a combination thereof. Similarly, in response to detecting dripping, filling component submersion, and / or an unsafe filling component-fluid distance in the filling process, the system can modify one or more parameters of the filling process to cause the filling station to adjust the flow of liquid from the filling component, the height of the filling component, or the timing of inserting / withdrawing the filling component, or a combination thereof. Thus, various embodiments described herein may enable a filling system to detect and correct errors in the filling process in time before the container may be discarded as in conventional systems at a later stage during inspection. Details of methods that may be implemented in system 100 for detecting anomalies during the filling process are further described with reference to Figures 10-15.

[0039] FIG. 10 is a flow chart illustrating an exemplary computerized method 1000 for detecting one or more anomalies using a video of the filling process and adjusting a filling machine based on the detected anomalies, according to some embodiments. In some embodiments, the method 1000 can be implemented within the system 100, for example, performed by the server 110. In some examples used to illustrate the techniques herein, the method 1000 begins with act 1002, where a video of a container is captured while the container is being filled. The video can include multiple frame images captured over time. The method 1000 can further analyze these images in act 1004 to detect possible occurrences of various anomalies. For example, act 1004 can include act 1006 to detect fill component immersion, act 1008 to detect splashing, act 1010 to detect unsafe fill component-fluid distance, and / or act 1012 to detect dripping. Exemplary techniques that can be used to detect anomalies are further described in FIGS. 11-15.

[0040] With further reference to FIG. 10 , in act 1014, method 1000 may determine whether any anomalies were detected from act 1004. In response to determining one or more anomalies, method 1000 may proceed to act 1016, which is adjusting the filling machine based on the one or more anomalies detected. Additionally and / or alternatively, act 1016 may include sending an alert, e.g., a notification, a warning message, and / or a report, to a user device via a communication line. The alert may include information about the anomaly (e.g., a time of the anomaly, a bottle or container associated with the anomaly, and / or a type of anomaly). Exemplary details of act 1016 are previously described in the embodiment of FIG. 1 . Upon completion of act 1016, method 1000 may proceed to act 1018. Method 1000 may end the process (e.g., upon completion of a set of fills) or continue the process by returning to act 1002. Similarly, in response to determining at act 1014 that no anomaly is detected, method 1000 may also return to act 1002 to capture additional video of the filling process and continue monitoring for anomalies in the video at act 1004.

[0041] FIG. 11 is a flow chart illustrating an exemplary computerized method 1100 for detecting various types of anomalies using images of a filling process, according to some embodiments. In some embodiments, the method 1100 can be implemented in act 1004 of FIG. 10 and / or can be performed in the system 100 of FIG. 1, such as by the server 110, for example. In some examples used to illustrate the techniques herein, the method 1100 begins with acts 1102, 1104 to identify a filling component and a fluid in an image, respectively. The images may be one or more frame images in a video, such as the video of the filling process captured in act 1002 of FIG. 10, as described herein. Thus, each image of the filling process may include a container, a filling component inserted in the container, and a fluid in the container, where the fluid is filled by the filling component in the filling process. Exemplary techniques that can be used to identify the filling component and the fluid will be further described with reference to FIGS. 15A and 15B.

[0042] With further reference to FIG. 11 , method 1100 may proceed to detect anomalies based on the relative positions of the fill component and the fluid. For example, in act 1106, method 1100 may determine whether the fill component overlaps with the fluid. In some embodiments, the identified fill component and fluid may each be represented by a respective bounding box. In the example of FIGS. 4A and 4B, a bounding box of the fill component 420 and a bounding box of the fluid 422 are identified from the image shown in FIG. 4B. In determining whether the fill component overlaps with the fluid, act 1106 may determine whether the bounding box of the fill component overlaps with the bounding box of the fluid. In response to determining that the fill component overlaps with the fluid, method 1100 may proceed to act 1108, which determines whether a width of the fluid is greater than a width of the fill component by a threshold width. For example, act 1108 may compare a width of the bounding box of the fill component to a width of the bounding box of the fluid. In response to determining that the width of the fluid bounding box is greater than the width of the fill component bounding box by a threshold width T, the method 1100 may proceed to act 1110 of determining that the image has a fill component immersion anomaly. The threshold width T may be expressed as a number of pixels or a number of distance units (e.g., millimeters, inches, etc.). For example, in some embodiments, the width of the fluid bounding box may be approximately 55-59 pixels, the width of the fill component bounding box may be approximately 12-14 pixels, and the threshold width T may be set at 8 pixels.

[0043] In some embodiments, a threshold width ratio that specifies a minimum required ratio of the width of the fluid bounding box to the width of the infill component bounding box may be used in place of the threshold width T. For example, once the width of the infill component bounding box is determined, act 1108 may determine whether the width of the fluid bounding box is at least 2.5 times greater than the width of the infill component bounding box. That is, the required ratio between the width of the fluid bounding box and the width of the infill component bounding box may be at least 2.5 to 1. In other embodiments, other suitable ratios may be used. By using a threshold width ratio, the determination in act 1108 may be made independent of the camera resolution and the camera distance from the infill component and fluid being imaged. Other methods of comparing the width of the fluid bounding box to the width of the infill component bounding box may also be used.

[0044] The inventors recognize and appreciate that fill component soaking may occur only after the filling process has begun for some time. Thus, optionally, the system may detect the fill component soaking anomaly in response to further determining that filling has begun. For example, the method 1100 may determine that the filling process has begun if the elapsed time in the filling process is greater than a threshold fill time. The threshold fill time may vary depending on the size of the container, the fill speed, or other filling parameters. For example, the faster the fill speed, the shorter the threshold fill time. In other embodiments, the method 1100 may determine the filling process based on a signal from a system filling the container that the system has begun the fill process.

[0045] Returning to act 1108, if the width of the bounding box of the fluid is not greater than the width of the bounding box of the fill component by the threshold width T (or a required threshold width ratio), then the scenario may be that the fluid does not take up the entire container width, such as bounding box 422 in FIG. 4B. Instead, what the system identifies as the fluid or fill component may be a splash. Thus, in response to determining that the width of the bounding box of the fluid is not greater than the width of the bounding box of the fill component by the threshold width T, method 1100 may detect a splash in the container at act 1116.

[0046] Returning to act 1106, in response to determining that the bounding box of the fill component does not overlap with the bounding box of the fluid, the method 1100 may proceed to act 1112 to further determine whether the vertical distance between the fill component and the fluid is less than a threshold distance T1. When the bounding box of the fill component does not overlap with the bounding box of the fluid, the vertical distance between the fill component and the fluid may be measured as the distance between the bottom edge of the bounding box of the fill component and the top edge of the bounding box of the fluid. In the above example where the width of the fluid may be 55-59 pixels, the fill component may be a needle, the width of which may be 12-14 pixels, and the threshold distance T1 may be 8 pixels. As shown in FIG. 4B, this vertical distance is shown as distance D.

[0047] In response to determining that the vertical distance between the fill component and the fluid is less than the threshold distance T1, the method 1100 may proceed to act 1114, where it determines whether the width of the bounding box of the fluid is not greater than the width of the bounding box of the fill component by a threshold width T2. If the width of the bounding box of the fluid is not greater than the width of the bounding box of the fill component by a threshold width T2, the scenario is that the fluid may not have a regular shape, such as the bounding box 422 of FIG. 4B. Instead, what the system has identified as the fluid or fill component may otherwise be a splash. Thus, in response to determining that the width of the bounding box of the fluid is not greater than the width of the bounding box of the fill component by a threshold width T2, the method 1100 may detect splashes in the container, in act 1116. In some embodiments, the threshold distance T1 and the threshold width T2 may vary depending on the size of the container, the size of the fill component, and / or one or more parameters of the filling process. For example, similar to the threshold width T1, a threshold width ratio of the width of the bounding box of the fluid to the width of the bounding box of the fill component may also be used in place of the threshold width T2.

[0048] Additionally and / or alternatively, act 1116 may detect a splash anomaly in response to determining that the fill has started positively (e.g., because splash may be more likely to occur at the beginning of the fill process compared to when the fill process has been performed for some time). In determining whether the fill has started positively, method 1100 may determine the elapsed time from the start of the fill process and determine whether the elapsed time in the fill process is less than a threshold fill time. The threshold fill time may vary depending on one or more parameters of the fill process, such as the size of the container, the size of the fill component, and / or the fill speed. For example, the faster the fill speed, the shorter the threshold fill time.

[0049] Returning to act 1114, if the width of the bounding box of the fluid is greater than the width of the bounding box of the fill component by a threshold width T2, the scenario is that the fluid may have a regular shape, such as bounding box 422 of FIG. 4B. Thus, act 1118 may determine that the image has an unsafe fill component-fluid distance in response to determining that the minimum vertical distance between the fill component and the fluid is less than the threshold distance T1. An unsafe fill component-fluid distance that is too small may indicate that liquid may or is about to contact the fill component. Returning to act 1112, in response to determining that the minimum vertical distance between the fill component and the fluid is not below the threshold distance T1, method 1100 may determine that no anomaly is present in act 1120.

[0050] The various actions shown in FIG. 11 may be performed on the images of the filling process. Additionally, these actions may be performed repeatedly across multiple images in the video of the filling process. The various actions of FIG. 11 may analyze additional images in the video of the filling process, and detection of anomalies may be performed at the video level. For example, the method 1100 may determine that the filling process has a fill component immersion anomaly if any of the images in the video are determined to have a fill component immersion anomaly. In another example, the method 1100 may determine that the filling process has a splash anomaly if any of the images in the video are determined to show a splash anomaly or if the number of images in the video having a splash anomaly is greater than a threshold number. In another example, the method 1100 may determine that the filling process has an unsafe fill component-fluid distance if at least one of the images in the video is determined to show an unsafe fill component-fluid distance.

[0051] In some embodiments, variations of the method 1100 for detecting different types of anomalies may be possible. For example, in response to determining that the fill component does not overlap with the fluid in act 1106, the method 1100 may determine whether a drip exists between the fill component and the fluid. In one example, the method 1100 may determine one or more boundary regions in a region between the fill component and the fluid. In some examples, when an image of the filling process is converted to a binary image, a drip in the image may include a plurality of high intensity pixels. Thus, the method 1100 may determine one or more boundary regions based on the plurality of high intensity pixels. The boundary regions may include connected pixels, such as, for example, 4-connected pixels or 8-connected pixels. The method 1100 may use various tracking algorithms to find the connected pixels. If one or more boundary regions between the fill component and the fluid are identified in act 1122, the method 1100 may determine a drip anomaly between the fill component and the fluid in act 1124. As an example, FIG. 6A shows an exemplary image of a filling process having a drip 610 of fluid 606 in a bottle 602, and FIG. 6B shows a boundary region 608 between a filling component 604 (e.g., a needle) and the fluid 606, where the boundary region exhibits a drip anomaly.

[0052] In another example, FIG. 12 is a flow chart illustrating an exemplary computerized method 1200 for detecting splash anomalies using images of a filling process, according to some embodiments. In some embodiments, the method 1200 can be implemented in act 1004 of FIG. 10 and / or can be performed in the system 100 of FIG. 1, such as by the server 110. In some examples used to illustrate the techniques herein, the method 1200 begins with act 1202 of identifying a filling component in an image. Similar to process 1100, the image may be one or more frame images in a video, such as the video of the filling process captured in act 1002 of FIG. 10. Thus, each image of the filling process may include a container, a filling component inserted in the container, and a fluid in the container, where the fluid is filled by the filling component in the filling process. Exemplary techniques that can be used to identify the filling component will be further described with respect to FIGS. 15A and 15B.

[0053] Method 1200 may proceed to inspect fluid characteristics in the area around the fill component. For example, method 1200 may proceed to act 1206 of determining an area in the image proximate to the identified fill component. Detecting an area proximate to a fill component is further described with reference to FIGS. 4A and 4B. FIG. 4A shows an example of a bounding box that can be used to detect splashes. FIG. 4B shows the example bounding box of FIG. 4A superimposed on an image of the filling process in which the bounding box of FIG. 4A is detected, according to some embodiments. The fill component in the image shown in FIG. 4B is identified and represented within a bounding box 420. As illustrated in FIG. 4A, bounding box 420 may have a rectangular shape with a top 420t, two sides 420s1 and 420s2, and a bottom 420b. In some embodiments, the region proximate the fill component can include a first region 424 located on one side of the bounding box of the fill component 420 and a second region 426 on an opposite side of the bounding box of the fill component 420.

[0054] As shown in FIG. 4A, the adjacent side edge of region 424 may be aligned with side edge 420s1 of bounding box 420, and the adjacent side edge of region 426 may be aligned with side edge 420s2 of bounding box 420. The top edge of region 424 and the top edge of region 426 may be aligned with bottom edge 420b of bounding box 420. In some examples, the width and height of regions 424, 426 may vary depending on the size of the container, the size of the filling component, and / or one or more parameters of the filling process. The width of the first region or the second region may be a fixed width. Alternatively, the width of the first region or the second region may be proportional to the width of the filling component. Similarly, the height of the first region or the second region may be a fixed height or may be proportional to the vertical distance between the filling component and the fluid. Exemplary techniques that can be used to identify the filling component will be described in more detail in FIG. 15A and FIG. 15B.

[0055] Returning to FIG. 12, method 1200 may further include act 1208 of determining fluid characteristics in the image. In some examples, act 1208 may determine fluid characteristics in a region proximate the fill component, such as bounding boxes 424, 426 shown in FIG. 4A. The fluid characteristics may include a slope (or rise) in the surface of the fluid. In some examples, act 1208 may determine the surface of the fluid in the region based on surface edge pixels in the region. The surface edge pixels may be determined by applying an edge detector, for example, a Canny operator or any other suitable edge detector, to the image in which the fill component and the fluid are identified. Act 1208 may determine one or more fluid edges based on the fluid surface edge pixels. For example, each of the one or more fluid edges may be a 4-connected edge or an 8-connected edge from the surface edge pixels. In the case of a 4-connected edge, a pixel may be connected to another pixel only if another pixel is immediately above / below / left / right of the pixel. For an 8-connected edge, any pixel that is in any of the 8 neighboring pixels (e.g., top, bottom, left, right, top / left, top / right, bottom / left, bottom / right) will be considered a connected pixel. Act 1208 may determine the maximum and minimum vertical coordinates among the fluid surface edge pixels in the region near the fill component, and may determine the slope within the surface of the fluid based on the difference between the maximum and minimum vertical coordinates. In some embodiments, the maximum and minimum vertical coordinates may be from any point among the fluid surface edge pixels. In other embodiments, the maximum and minimum vertical coordinates may be from a selected fluid surface edge.

[0056] Optionally, before determining the slope, act 1208 may exclude all vertical edges from one or more fluid edges. Thus, edge pixels belonging to those excluded vertical edges will not be counted when determining the maximum and minimum vertical coordinates. Once the fluid characteristics within the region are determined, method 1200 may proceed to act 1210, which determines whether the slope exceeds a threshold. In response to determining that the slope of the surface of the fluid within the region exceeds a threshold, act 1210 may detect splashes within the region. Figure 3B illustrates an example of surface slope when detecting splashes in the image of Figure 3A.

[0057] As previously described with reference to FIG. 4A, the region near the fill component may include a first region and a second region (424, 426). In such a case, detecting the splatter may be performed separately for each of the first region and the second region. It is also understood that other variations may be possible. For example, the region near the fill component may be a single merged region combining the first region and the second region, or may be a separate region proximate to the tip of the fill component. In such a case, detecting the splatter may be performed for the single region. In some embodiments, detecting the splatter may be performed for the first region and the second region in the image, as described in method 1200. In response to detecting the splatter in at least one of the first region and the second region, method 1200 may determine that the image has splatter.

[0058] With further reference to FIG. 12 , in some embodiments, method 1200 may optionally determine that the fill component is fully inserted into the container (e.g., a predetermined distance into the container) at act 1204 before performing act 1206. Waiting for the fill component to be fully inserted into the container may ensure, for example, that the area around the fill component is accurately determined. In particular, the inventors recognize and understand that the detected width of the fill component may change as the fill component is inserted into the container. Thus, act 1204 may determine that the fill component is fully inserted based on a variation in the fill component width. In response to determining that the variation in the fill component width is below a threshold width, act 1204 may determine that the fill component is fully inserted into the container.

[0059] In some examples used to illustrate the techniques herein, act 1204 may determine the variation in fill component width by analyzing a series of images in a video of the filling process. In some embodiments, the system determines the variation in fill component width based on a maximum width difference among the fill component widths in multiple images in the video. In some embodiments, in determining whether the fill component is fully inserted into the container in a given image, act 1204 may analyze multiple additional images preceding the given image in the video. The techniques to be described with reference to Figures 15A and 15B may be used to identify a fill component for each of the multiple images in the video. As previously described, the identified fill component in an image may be represented by a bounding box. In such a case, the fill component width in each image may be determined based on the width of the bounding box of the identified fill component.

[0060] Additionally and / or alternatively, method 1200 may determine that the filling process has started prior to determining that the filling component is fully inserted into the container in act 1204. This may further improve the accuracy of filling component detection / identification in the image. In some examples, method 1200 may determine that the filling process has started by determining whether fluid is detected in the image. If no fluid is detected, method 1200 may determine that the filling process has not started. Otherwise, method 1200 may determine that filling has started. Techniques to be described with reference to Figures 15A and 15B may be used to detect / identify fluid.

[0061] The various acts shown in FIG. 12 may be performed on images of the filling process. Additionally, these acts may be performed repeatedly across multiple images in a video of the filling process. The various acts of FIG. 12 may analyze additional images in the video of the filling process such that the system can detect splashing at a video level (e.g., using multiple frame images of the video as opposed to using only one image). For example, method 1200 may determine that splashing occurred during the filling process if the system detects splashing in more than a threshold number of images.

[0062] In another example, FIG. 13 is a flow chart illustrating an exemplary computerized method 1300 for detecting safe / unsafe filling component-fluid distances from a video of a filling process, according to some embodiments. In some examples used to illustrate the techniques herein, the method 1300 begins with acts 1302, 1304 to determine whether the filling process is nearing completion. For example, the filling process may be nearing completion when a valve controlling the flow of fluid closes and / or begins to close (e.g., and / or a filling component is about to be withdrawn), thereby blocking the flow of liquid. At this point, the height of the fluid in the container should be stabilized and / or nearly stabilized. Thus, the method 1300 may check the height of the fluid in the container to determine whether the filling process is nearing completion. For example, act 1302 may determine the variation in the height of the surface of the fluid in several consecutive images in the video. Exemplary techniques that will be described with reference to FIG. 15A and FIG. 15B may be used to identify the fluid for each of the multiple images in the video. As previously described, the identified fluid in the image may be represented by a bounding box. In such an example, the top edge of the bounding box of the identified fluid may indicate the height of the top surface of the fluid. Thus, act 1304 may compare the height of the top surface of the fluid in the multiple images in the video by comparing the vertical coordinate (y coordinate) of the top edge of the bounding box of the fluid in these multiple images, and determine the variation in the vertical coordinate as the height variation. In response to determining that the height variation is below the threshold, act 1304 may determine that the filling process is nearing completion. If not, act 1304 may proceed to analyze additional images in the video at act 1302.

[0063] When it is determined that the filling process is approaching completion, method 1300 proceeds to acts 1306, 1308 to be able to identify the filling components and the fluid within the image respectively. As previously described with reference to FIGS. 4A and 4B, for example, the identified filling components and fluid can each be represented by their respective bounding boxes. When acts 1306 and 1308 are completed, method 1300 proceeds to act 1310 to be able to determine the distance between the filling component and the fluid. As previously described with reference to FIG. 4B, for example, the distance between the filling component and the fluid can be determined as the perpendicular distance between the bounding box of the filling component and the bounding box of the fluid, such as the distance D shown in FIG. 4B.

[0064] Returning to FIG. 13, method 1300 can proceed to determine whether the distance between the filling component and the fluid is not safe (e.g., is too large and may cause dripping, or is too small and may cause immersion of the filling component). In some examples, two threshold distances can be provided, for example, a first threshold distance T1 and a second threshold distance T2, where T1 < T2. In act 1312, the method can determine whether the distance between the filling component and the fluid is less than a first threshold distance such as T1. In response to determining that the distance is less than the first threshold distance, method 1300 can detect an unsafe filling component-fluid distance anomaly in act 1314 by determining that the filling component-fluid distance is too small. Similarly, in act 1316, in response to determining that the distance exceeds the second threshold distance, method 1300 can also detect an unsafe filling component-fluid distance anomaly in act 1318 by determining that the filling component-fluid distance is too large. When the distance between the filling component and the fluid is between T1 and T2, method 1300 can determine a safe filling component-fluid distance in act 1320.

[0065] In another example, FIG. 14 is a flow chart illustrating an exemplary computerized method for detecting dripping using images of a filling process, according to some embodiments. In some embodiments, the method 1450 can use a machine learning model 1406 to detect dripping using images of a filling process. In some embodiments, the machine learning model 1406 can include a neural network model or any other suitable model or models thereof, which will be further described herein (e.g., with reference to FIG. 9). The method 1400 can train the machine learning model 1406 using a plurality of training images. In some embodiments, the methods 1400, 1450 can be implemented in act 1004 of FIG. 10 and / or can be performed in the system 100 of FIG. 1, such as, for example, by the server 110. In other embodiments, the machine learning model 1406 can be trained offline. For example, the machine learning model 1406 can be pre-trained by the same and / or a different system (e.g., a different system other than the system that detects dripping). Thus, the methods 1400 and 1450 can each be performed by one or more different computing devices.

[0066] In some examples used to illustrate the techniques herein, method 1400 begins with acquiring a training video at act 1402 and training a machine learning model at act 1404 to generate a machine learning model 1406. In some embodiments, the training images can include a plurality of training images acquired from one or more training videos of the filling process. Each of the training images can include an image of the filling process as the container is being filled with fluid. For example, each training image can include a filling component and a container being filled with fluid through the filling component. In some examples used to illustrate the techniques herein, method 1450 begins with act 1452 acquiring an image of the filling process. For example, the image can be a frame image of a video of the filling process. Similar to methods 1100, 1200, and 1300, the image can be one or more frame images in a video, such as the video of the filling process captured at act 1002 of FIG. 10. Act 1454 can use the trained machine learning model 1406 to classify the image into one of a plurality of classifications, where at least one of the plurality of classifications indicates dripping in the image. Based on the output of the machine learning model, method 1400 may determine whether the image does not have a drip, at act 1456, or whether the image has a drip, at act 1458.

[0067] In some examples, the machine learning model 1406 may include a neural network model that may be trained over a training dataset in the training method 1400. In some embodiments, the training method 1400 may obtain a plurality of training videos in act 1402, where each training video captures an associated fluid filling process. Each training video may include a plurality of training images (e.g., frame images) each labeled with one of a plurality of classifications, where one of the classifications indicates a drip in the training image. The trained machine learning model may include trained weights, e.g., trained weights of a neural network model. An example of a neural network model may include a convolutional neural network (CNN) or any variation thereof. The training method 1400 may use any training algorithm or multiple training algorithms, such as, for example, gradient descent and / or any other suitable algorithm.

[0068] FIG. 9 is a schematic diagram illustrating an exemplary convolutional neural network that may be implemented to detect dripping using images of a filling process, according to some embodiments. For example, the convolutional neural network 900 may be implemented in the machine learning model 1406 of FIG. 14. As shown in FIG. 9, the neural network 900 may include multiple convolutional layers 902. The multiple convolutional layers may be feature extraction layers configured to extract various features of an input image to generate a feature map. According to some embodiments, the feature maps of the convolutional layers of the CNN or other neural network model may be used to calculate visual similarity between images with dripping. For example, FIGS. 5A-5C show exemplary images that show no dripping or various degrees of dripping. For example, FIG. 5A shows exemplary images of a filling process for a glass bottle 502 and a filling component 504 (e.g., a needle) that show no dripping of fluid 506. For comparison, FIGS. 5B and 5C both show dripping of fluid 508 and 510, respectively. 5D shows an example saliency map of an image of a filling process, where samples in the saliency map are shown with higher intensity around the drop 520. In other words, the drop in the image of the filling process may be a visual feature that can be extracted by a convolutional layer of a neural network model. Returning to FIG. 9, the neural network model 900 may include one or more output layers 904 configured to classify the output of the feature extraction layer into one of a plurality of classifications, one of which indicates a drop.

[0069] Returning to FIG. 14, in some examples, the training method 1400 can instead provide a portion of each of the training images to train the machine learning model 1406, rather than the entire image. Similarly, a portion of the image can be provided to the trained machine learning model 1406 to detect the drop anomaly in the method 1450. In this manner, an image having a reduced size is provided to the machine learning model, resulting in less calculations and shorter processing time to detect the anomaly. In some embodiments, the portion of the image to provide to the machine learning model can be determined by reducing the field of view of the image. In some examples, the portion of the image to be provided to the machine learning model can be determined based on the intensity of the image. For example, the portion of the image to be provided to the machine learning model may be the central view of the image, with the leftmost and rightmost columns of the central view being determined based on the intensity of the image.

[0070] In a non-limiting example, for a given image, the sum of the intensities of all pixels in each column is calculated. The sums of all columns are compared and the maximum sum of intensities is determined among all columns. In some embodiments, the columns in the image are analyzed in a left-to-right direction, starting with the first (or left-most) column. For each column, the sum of the intensities of all pixels for that column is compared to a threshold intensity. In some embodiments, the threshold intensity may be a fixed value. In other embodiments, the threshold intensity may be proportional to the maximum sum of intensities determined among all columns in the image. For example, the threshold intensity may be 1 / 3, 1 / 2, 1 / 4 of the maximum sum of intensities. If the sum of intensities for a column exceeds the threshold intensity, then that column in the image is determined to be the first column (or the left-most column of the central field). Once the first column is determined, the remaining columns in the image are analyzed in a right-to-left direction, starting with the last (or right-most) column. For each column, the sum of the intensities of all pixels for that column is compared to the threshold intensity. If the sum of the intensities exceeds the threshold intensity, then the column in the image is determined to be the second column (or the rightmost column of the central view). The central view of the image is then formed based on the columns between the first and second columns in the image.

[0071] In various embodiments, the central view image can be generated in methods 1400, 1450. Alternatively, the central view image can be determined by an image capture sensor (e.g., 112 in FIG. 1), which can be configured to determine an appropriate field of view and generate a portion of the image based on the determined field of view. For example, based on the intensity of the image or motion within the image, the image sensor can determine a central view of the image and generate an image that includes a central view for anomaly detection.

[0072] 14 may analyze additional images in the video of the filling process, and detection of drip anomalies may be performed at the video level. For example, method 1450 may determine that the filling process has splashing if any of the images in the video or the additional images are determined to exhibit drip anomalies.

[0073] In various embodiments described herein, the system can process the images to identify filling components and / or fluids, where the identified filling components and / or fluids are used by other processes to detect anomalies in the filling process. FIG. 15A is a flow chart illustrating an exemplary computerized method for detecting filling components in an image of a filling process, according to some embodiments. FIG. 15B is a flow chart illustrating an exemplary computerized method for detecting fluids in an image of a filling process, according to some embodiments. The methods of FIGS. 15A and 15B can be implemented in various acts, such as act 1004 of FIG. 10, acts 1102, 1104 of FIG. 11, act 1202 of FIG. 12, and acts 1306, 1308 of FIG. 13, all of which acts can be implemented in the system 100 of FIG. 1, such as by the server 110.

[0074] In identifying the fill component and / or fluid in the image, the system may identify edge pixels in the image and determine one or more boundary regions based on the edge pixels. In some examples, pixel values ​​for the fill component and the fluid in the image may be different. For example, the fluid may be shown in a light color with a higher intensity value and the fill component may be shown in a dark color with a lower intensity value. Thus, in identifying the fill component and / or fluid, the system may perform different operations to identify edge pixels in the image. In some examples used to explain the techniques herein, a method 1500 may be provided to identify a fill component from an image in a video of a filling process. The method 1500 may begin with act 1502, where a difference between an image in the video of the filling process and a reference image may be generated. In a non-limiting example, act 1502 may subtract an image from a reference image that precedes the image in the video of the filling process. The reference image in the video may be the first image in the filling process. In other examples, the reference image may be the second image, the third image, or any image that precedes the image in the video. The inventors recognize and understand that during the filling process, the container may be positioned on the filling platform (e.g., 103 in FIG. 1) without moving. Thus, when subtracting the image from the reference image, the pixels of the container (which do not move during the filling process) may be zeroed out and generate no signal. In contrast, if the fill components are shown in a dark color (e.g., low intensity), subtracting the image from the reference image will reveal the high intensity pixels where the fill components are located.

[0075] Method 1500 may proceed to act 1504 to determine edge pixels based on the subtracted image from act 1502. For example, act 1504 may binarize the subtracted image by a threshold. Optionally, before determining the edge pixels in act 1504, method 1500 may apply a filter, such as a bilateral filter, to remove noise from the subtracted image while preserving the edge pixels. In some embodiments, these operations may also be performed in other orders. Method 1500 may continue to act 1506 to determine one or more boundary regions based on the edge pixels. In some embodiments, the boundary regions may include connected edge pixels, such as, for example, 4-connected edge pixels or 8-connected edge pixels. Various tracking algorithms may be used to find the connected edge pixels. Once one or more boundary regions are determined, method 1500 may proceed to act 1508 to select the boundary regions as fill components. In some embodiments, act 1508 may determine, for each of the boundary regions, whether the size of the boundary region exceeds a respective threshold region for the fill component. If the size of the bounding region exceeds the threshold region of the fill component, the method may select the bounding region as the identified fill component. In some embodiments, for the selected bounding region, the method may determine the bounding box of the identified fill component by determining the bounding box that is the smallest rectangular box that encompasses the selected bounding region. An example is shown in FIG. 7B, where high intensity edge pixels indicate the fill component detected from the original image of FIG. 7A.

[0076] In some embodiments, method 1550 may be implemented in a manner similar to method 1500 to provide for identifying fluid in an image of a video of a filling process. Method 1550 may begin with act 1552, in which a difference image between an image and a reference image in the video of the filling process is determined. In a non-limiting example, act 1552 may subtract a reference image preceding the image from the image. As with act 1502, the reference image in the video may be the first image in the filling process. In other examples, the reference image may be the second image, the third image, or any image preceding the image in the video. The difference between act 1502 and act 1552 is that the order of subtraction of the images is reversed. This is because in an image, the fill components typically have low intensity pixels, whereas the fluid typically has high intensity pixels. Thus, if the fluid is shown in a light color (e.g., high intensity), subtracting the reference image from the image will reveal the high intensity pixels where the fluid is located.

[0077] The method 1550 may proceed to act 1554 to determine edge pixels based on the subtracted image from act 1552. For example, act 1554 may binarize the subtracted image by a threshold. Optionally, before determining the edge pixels in act 1554, the method 1550 may apply a filter, such as a bilateral filter, to remove noise from the subtracted image while preserving the edge pixels. In some embodiments, these operations may also be performed in other orders. The method 1550 may continue to act 1556, which determines one or more boundary regions based on the edge pixels. In some embodiments, the boundary regions may include connected edge pixels, such as, for example, 4-connected edge pixels or 8-connected edge pixels. Various tracking algorithms may be used to find the connected edge pixels. Once one or more boundary regions are determined, the method 1550 may proceed to act 1558, which selects the boundary regions as fluid. In some embodiments, act 1558 may determine, for each of the boundary regions, whether the size of the boundary region exceeds a respective threshold region for fluid. In some examples, the threshold area for selecting the fill component (in act 1508) and the threshold area for selecting the fluid (in act 1558) may be different or the same. If the size of the boundary area exceeds the threshold area for the fluid, the method may select the boundary area as the identified fluid. In some embodiments, for the selected boundary area, the method may determine the boundary box of the identified fluid by determining the boundary box that is the smallest rectangular box that encompasses the selected boundary area. An example is shown in FIG. 7C, where high intensity edge pixels indicate the fluid detected from the original image of FIG. 7A. As shown in FIG. 15A and FIG. 15B, the methods 1500 and 1550 are similar with a difference in the subtraction steps 1502, 1552, where in step 1502 (used to detect the fill component), the image is subtracted from the reference image, whereas in step 1552 (used to detect the fluid), the reference image is subtracted from the image.In some embodiments, the fill component may appear as dark pixels (i.e., having a low intensity level) in the video image. Thus, subtracting the image from a previous reference image in the video (as described above for subtraction step 1502) will show a movement and / or different arrangement of the dark pixels (e.g., the fill component), allowing for identification of the fill component. In some embodiments, the fluid may appear as bright pixels (i.e., having a high intensity level) in the video image. Thus, subtracting the image from a previous reference image in the video (as described above for subtraction step 1552) will show a movement and / or different arrangement of the bright pixels (e.g., the fluid), allowing for identification of the fluid. It will be appreciated that in other variations, the operations at 1502, 1552 may be adjusted accordingly depending on whether the fill component / fluid includes dark / bright pixels in the image.

[0078] In some embodiments, variations of identifying the fill component and / or fluid may be possible in the methods 1500, 1550. For example, other variations may include binarizing the image before performing image subtraction in acts 1502, 1552. In other variations, the system may identify the fill component and / or fluid by selecting a boundary region with the largest area among multiple boundary regions, rather than selecting a boundary region with an area above a threshold area.

[0079] An exemplary implementation of a computer system 1600 that may be used to implement any of the aspects of the techniques and embodiments disclosed herein is illustrated in FIG. 16. For example, the computer system 1600 may be located in the system 100 of FIG. 1, such as by the server 110. The computer system 1600 may be configured to implement various methods and acts such as those described in FIGS. 10-15. The computer system 1600 may include one or more processors 1610, one or more non-transitory computer-readable storage media (e.g., memory 1620 and one or more non-volatile storage media 1630), and a display 1640. The processor 1610 may control writing and reading data to and from the memory 1620 and the non-volatile storage device 1630 in any suitable manner, as the inventive aspects described herein are not limited in this respect. In some embodiments, the computer system 1600 may also be a complete system on module (SOM), such as NVIDIA's Jetson module, including a CPU, GPU, memory, and other components in the system. In some embodiments, computer system 1600 may be located at any suitable site. For example, computer system 1600 (e.g., server 110) may be co-located with filling station 102 or may be on a network. In other variations, the system may not need to include memory, with programming instructions instead executing on one or more virtual machines or one or more containers on a cloud. For example, the various methods illustrated above may be implemented by a server on a cloud that includes multiple virtual machines, each having an operating system, virtual disks, virtual networks, and applications, and programming instructions for detecting various anomalies may be stored on one or more of those virtual machines on the cloud.

[0080] To implement the functions and / or techniques described herein, the processor 1610 may execute one or more instructions stored in one or more computer-readable storage media (e.g., memory 1620, storage media, etc.), which may function as a non-transitory computer-readable storage medium that stores instructions for execution by the processor 1610.

[0081] Code used for the techniques described herein, for example, to detect anomalies in an image / video, may be stored on one or more computer-readable storage media of computer system 1600. Processor 1610 may execute any such code to provide any technique for detecting anomalies, as described herein. Any other software, programs, or instructions described herein may also be stored and executed by computer system 1600. It will be understood that computer code may be applied to any aspect of the methods and techniques described herein. For example, computer code may be applied to interact with an operating system to detect anomalies through conventional operating system processes.

[0082] The various methods or processes outlined herein may be coded as software executable on one or more processors using any one of a variety of operating systems or platforms. Additionally, such software may be written using any of a number of suitable programming languages ​​and / or programming or scripting tools, and may be compiled as executable machine code or intermediate code that runs on a virtual machine or suitable framework.

[0083] In this regard, the various inventive concepts may be embodied as at least one non-transitory computer-readable storage medium (e.g., a computer memory, one or more floppy disks, compact disks, optical disks, magnetic tapes, flash memories, circuitry in a field programmable gate array or other semiconductor device, etc.) encoded with one or more programs that, when executed on one or more computers or other processors, perform various embodiments of the invention. The non-transitory computer-readable medium or media may be portable, such that the one or more programs stored thereon may be loaded into any computer resource to implement various aspects of the invention as discussed above.

[0084] The terms "program," "software," and / or "application" are used herein in a generic sense to refer to any type of computer code or set of computer-executable instructions employed to program a computer or other processor to perform various aspects of the embodiments as discussed above. Additionally, according to one aspect, it should be understood that one or more computer programs that, when executed, perform the methods of the present invention need not reside on a single computer or processor, but may be distributed in a modular manner among different computers or processors to perform various aspects of the present invention.

[0085] Computer-executable instructions may be in many forms, such as program modules, executed by one or more computers or other devices. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. Typically the functionality of the program modules may be combined or distributed as desired in various embodiments.

[0086] Also, data structures may be stored in a non-transitory computer-readable storage medium in any suitable form. A data structure may have fields that are related through their locations within the data structure. Such relationships may be achieved by allocating storage to the fields that in turn have locations within the non-transitory computer-readable medium that convey the relationships between the fields. However, any suitable mechanism may be used to establish relationships between information in fields of a data structure, including through the use of pointers, tags, or other mechanisms that establish relationships between data elements.

[0087] Various inventive concepts may be embodied as one or more methods, examples of which are provided. The acts performed as part of a method may be ordered in any suitable manner. Thus, while shown as sequential acts in the exemplary embodiments, embodiments may be constructed in which acts are performed in an order different from that illustrated, which may include performing some acts simultaneously.

[0088] The indefinite articles "a" and "an" as used herein in the specification and claims should be understood to mean "at least one" unless expressly stated otherwise. When used herein in the specification and claims, the phrase "at least one" in connection with a list of one or more elements should be understood to mean at least one element selected from any one or more of the elements in the list of elements, and does not necessarily include at least one of each and every element specifically listed in the list of elements, nor does it exclude any combination of elements in the list of elements. This allows for the optional presence of elements other than those specifically identified in the list of elements to which the phrase "at least one" refers, whether related or unrelated to those specifically identified elements.

[0089] The phrase "and / or" as used herein in the specification and claims should be understood to mean "either or both" of the elements so conjoined, i.e., elements that are conjunctively present in some cases and disjunctively present in other cases. Multiple elements listed with "and / or" should be interpreted in the same manner, i.e., "one or more" of the elements so conjoined. Other elements other than the elements specifically identified by the "and / or" clause may optionally be present, whether related to those elements specifically identified or not. Thus, as a non-limiting example, a reference to "A and / or B," when used in conjunction with open-ended language such as "comprising," in one embodiment refers to only A (optionally including elements other than B), in another embodiment refers to only B (optionally including elements other than A), in yet another embodiment refers to both A and B (optionally including other elements), and so forth.

[0090] As used herein and in the claims, "or" should be understood to have the same meaning as "and / or" as defined above. For example, when separating items in a list, "or" or "and / or" should be interpreted as inclusive; that is, to include at least one of, but not only more than one, of a plurality of elements or a list of elements, and, optionally, to include additional items not listed. Only terms clearly indicated to the contrary, such as "only one of" or "exactly one of," or, when used in the claims, "consisting of," refer to the inclusion of exactly one element of a plurality of elements or a list of elements. In general, as used herein, the term "or" should be interpreted as indicating an exclusive selection (i.e., "one or the other but not both") when followed by an exclusive term, such as "either," "one of," "only one of," or "exactly one of." "Consisting essentially of," when used in the claims, shall have its ordinary meaning as used in the field of patent law.

[0091] In the claims, the use of ordinal terms such as "first," "second," "third," etc. to modify a claim element does not, by itself, imply any priority, precedence, or ordering of the claim element relative to another claim element, or the temporal order in which the acts of a method are performed. Such terms are used merely as labels to distinguish an element of one claim having a particular name from another element having the same name (in the absence of the use of ordinal terms).

[0092] The phraseology and terminology used herein is for the purpose of description and should not be regarded as limiting. The use of "including," "comprising," "having," "containing," "involving," and variations thereof are meant to encompass the items listed thereafter as well as additional items.

[0093] Although several embodiments of the present invention have been described in detail, various modifications and improvements will readily occur to those skilled in the art. Such modifications and improvements are intended to be within the spirit and scope of the present invention. Accordingly, the foregoing description is by way of example only and is not intended to be limiting.

[0094] Various aspects are described in this disclosure, including but not limited to the following aspects. (1) A computerized method for detecting anomalies in a filling process in which a container is filled with fluid using a filling component, the method comprising: capturing video of the container while the container is being filled with fluid through the filling component; analyzing images from the video using one or more processors to detect anomalies including one or more of a fill component immersion anomaly, a splash anomaly including splashing of fluid in the container, a drip anomaly including dripping of fluid in the container, or an unsafe fill component-fluid distance anomaly in which the filling component comes within an unsafe distance of a surface of the fluid in the container; and generating an alert indicating the detected anomaly. (2) The method of embodiment 1, further comprising adjusting a filling machine configured to fill the container with fluid by altering one or more parameters of a filling process that the filling machine is configured to perform based on the detected anomaly. The method of aspect 1 or 2, further comprising (3) analyzing additional images of the video and (i) determining that the filling process has a fill component immersion anomaly if any of the image or the additional images are determined to indicate a fill component immersion anomaly, or (ii) determining that the filling process has a splash anomaly if any of the image or the additional images are determined to indicate a splash anomaly, or (iii) determining that the filling process has an unsafe fill component-fluid distance anomaly if none of the images or the additional images are determined to indicate a fill component immersion anomaly and at least one of the images or the additional images is determined to indicate a minimum vertical distance between the fill component and the fluid is less than a threshold distance. (4) The method of any one of aspects 1 to 3, wherein detecting the anomaly includes identifying a filling component in the image, identifying a fluid in the container in the image, and detecting the anomaly based on a relative position of the filling component and the fluid. (5) The method of aspect 4, wherein identifying the fill component includes determining a first bounding box of the fill component, and identifying the fluid includes determining a second bounding box of the fluid. (6) The method of aspect 5, wherein detecting the anomaly includes determining that a first bounding box of the filling component overlaps with a second bounding box of the fluid. (7) The method of aspect 6, further comprising classifying the detected anomaly as a fill component immersion anomaly when a width of the second bounding box of the fluid is greater than a width of the first bounding box of the fill component by at least one of a threshold width ratio and a threshold width difference. (8) The method of embodiment 7, wherein detecting the anomaly further includes determining that an elapsed time in the filling process is greater than a threshold filling time. (9) The method of embodiment 6, further comprising classifying the detected anomaly as a splash anomaly when a width of the second bounding box of the fluid is not greater than a width of the first bounding box of the fill component by at least one of a threshold width ratio and a threshold width difference. (10) The method of aspect 5, further comprising classifying the detected anomaly as a splash anomaly when the first bounding box of the fill component does not overlap with the second bounding box of the fluid, a minimum vertical distance between an edge of the first bounding box of the fill component and an edge of the second bounding box of the fluid is less than a first threshold distance, and a width of the second bounding box of the fluid is not greater than a width of the first bounding box of the fill component by a second threshold width difference. (11) The method of aspect 9 or 10, wherein detecting the anomaly further includes determining that the elapsed time in the filling process is less than a threshold filling time. (12) The method of aspect 5, further comprising classifying the detected anomaly as an unsafe fill component-fluid distance anomaly when the first bounding box of the fill component does not overlap with the second bounding box of the fluid, a minimum vertical distance between an edge of the first bounding box of the fill component and an edge of the second bounding box of the fluid is less than a first threshold vertical distance, and a width of the second bounding box of the fluid is greater than the width of the first bounding box of the fill component by a second threshold width difference. (13) The method of aspect 1 or 2, further comprising: analyzing additional images of the video; and determining that the filling process has a splash anomaly if the number of images from the image with the splash anomaly and the additional images exceeds a splash threshold. (14) The method of any one of aspects 1, 2, and 13, wherein detecting the anomaly includes identifying a fill component in the image, determining an area in the image proximate the fill component, determining fluid characteristics in the image, and detecting a splash anomaly in the area based on the fluid characteristics. (15) The method of embodiment 14, further comprising determining that the filling component is fully inserted into the container before determining the region. (16) The method of aspect 15, wherein determining that the fill component is fully inserted into the container includes determining a fill component width of the fill component in the image and each of the fill component widths in one or more additional images preceding the image in the video, determining a variation in the fill component width based on the fill component width in the image and each of the fill component widths in the one or more additional images preceding the image, and determining that the variation in the fill component width is below a threshold fill component width. (17) The method of aspect 16, wherein the variation in fill component width is a maximum width difference between the fill component width in the image and each fill component width in the one or more additional images. (18) The method of embodiment 15, further comprising determining that the filling process has started prior to determining that the filling component is fully inserted into the container. (19) The method of aspect 18, wherein determining that the filling process has started includes determining whether fluid is detected in the image, and if fluid is not detected, determining that the filling process has not started, and if not, determining that filling has started. (20) The method of embodiment 14, wherein the region includes a first region on a first side of the filling component and a second region on a second side of the filling component opposite the first side. (21) The method of any one of aspects 14 to 20, wherein the fluid characteristic includes a slope within a surface of the fluid, and detecting a splash anomaly within the region includes determining whether the slope exceeds a threshold value, and detecting a splash anomaly within the region if the slope is determined to exceed the threshold value. (22) The method of aspect 21, wherein determining fluid characteristics in the image includes determining fluid surface edge pixels in a region in the image, determining one or more fluid edges based on the fluid surface edge pixels, and determining a slope based on a difference between a maximum vertical coordinate among the fluid surface edge pixels and a minimum vertical coordinate among the fluid surface edge pixels. (23) The method of embodiment 22, further comprising excluding vertical edges from the fluid edge in the region in the image before determining the slope. (24) The method of aspect 22, wherein determining the fluid surface edge pixels includes applying a Canny operator to a region within the image. (25) The method of any one of aspects 1 to 2, further comprising determining that the filling process is nearing completion if a variation in fluid height in several consecutive images, including images from a video, falls below a threshold value. (26) The method of claim 25, wherein the image corresponds to a time when the filling process is nearing completion, and detecting the anomaly includes: (i) identifying a filling component in the image; (ii) identifying a fluid in the container in the image; (iii) determining a distance between the filling component and the fluid; and (iv) detecting an unsafe filling component-fluid distance anomaly at the time when the filling process is nearing completion if the distance between the filling component and the fluid is determined to be below a threshold distance. (27) The method of aspect 26, wherein determining a distance between the fill component and the fluid includes determining a vertical distance between a nearest edge of a bounding box of the fill component and a bounding box of the fluid. (28) The method of aspect 1 or 2, further comprising: analyzing additional images of the video; and determining that the filling process has a dripping anomaly if either the images or the additional images are determined to show the dripping anomaly. (29) The method of any one of aspects 1, 2, and 28, wherein determining the anomaly includes detecting a drip anomaly in the image by providing the image to a trained neural network model and determining one of a plurality of classifications using the trained neural network model, where at least one of the plurality of classifications is indicative of a drip anomaly in the image. (30) The method of embodiment 29, wherein providing the image to the trained neural network model includes determining a portion of the image based on an intensity of the image; and providing the portion of the image to the trained neural network model. (31) The method of embodiment 29, further comprising: (i) acquiring a plurality of training videos, each training video of the plurality of training videos capturing an associated fluid filling process and including a plurality of training images each labeled with one of a plurality of classifications; and (ii) training the neural network model using the plurality of training videos. (32) The method of any one of aspects 4, 14, or 26, wherein identifying a fill component or fluid includes identifying edge pixels in the image; determining one or more boundary regions based on the edge pixels; and, for each of the one or more boundary regions, (i) determining whether a size of the boundary region exceeds a respective threshold region for the fill component or fluid; and (ii) selecting the boundary region as a fill component or fluid if the size of the boundary region exceeds the respective threshold. (33) The method of aspect 32, wherein identifying edge pixels in the image includes subtracting the image from a reference image that precedes the image in the video to generate a subtracted image, and selecting the border region as the fill component includes selecting the border region as the fill component based on the subtracted image. (34) The method of aspect 32, wherein identifying edge pixels in the image includes subtracting from the image another image that precedes the image in the video to generate a subtracted image, and selecting the boundary region as fluid includes selecting the boundary region as fluid based on the subtracted image. (35) The method of any one of aspects 33 and 34, wherein identifying edge pixels includes binarizing the subtracted image. (36) The method of aspect 33 or 34, wherein identifying edge pixels includes applying a bilateral filter to the subtracted image to generate a filtered subtracted image, and binarizing the filtered subtracted image. (37) The method of aspect 32, further comprising generating a bounding box for the fill component or fluid based on the bounding region, the bounding box being a smallest rectangle that encompasses the bounding region. (38) The method of any one of aspects 4, 14, or 26, wherein identifying the fill component or fluid includes generating a difference image between the image and a reference image in the video, determining one or more edges in the image based on the difference image, determining one or more boundary regions using the one or more edges, and selecting the boundary region having the largest area among the one or more boundary regions as the fill component or fluid. (39) The method of embodiment 38, further comprising binarizing the image and the reference image before generating the difference image. (40) The method of aspect 38, further comprising generating a bounding box for the filled component or fluid based on the bounding region, the bounding box being a minimum rectangular area that encompasses the bounding region. (41) The method of aspect 38, wherein determining the difference image includes subtracting an image from a reference image in the video, and determining the boundary region as a fill component includes determining the boundary region as a fill component based on the difference image. (42) The method of aspect 38, wherein determining the difference image includes subtracting a reference image in the video from the image, and determining the boundary region as fluid includes determining the boundary region as fluid based on the difference image. (43) The method of any one of aspects 1 to 42, wherein the fluid is a medicinal fluid. (44) The method of any one of aspects 1 to 43, wherein the container is a transparent drug product container. (45) A non-transitory computer-readable medium comprising instructions, the instructions being operable, when executed by one or more processors on a computing device, to cause the one or more processors to perform a method as described in any one of aspects 1-44. (46) A system comprising: a filling machine configured to perform a filling process in which a container is filled with a fluid using a filling component; an image capture device configured to capture video of the container while the container is filled with the fluid; and a processor configured to execute programming instructions to perform operations including any one of aspects 1-44.

Claims

1. 1. A computerized method for detecting anomalies in a filling process in which a container is filled with a fluid using a filling component, comprising: capturing a video of the container while the container is being filled with the fluid through the filling component; analyzing the images of the video using one or more processors to detect anomalies including one or more of a fill component immersion anomaly, a splash anomaly including splashing of fluid in the vessel, a drip anomaly including dripping of fluid in the vessel, or an unsafe fill component-fluid distance anomaly where the fill component comes within an unsafe distance of a surface of the fluid in the vessel; generating an alarm indicative of the detected anomaly; Detecting the anomaly includes: identifying the fill component within the image; identifying the fluid within the container in the image; and detecting the anomaly based on a relative position of the fill component and the fluid.

2. 10. The method of claim 1, further comprising adjusting a filling machine configured to fill the container with the fluid by altering one or more parameters of the filling process that the filling machine is configured to perform based on the detected anomaly.

3. analyzing additional images of the video; and determining that the filling process has a fill component immersion anomaly if either the image or the additional image is determined to show the fill component immersion anomaly; or determining that the filling process has the splash anomaly if either the image or the additional image is determined to show the splash anomaly; or None of the images or the additional images are determined to be indicative of the fill component immersion anomaly; and 3. The method of claim 1 or 2, further comprising: determining that the filling process has an unsafe filling component-fluid distance anomaly if at least one of the image or the additional image is determined to indicate that a minimum vertical distance between the filling component and the fluid is less than a threshold distance.

4. Identifying the fill component includes determining a first bounding box of the fill component; The method of claim 1 , wherein identifying the fluid includes determining a second bounding box of the fluid.

5. The method of claim 4 , wherein detecting the anomaly comprises determining that the first bounding box of the filling component overlaps with the second bounding box of the fluid.

6. 6. The method of claim 5, further comprising classifying the detected anomaly as a fill component immersion anomaly when a width of the second bounding box of the fluid is greater than a width of the first bounding box of the fill component by at least one of a threshold width ratio and a threshold width difference.

7. the first bounding box of the fill component does not overlap with the second bounding box of the fluid; a minimum perpendicular distance between an edge of the first bounding box of the fill component and an edge of the second bounding box of the fluid is less than a first threshold distance; and 5. The method of claim 4, further comprising classifying the detected anomaly as the splash anomaly when a width of the second bounding box of the fluid is not greater than a width of the first bounding box of the fill component by a second threshold width difference.

8. the first bounding box of the fill component does not overlap with the second bounding box of the fluid; a minimum perpendicular distance between an edge of the first bounding box of the fill component and an edge of the second bounding box of the fluid is less than a first threshold perpendicular distance; and 5. The method of claim 4, further comprising classifying the detected anomaly as the unsafe fill component-fluid distance anomaly when a width of the second bounding box of the fluid is greater than a width of the first bounding box of the fill component by a second threshold width difference.

9. A computerized method for detecting anomalies in a filling process in which a container is filled with a fluid using a filling component, comprising: capturing a video of the container while the container is being filled with the fluid through the filling component; analyzing the images of the video using one or more processors to detect anomalies including one or more of a fill component immersion anomaly, a splash anomaly including splashing of fluid in the vessel, a drip anomaly including dripping of fluid in the vessel, or an unsafe fill component-fluid distance anomaly where the fill component comes within an unsafe distance of a surface of the fluid in the vessel; generating an alarm indicative of the detected anomaly; Detecting the anomaly includes: identifying the fill component within the image; determining an area within the image that is proximate to the fill component; determining a fluid characteristic within the image; and detecting the splash anomaly in the region based on the fluid characteristics.

10. The method of claim 9 , further comprising determining that the filling component is fully inserted into the container before determining the region.

11. Determining that the filling component is fully inserted into the container comprises: determining a fill component width of the fill component in the image and a fill component width of each of the fill components in one or more additional images preceding the image in the video; determining a variation in fill component width based on the fill component width in the image and the respective fill component widths in the one or more additional images preceding the image; and determining that the variation in fill component width is below a threshold fill component width.

12. determining that the filling process has started prior to determining that the filling component has been fully inserted into the container, wherein determining that the filling process has started includes: determining whether fluid is detected in the image; The method of claim 10 , further comprising: if no fluid is detected, determining that the fill process has not started; otherwise, determining that the fill has started.

13. The region is a first region on a first side of the filling component; and a second region on a second side of the filler component opposite the first side.

14. The fluid characteristic includes a gradient within a surface of the fluid, and detecting the splash anomaly within the region includes: determining whether the slope exceeds a threshold; and detecting the splash anomaly in the region if the slope is determined to exceed the threshold.

15. Determining the fluid properties in the image includes: determining fluid surface edge pixels within the region in the image; determining one or more fluid edges based on the fluid surface edge pixels; and determining the slope based on a difference between a maximum vertical coordinate among the fluid surface edge pixels and a minimum vertical coordinate among the fluid surface edge pixels.

16. The method of claim 15 , further comprising excluding vertical edges from the fluid edge in the region in the image before determining the slope.

17. A computerized method for detecting anomalies in a filling process in which a container is filled with a fluid using a filling component, comprising: capturing a video of the container while the container is being filled with the fluid through the filling component; analyzing the images of the video using one or more processors to detect anomalies including one or more of a fill component immersion anomaly, a splash anomaly including splashing of fluid in the vessel, a drip anomaly including dripping of fluid in the vessel, or an unsafe fill component-fluid distance anomaly where the fill component comes within an unsafe distance of a surface of the fluid in the vessel; generating an alarm indicative of the detected anomaly; determining that the filling process is nearing completion if a variation in height of the fluid in several consecutive images including the image of the video is below a threshold; the image corresponds to a time when the filling process is nearing completion; Detecting the anomaly includes: identifying the fill component within the image; identifying the fluid within the container in the image; determining a distance between the fill component and the fluid; detecting the unsafe filling component-fluid distance anomaly when the filling process is nearing completion if the distance between the filling component and the fluid is determined to be below a threshold distance.

18. analyzing additional images of the video; and The method of claim 1 or 2, further comprising: determining that the filling process has the dripping anomaly if either the image or the additional image is determined to exhibit the dripping anomaly.

19. Determining the abnormality includes: providing said images to a trained neural network model; 3. The method of claim 1 or 2, comprising detecting the dropping anomaly in the image by: determining one of a plurality of classifications using the trained neural network model, at least one of the plurality of classifications being indicative of the dropping anomaly in the image.

20. A plurality of training videos, each training video of the plurality of training videos comprising: Capture the associated fluid filling process; obtaining a plurality of training videos including a plurality of training images each labeled with one of the plurality of classifications; 20. The method of claim 19, further comprising: training the neural network model using the plurality of training videos.

21. Identifying the fill component or the fluid includes: identifying edge pixels in the image; determining one or more boundary regions based on the edge pixels; For each boundary region of the one or more boundary regions: determining whether the size of the boundary area exceeds a respective threshold area for a fill component or fluid; and selecting the boundary region as the fill component or the fluid if the size of the boundary region exceeds the respective threshold.

22. identifying the edge pixels in the image includes subtracting the image from a reference image that precedes the image in the video to generate a subtracted image; The method of claim 21 , wherein selecting the boundary region as the fill component comprises selecting the boundary region as the fill component based on the subtracted image.

23. identifying the edge pixels in the image includes subtracting from the image another image that precedes the image in the video to generate a subtracted image; The method of claim 21 , wherein selecting the boundary region as the fluid comprises selecting the boundary region as the fluid based on the subtracted image.

24. The method of claim 21 , further comprising generating a bounding box for the filled component or the fluid based on the bounding region, the bounding box being a smallest rectangle that encompasses the bounding region.

25. Identifying the fill component or the fluid includes: generating a difference image between the image and a reference image in the video; determining one or more edges in the image based on the difference image; determining one or more boundary regions using the one or more edges; and selecting a boundary region having a largest area among the one or more boundary regions as the fill component or the fluid.

26. The method of claim 1 or 2, wherein the fluid is a medicinal fluid.

27. The method of claim 1 or 2, wherein the container is a transparent drug product container.

28. 3. A non-transitory computer-readable medium comprising instructions that, when executed by one or more processors on a computing device, are operable to cause the one or more processors to perform the method of claim 1 or 2.

29. 1. A system comprising: a filling machine configured to perform a filling process using the filling component, in which the container is filled with a fluid; an image capture device configured to capture video of the container while it is being filled with the fluid; and A processor configured to execute programming instructions to perform the operations of claim 1 or 2.

Images