Acoustic detection of glass breaking events
Through machine learning models, glass breaking events in the biological manufacturing process are identified, and audio data is used to identify glass breaking sounds, which realizes automatic detection and operation stopping, solving the problem of glass breaking events detection in the biological manufacturing process, and improving production efficiency and safety.
Patent Information
- Application Number
- CN202380035816.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2022-09-23
- Filing Date
- 2023-04-25
- Publication Date
- 2024-12-03
AI Technical Summary
In the biomanufacturing process, glass breaking events are difficult to detect in a timely manner, resulting in production interruptions, product scrapping and personal safety risks. The existing technology cannot effectively monitor glass breaking events, especially in the case of high productivity and unmanned monitoring.
Using machine learning models, glass breaking events are identified by training audio data, including ambient sound, glass sound and glass breaking sound, audio data processing is used to automatically detect and indicate glass breaking events, and then manufacturing operations are automatically stopped.
It improves the detection efficiency of glass breakage events, reduces production interruptions and product scrapping, reduces personal safety risks, and improves resource and energy efficiency of the biomanufacturing process and reduces production costs.
Smart Images

Figure CN119072746A_ABST
Abstract
Description
[0001] CROSS-REFERENCE TO RELATED APPLICATIONS
[0002] U.S. Patent Application No. 63 / 409,498 filed on September 23, 2022 and U.S. Patent Application No. 63 / 336,847 filed on April 29, 2022 are incorporated herein by reference in their entirety. Technical Field
[0003] The present application generally relates to using predictive models to identify glass breaking events. More specifically, the present application may relate to using audio data to identify glass breaking events in a biomanufacturing process. Background Art
[0004] Many manufacturing processes operate on or otherwise utilize glass. For example, glass may be included in parts of machinery or equipment used in a manufacturing process, used to contain or package products produced by a manufacturing process, included in the product itself, etc. A biomanufacturing process is a manufacturing process that often involves glass. In a biomanufacturing process, the fill-finish process involves preparing a biological product (e.g., a drug product) into its delivery container. In many biological products, the delivery container is a glass vial. Other biomanufacturing processes / biological products that include glass are prefilled glass syringes, auto-injectors, or intravenous (IV) infusion bags. Glass is a typical material used in biomanufacturing processes due to its chemical durability, airtightness, strength, cleanliness, and clarity. However, glass containers are not without risks. Problems with glass containers include breakage, delamination, and glass particles or fragments, all of which may affect the safety and efficacy of the biological product. In fact, over the past decade, product recalls due to glass issues, including contamination, breakage, and particulate matter caused by delamination (flaking) of glass vials, have resulted in hundreds of millions of units of biologic products packaged in vials, syringes, or glass auto-injectors being removed from the market. Traditionally, detection of a glass break event during manufacturing triggers an immediate, predefined process to stop operations, document, and subsequently investigate the root cause of the glass break event, which can typically take days. As a result, glass break events are costly, not only in terms of lost man-hours during downtime, but also in scrap costs, as all drug product produced during the run typically must be discarded, which alone can cost tens of thousands of dollars. Finally, glass breakage presents a personal safety concern for operators, as glass shards can create a laceration hazard during cleanup.
[0005] While the risk and cost of glass break events in manufacturing processes (e.g., biomanufacturing processes) remain high, advances and continued growth in manufacturing processes enabled by automation, robotics, and single-use technologies have resulted in production rates that are too high to be feasibly monitored or inspected from start to finish by humans alone. Certain events, such as glass break events, may go undetected without human monitoring. Many of the negative impacts of glass break events discussed previously will only become more severe if glass break events go undetected for extended periods of time. Furthermore, even in manufacturing processes where production rates are low enough to allow for human monitoring of the process, glass break events may be difficult to detect through human vision, and glass break events may sometimes only produce small particles or microcracks.
[0006] Therefore, for conventional manufacturing processes (e.g., biomanufacturing processes), there is an increased likelihood that a glass break event will go undetected for an extended period of time, thus presenting a significant risk and cost burden. Summary of the invention
[0007] Various aspects of the present disclosure provide a method for identifying glass-breaking events using machine learning, the method comprising: (a) accessing or obtaining a machine learning model trained using training audio data, the training audio data representing (i) training environment sounds, (ii) training glass sounds, and (iii) training glass-breaking sounds; (b) obtaining, by one or more processors, audio data within a time period of interest; (c) determining whether a glass-breaking event occurred during the time period of interest by processing the audio data using the machine learning model; and (d) indicating, by one or more processors, that a glass-breaking event has occurred when it is determined that a glass-breaking event has occurred.
[0008] In some aspects, the training audio data and the audio data each include one or more spectrograms; and obtaining the audio data for the time period of interest includes generating a spectrogram of the audio data from the original audio data using a Fourier transform.
[0009] In some aspects, (i) the training environment sounds include sounds caused by machine operation, (ii) the training glass sounds include sounds caused by a first glass surface contacting a second glass surface or a non-glass surface, and (iii) the training glass breaking sounds include sounds caused by glass cracking or glass shattering. In some aspects, the machine implements a biomanufacturing process, and the training glass sounds and the training glass breaking sounds are generated by one or more containers of one or more pharmaceutical products.
[0010] In some aspects, the method further includes automatically stopping operation of the machine by the one or more processors after identifying the glass break event.
[0011] In some aspects, the method further includes iteratively adjusting, by one or more processors, an amplification gain applied to one or both of the training audio signal or the audio signal until a performance threshold is met, wherein the training audio signal and the audio signal correspond to the training audio data and the audio data, respectively.
[0012] In some aspects, the machine learning model is a convolutional neural network.
[0013] Another aspect of the present disclosure provides a computer system comprising (a) one or more processors; and (b) a program memory coupled to the one or more processors and storing executable instructions that, when executed by the one or more processors, cause the computer system to perform the method described in any of the foregoing aspects.
[0014] Other aspects of the present disclosure provide a method and executable instructions for training a machine learning model for identifying glass breaking events, the method and executable instructions comprising: (a) obtaining training audio data, (b) classifying the training audio data into multiple subsets, each subset corresponding to different actual result data, the subsets including (i) at least one subset representing training environment sounds, (ii) at least one subset representing training glass sounds, and (iii) at least one subset representing training glass breaking sounds, and (c) using the classified subsets of the training audio data to generate a machine learning model for identifying glass breaking events. BRIEF DESCRIPTION OF THE DRAWINGS
[0015] Those skilled in the art will appreciate that the drawings described herein are included for illustrative purposes and are not intended to limit the present disclosure. The drawings are not necessarily drawn to scale, but rather focus on illustrating the principles of the present disclosure. It should be understood that in some cases, various aspects of the described embodiments may be exaggerated or enlarged to aid in understanding the described embodiments. In the drawings, similar reference numerals generally refer to components that are similar in function or structure throughout the drawings.
[0016] Figure 1A and Figure 1B is a simplified block diagram of an example system for identifying glass break events in a biomanufacturing process.
[0017] Figure 2 An example of a scenario depicting a glass breakage incident in biomanufacturing process machinery.
[0018] Figure 3 Depicts an example process for preprocessing audio data before feeding it into a machine learning model.
[0019] Figure 4A and Figure 4B Depicted is an example of a machine learning model used to identify glass breakage events in a biomanufacturing process.
[0020] Figure 5A and Figure 5B Depicted is an example of experimental performance data for a machine learning model used to identify glass breakage events in a biomanufacturing process.
[0021] Figure 6 is a flow chart depicting an example method for identifying glass break events in a biomanufacturing process.
[0022] Figure 7 is a flow chart depicting an example method for training a machine learning model to identify glass break events in a biomanufacturing process. DETAILED DESCRIPTION
[0023] As the pace of biomanufacturing processes accelerates, the need to be able to monitor and detect glass-breaking events as quickly as possible increases. The present disclosure is intended to reduce the problems with conventional techniques (e.g., as described in the background section) by providing techniques for identifying glass-breaking events using machine learning techniques. The present technology can train a machine learning model using training audio data representing environmental sounds, glass sounds, and glass-breaking sounds, which can be labeled accordingly in the training audio data. The present technology can use a machine learning model to determine whether a glass-breaking event occurred during a time period of interest represented in the audio data, and if so, to notify or indicate. By determining and then notifying / indicating when a glass-breaking event occurs, the present technology is intended to provide insights to operators of biomanufacturing process systems and reduce the negative impacts associated with glass-breaking events (e.g., as described in the background section).
[0024] The present technology detects when a glass-breaking event occurs to ensure that patients receiving biological products (e.g., pharmaceutical products) manufactured by a biomanufacturing process receive the highest product quality by immediately stopping manufacturing operations, thereby reducing operating costs and reducing scrap. Audio was selected as the most viable medium for detecting glass-breaking events due to the following four main factors: cost, data capacity, visibility, and adaptability. Compared to other potential detection means (e.g., video, laser curtain systems, pressure sensors, accelerometers, etc.), basic audio capture equipment is relatively inexpensive and requires little calibration other than gain settings. On the other hand, video detection may require expensive equipment because glass-breaking events occur extremely quickly and sometimes occur in a small area. Since glass-breaking events must be continuously monitored, it is very important to minimize the computing power required to operate and store related data. Finally, since a large number of biomanufacturing processes may theoretically result in glass breaking, audio detection is very suitable because no line of sight is required (unlike video detection), so audio detection can be adapted to a variety of biomanufacturing processes. This is particularly important because many processes completed by vials, auto-injectors, and syringe filling are completed without glass parts visible to the outside.
[0025] Advantageously, by providing improved insights into glass break events, the present technology reduces the negative impacts associated with conventional methods of handling glass break events. One advantage of these insights is that fewer resources (e.g., bioproducts) are wasted during glass break events that are detected more quickly using the present technology, thereby improving resource efficiency and improving the sustainability of the biomanufacturing process system. By making the biomanufacturing process system more sustainable in terms of resource usage, the energy efficiency of the biomanufacturing process system can also be improved, and the financial or economic cost of producing each bioproduct can be reduced. Another advantage of the improved insights is that production can be increased because more bioproducts can be produced in a given amount of time (reduced reset time in the case of glass breakage). Resource, energy, and cost efficiencies can also be improved when handling glass break events in biomanufacturing processes because the currently disclosed technology provides insights into the availability of bioproducts produced in the event of glass breakage (e.g., by providing insights into the likelihood of broken glass contaminating the bioproduct). Furthermore, by more easily identifying glass break events, data (e.g., audio data, operational data, etc.) from the biomanufacturing process before, during, and after the glass break events can be analyzed to provide insights into the causes and consequences of the glass break events, thereby improving the technical field of glass break event diagnosis.
[0026] Throughout this disclosure, those of ordinary skill in the art will appreciate the additional advantages of the presently disclosed technology over conventional methods of identifying glass break events. The various concepts and techniques introduced above and discussed in more detail below can be implemented in any of a variety of ways, and the described concepts are not limited to any particular manner of implementation. For illustrative purposes, examples of implementation are provided below.
[0027] Exemplary Systems
[0028] Figure 1A 1 is a simplified block diagram of an example system 100A for identifying glass break events in a biomanufacturing process machine 160, which may, for example, produce a pharmaceutical product. In some embodiments, system 100A comprises a standalone device, but in other embodiments, system 100A is incorporated into other devices. At a high level, system 100A comprises components such as a computing device 110, one or more training audio data sources 150, biomanufacturing process machine 160, and one or more audio sensors 162. Figure 1A In the embodiment, computing device 110, biomanufacturing process machinery 160, and training audio data source 150 are communicatively coupled via network 170, which may be or include a proprietary network, a secure public Internet, a virtual private network, and / or any other type(s) of suitable wired or wireless networks (e.g., dedicated access lines, satellite links, cellular data networks, combinations of these networks, etc.). In embodiments where network 170 includes the Internet, data communications may be conducted through network 170 via Internet communication protocols. In some aspects, system 100A may include more than one network. Figure 1A More or fewer instances of the various components of system 100A are shown (e.g., one instance of computing device 110, ten instances of biomanufacturing process machinery 160, ten instances of audio sensor 162, two instances of training audio data source 150, etc.).
[0029] It is worth noting that although the system 100A is shown to include a biomanufacturing process machine 160, a person of ordinary skill in the art will understand that the technology and components of the system 100A can be applied to detecting glass breakage events in other processes or fields. For example, in addition to the biomanufacturing process machine 160, the technology and components of the system 100A can be applied to the manufacturing of food / beverage, automotive, electronics, chemicals and / or other industries.
[0030] The biomanufacturing process machinery 160 may include a single biomanufacturing process machine, or a plurality of biomanufacturing process machines that are co-located or remote from one another and are suitable for producing a biological product, such as a pharmaceutical product. The biomanufacturing process machinery 160 may generally include physical equipment configured for producing (e.g., manufacturing) a biological product (e.g., a pharmaceutical product), such as a filling device, a mixing device, a star wheel or other container transport, and the like.
[0031] In some embodiments, the biomanufacturing process machine 160 may be connected to the computing device 110 via the network 170 or directly, thereby allowing at least some functions of the biomanufacturing process machine 160 to be controlled by the computing device 110. In some embodiments, the biomanufacturing process machine 160 may be able to receive instructions directly from a user (e.g., the biomanufacturing process machine 160 may be manually configured). For example, in some embodiments, the biomanufacturing process machine 160 may receive instructions directly from a user to control an operation (e.g., to start or stop an operation).
[0032] The audio sensor 162 may be included in the biomanufacturing process machine 160 (e.g., integrated into the biomanufacturing process machine 160), or may be an external sensor connected to the biomanufacturing process machine 160. The audio sensor 162 may be used to (e.g., directly or indirectly) collect audio data inside, outside, or around the biomanufacturing process machine 160. The audio sensor 162 may provide audio data to, for example, the computing device 110 (e.g., via the network 170). The audio data may be any suitable data type, such as categorized data, sequenced data, discrete data, or continuous data. The audio data may be in the form of a suitable data structure, which may be stored in a suitable format, such as one or more of the following formats: M4A, FLAC, MP3, MP4, WAV, WMA, AAC, JSON, XML, CSV, etc. The audio data may be collected or provided automatically or in response to a request. For example, a user of the computing device 110 may wish to monitor glass breaking events in the biomanufacturing process machine 160 over a certain period of time. In response, one or more of the audio sensors 162 may collect audio data over the period of time and provide it to the computing device 110. In some aspects, the audio sensors 162 can collect audio data in response to the operation of the biomanufacturing process machine 160. For example, the audio sensors 162 can begin collecting audio data when the biomanufacturing process machine 160 is powered on / begins operation, and can continue to collect audio data until the biomanufacturing process machine 160 is powered off / ends operation. In some embodiments, one or more of the audio sensors 162 can include a database of data / information related to product quality, or can be configured to receive data / information related to product quality, such as via user input.
[0033] The biomanufacturing process machine 160 may include one or more devices (not shown) for manufacturing a biological product (e.g., a pharmaceutical product as discussed in the background section). The biomanufacturing process machine 160 may be configured to be controllable via manual or automatic input. In some embodiments, the biomanufacturing process machine 160 may be configured to receive such control input locally, such as via a user input device local to the biomanufacturing process machine 160. In some embodiments, the biomanufacturing process machine 160 is configured to receive control input remotely, such as from a computing device 110 (e.g., via a network 170). The control input may include an operating instruction, such as instructing the biomanufacturing process machine 160 to power on / start operation. In some aspects, the biomanufacturing process machine 160 may terminate the operation in response to one or more of the following: (i) the biomanufacturing process machine 160 completes the production of the biological product (e.g., completes the production of a batch of pharmaceutical products), (ii) a glass breakage event is detected, or (iii) a manual instruction to terminate the operation is received.
[0034] The training audio data source 150 generally includes training audio data that may correspond to one or more biomanufacturing processes (e.g., may be collected during the execution of one or more biomanufacturing processes) for producing one or more bioproducts using the biomanufacturing process machinery 160. The training audio data may represent: (i) training environmental sounds, (ii) training glass sounds, and (iii) training glass breaking sounds, and may be collected using (multiple) audio sensors 162 or other similar sensors (by the computing device 110 or another device / system). More specifically, in some aspects, (i) training environmental sounds include sounds caused by the operation of machinery (e.g., the biomanufacturing process machinery 160), (ii) training glass sounds include sounds caused by a first glass surface (e.g., a first drug product container) contacting a second glass surface (e.g., a second drug product container) or a non-glass surface, and / or (iii) training glass breaking sounds include sounds caused by glass breaking or glass shattering. Exemplary training labels may include "cracked glass", "cracked glass", "cracked glass", "glass contacting metal", "glass contacting plastic", "glass contacting glass", "glass contacting other", "environmental machinery", "environmental other", etc. In some aspects, at least a portion of the training audio data includes sounds from the biomanufacturing process machinery 160. However, in some aspects, different biomanufacturing process systems are used to collect all training audio data. The training audio data may include data from a biomanufacturing process having similar scales / sizes, settings / parameters, equipment models, etc. to the biomanufacturing process machinery 160, and / or data from a biomanufacturing process having different scales / sizes, settings / parameters, equipment models, etc. from the biomanufacturing process machinery 160. In some embodiments, the system 100A may omit the training audio data source 150, but instead receive the training audio data locally via user input such as at the computing device 110 (e.g., the user provides the training audio data to the portable memory drive). In some examples, the training audio data includes training glass sounds or training glass breaking sounds, without including any training environmental sounds combined (or mixed) with the second audio data, and the second audio data includes environmental sounds, without including any training glass sounds or training glass breaking sounds. In some aspects, the training audio data source 150 or the computing device 110 combines (or mixes) the first audio sound and the second audio sound.
[0035] The computing device 110 may include a single computing device, or multiple computing devices located at the same location or remote from each other. The computing device 110 is generally configured to apply the audio data within the time period of interest as input to a model trained using the training audio data to determine whether a glass breaking event occurred during the time period of interest by processing the audio data using the machine learning model. The components of the computing device 110 may be interconnected via an address / data bus or other means. The components included in the computing device 110 may include a processing unit 120, a network interface 122, a display 124, a user input device 126, and a memory 128, which will be discussed in further detail below.
[0036] The processing unit 120 includes one or more processors, each of which may be a programmable microprocessor that executes software instructions stored in the memory 128 to perform some or all of the functions of the computing device 110 as described herein. Alternatively, one or more processors in the processing unit 120 may be other types of processors (e.g., an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), etc.).
[0037] The network interface 122 may include any suitable hardware (e.g., front-end transmitter and receiver hardware), firmware, or software configured to communicate with an external device or system (e.g., audio sensor 162, biomanufacturing process machinery 160, training audio data source 150, etc.) via the network 170 using one or more communication protocols. For example, the network interface 122 may be or include an Ethernet interface.
[0038] The display 124 may use any suitable display technology (e.g., LED, OLED, LCD, etc.) to present information to the user, and the user input device 126 may be a keyboard or other suitable input device. In some aspects, the display 124 and the user input device 126 are integrated into a single device (e.g., a touch screen display). Typically, the display 124 and the user input device 126 may be combined to enable a user to interact with a graphical user interface (GUI) or other (e.g., text) user interface provided by the computing device 110 (e.g., for purposes such as notifying a user of broken glass).
[0039] The memory 128 includes one or more physical memory devices or units including volatile or non-volatile memory, and may or may not include memory located in different computing devices of the computing device 110. Any suitable memory type or types may be used, such as read-only memory (ROM), solid-state drive (SSD), hard disk drive (HDD), etc. The memory 128 may store instructions of one or more software applications included in a glass break (GB) application 130 that may be executed by the processing unit 120. In the example system 100A, the GB application 130 includes a data collection unit 132, a model training unit 134, a user interface unit 136, a glass break detection unit 138, and a notification unit 140. The units 132 to 140 may be different software components or modules of the GB application 130, or may simply represent functionality of the GB application 130 that is not necessarily divided between different components / modules. For example, in some embodiments, the data collection unit 132 and the user interface unit 136 are included in a single software module. Furthermore, in some embodiments, units 132 to 140 may be distributed among multiple copies of GB application 130 (e.g., multiple copies executed at different components in computing device 110), or among different types of applications stored and executed at one or more devices of computing device 110.
[0040] The data collection unit 132 is generally configured to receive data (e.g., audio data, operator instructions, etc.). In some embodiments, the data collection unit 132 receives training audio data of a biomanufacturing process for producing a biological product (e.g., historical audio data including multiple instances of the biomanufacturing process and corresponding historical audio data). The data collection unit 132 may receive the training audio data via, for example, the training audio data source 150, user input received via the user interface unit 136 and the user input device 126, or other suitable means. In some embodiments, the data collection unit 132 may receive the audio data via, for example, the audio sensor 162, user input received via the user interface unit 136 using the user input device 126, or other suitable means. In some embodiments, the computing device 110 may receive an indication that the biomanufacturing process has begun, for example, at the data collection unit 132, and one or more components of the computing device 110 may begin monitoring the audio data provided, for example, by the audio sensor 162. In some aspects, the data collection unit 132 may apply pre-processing to the received audio data, such as applying an amplification gain to one or both of the training audio signal or the audio signal, where the training audio signal and the audio signal correspond to the training audio data and the audio data, respectively.
[0041] The model training unit 134 is generally configured to generate, train or apply a model. The model can be any suitable model for detecting glass breaking events in audio data. In some embodiments, and as further discussed below, the model can be trained using at least some of the system 100A, or, in some embodiments, the model can be pre-trained (i.e., trained before being obtained by the computing device 110). The model can be trained using training audio data, which represents (i) training environmental sounds, (ii) training glass sounds, and (iii) training glass breaking sounds. In some embodiments, the model can include a statistical model that can be parametric, non-parametric or semi-parametric. A suitable example of a statistical model that can be included in the model is a linear regression model. In other embodiments, the model includes a machine learning model. For example, the model can use a neural network, such as a convolutional neural network or a deep learning neural network. Other examples of machine learning models in the model are models using support vector machine (SVM) analysis, K-nearest neighbor analysis, naive Bayes analysis, clustering, reinforcement learning or other machine learning algorithms or techniques. The machine learning model included in the model can recognize and discern patterns in the training data to facilitate prediction of new data. The model training unit 134 can train the model using training audio data received from the training audio data source 150.
[0042] The user interface unit 136 is generally configured to receive user input. In one example, the user interface unit 136 can generate a user interface for presentation via the display 124, and receive user input training audio data to be used by the model training unit 134 in training the model via the user interface and user input device 126. In another example, the user interface unit 136 can receive input via the user interface and user input device 126 to initiate operation of the biomanufacturing process machinery 160 or the audio sensor 162. The user interface unit 136 can also be used to display information. For example, the user interface unit 136 can be used to display an indication of whether a glass breaking event has been detected.
[0043] In determining whether a glass breaking event has occurred, the glass breaking detection unit 138 may also apply or access a model trained by the model training unit 134 (or a model otherwise obtained by the computing device 110 as a pre-trained model). In some embodiments, the glass breaking detection unit 138 begins monitoring for glass breaking events in response to the data collection unit 132 receiving audio data. The glass breaking detection unit 138 may monitor the audio data while the data collection unit 132 is collecting the audio data in real time, near real time (i.e., using a buffer), or asynchronously (i.e., after the audio data has been completely collected over a period of interest). It should be understood that when referring to the glass breaking detection unit 138 detecting whether a glass breaking event has occurred, this also includes detecting whether a glass breaking event is occurring (because the glass breaking detection unit 138 may perform real-time monitoring).
[0044] The notification unit 140 is generally configured to notify a user (i) if a glass-breaking event has occurred, or (ii) that a glass-breaking event has occurred. The notification unit 140 may cooperate with the user interface unit 136 to display a notification. The notification unit 140 may send an electronic message (e.g., an email, a text message, etc.) with the notification to a user of the computing device 110 or an external computing device. In some aspects, if the glass-breaking detection unit 138 detects a glass-breaking event, the notification unit 140 may send a control signal to stop the operation of the biomanufacturing process machine 160. In some aspects, the indication may be stored (e.g., in the memory 128), possibly along with other data (e.g., operational data) related to the biomanufacturing process machine 160 that may be helpful in diagnosing the cause of the glass-breaking event.
[0045] In some aspects, some or all of the functionality of the GB application 130 may be provided by a third party (i.e., a third party that is not on the computing device 110). For example, machine learning may be hosted by a third party, and the GB application 130 may remotely access the machine learning model by sending data (e.g., audio data) and receiving data (e.g., an indication of whether a glass breaking event is detected). In such an example, the functionality of the glass breaking detection unit 138 may be hosted by a third party. Turning to a different example, the machine learning model may be trained by a third party, and the GB application 130 may remotely receive the machine learning model from the third party (e.g., receiving one or more elements of the machine learning model, such as weights or architecture, via the computing device 110). In such an example, the functionality of the model training unit 134 may be hosted by a third party. In other examples, one or more instances of the functionality of any one of the units 132 to 140 may be hosted by a third party on a remote server accessible, for example, via the network 170.
[0046] Figure 1BAn exemplary system 100B is depicted representing one embodiment, wherein: Figure 1A The computing device 110 may be an Internet of Things (IoT) device and may be communicatively coupled to one or more other devices (e.g., the audio sensor 162). Figure 1B The architecture shown in is the same or similar, in which computing device 110 is a Raspberry Pi, which includes an interface for a USB microphone (which may be one or more of the audio sensors 162), an optional Coral USB TPU accelerator, and a host of Amazon Web Services (AWS) services (including SnS, IoT Core, or S3 storage buckets). Raspberry Pi can be selected as computing device 110 because its technical specifications enable recording and storage of live audio data while providing remote access and a simple user interface. As understood, Raspberry Pi also has certain cost advantages and has been well supported and technically supported by crowdsourcing software. In addition, Raspberry Pi can be easily customized and additional functions can be added. As further shown, the Coral USB TPU accelerator can be used to host and process deep learning models, thereby realizing real-time audio data processing.
[0047] Example glass breaking incident
[0048] Figure 2 An example scenario 200 is depicted in which a glass breaking event occurs. As shown, the scenario 200 occurs in a biomanufacturing process machine 260 (which may be associated with Figure 1A As shown, the events of scenario 200 occur in conjunction with IoT device 210 (which can be connected to Figure 1A 100B) and audio sensors 262A and 262B (which may be connected to the system 100B) that are communicatively coupled. Figure 1A The audio sensor 162 is the same or similar to the audio sensor 162.
[0049] Biomanufacturing process machine 260 is depicted as implementing container (glass vial, as shown) filling with drug product (liquid drug product, as shown). More specifically, some containers (including container 270A) are shown as not filled with drug product, some containers (including container 270B) are shown as filled with drug product, and container 270C is shown as partially filled with drug product. In addition, unfilled broken container 272A and corresponding debris 272B are also shown. As shown, broken container 272A or debris 272B may pose a safety hazard to, for example, an operator of biomanufacturing process machine 260 and / or an end user / consumer of the drug product.
[0050] Two audio sensors 262A and 262B are shown as included near the event of scene 200. Standard microphone 262A can be farther from dispenser 280 than surface microphone 262B. Standard microphone 262A can be an omnidirectional microphone. Surface microphone 262B can be a "contact" microphone that senses mechanical audio vibrations via direct contact with an object. Surface microphone 262B can be located on a disk (e.g., a metal disk).
[0051] Audio data corresponding to the event of container 272B breaking in scene 200 can be captured by two audio sensors 262A and 262B. In addition, two audio sensors 262A and 262B can capture ambient audio data corresponding to the operating sounds of biomanufacturing process machinery 260. In addition, two audio sensors 262A and 262B can capture the sound of glass touching glass or the sound of glass touching other surfaces (e.g., metal, wood, plastic, etc.).
[0052] The two audio sensors 262A and 262B can be communicatively coupled (e.g., via a wired connection or WLAN, etc.) to the IoT device 210. As shown, the IoT device 210 can be external to the biomanufacturing process machinery 260, while in some embodiments, the IoT device 210 can be integrated into the biomanufacturing process machinery 260. The IoT device 210 can detect a glass break event involving the container 272A and the debris 272B of the scene 200, and in response, can instruct the biomanufacturing process machinery to stop operating. As shown, the dispenser 280 that dispenses the drug product can stop dispensing the drug product in response to detecting the glass break event of the scene 200 (e.g., via the GB application 130). By quickly stopping the dispenser 280 from dispensing the drug product and the operation of the entire biomanufacturing process machinery 260, the negative impact caused by the event of the scene 200 can be mitigated (e.g., as described in the background technology section).
[0053] Exemplary Audio Data Processing
[0054] Figure 3 An example data representation 300 is depicted of processing raw audio data 310 into a mel-spectrogram 320 and tiles 330. The data representation 300 may correspond to the same or similar devices / apparatus as those discussed above in connection with the systems 100A / 100B. For example, the computing device 110 may implement / generate (e.g., using the GB application 130 of audio data collected from the biomanufacturing process machinery 160 by the audio sensor 162) some or all of the data representation 300. Further, the data representation 300 may correspond to a detection process for glass breaking events, which may be similar to Figure 2 Scene 200 of the glass breaking incident.
[0055] The raw audio data 310 is collected via an audio sensor (e.g., audio sensor 162 or two audio sensors 262A and 262B), which may be one or more of a standard microphone, a surface microphone, or other suitable microphones for collecting audio data. As shown, the raw audio data 310 may be sampled as a 16kHz mono channel as input, while in other aspects, other suitable samplings may also be feasible.
[0056] The raw audio data 310 may be processed (e.g., by the computing device 110 using the data collection unit 132) to compute a spectrogram, as shown. In some aspects, a Fourier transform (e.g., a short-time Fourier transform) is performed to compute a mel-spectrogram 320 from the raw data 310. The spectrogram is converted to a stable logarithmic mel-spectrogram (i.e., mel-spectrogram 320) by mapping the spectrogram to 64 mel bins covering a range of 125 Hz to 7500 Hz and then taking the logarithm of the mel-spectrogram 320 (adding a small buffer to avoid zeroing the logarithm).
[0057] The mel-spectrogram 320 can be processed (e.g., by the computing device 110 using the data collection unit 132) into tiles 330. As shown, the tiles 330 span 0.96 seconds with an overlap of 50%. The tiles 330 can correspond to a plurality of scores that can be generated by the model (e.g., using the GB application 130), wherein each score corresponds to a possible classification of a single tile in the tiles 330. The tiles 330 may have multiple classification possibilities, such as, for example, "cracked glass", "broken glass", "glass in contact with metal", "glass in contact with plastic", "environment", etc. It is worth noting that the scores are not necessarily calibrated and may ultimately be unitless. For example, a score of 0.5 for a particular classifier does not necessarily mean that the model has a 50% probability of detecting the corresponding classifier, and the sum of all output scores for each class is not necessarily 1.
[0058] Superimposed on the data representation 300 are time window 340A and time window 340B. Time window 340A represents the first 0.96 second audio chunk that is temporally consistent in both the original waveform and the mel-spectrogram. It should be noted that despite the 50% overlap between time window 340A and time window 340B, the scores included in the chunk 330 of time window 340B are significantly different from the scores of time window 340A.
[0059] Example Machine Learning Model
[0060] Figure 4A and Figure 4B Describes the Figure 1A The computing device 110 generates, trains, and / or uses (e.g., executes or accesses) exemplary models 400A and 400B to predict whether a glass breaking event has occurred. Models 400A and 400B can be used to predict glass breaking events using audio data, such as Figure 2 The glass breaking event in scene 200 shown in FIG. 1 may be represented by the audio data in a manner similar to that of FIG. Figure 3 The data representation 300 is processed in the manner described. Models 400A and 400B can determine whether a glass breaking event occurred during the period of interest by processing the audio data. As previously described, although as shown, models 400A and 400B correspond to machine learning models, models 400A and 400B can be statistical models or machine learning models.
[0061] In an embodiment where models 400A and 400B are machine learning models, models 400A and 400B may be general (i.e., applicable to all situations) or more specialized (i.e., different models for different situations). The machine learning model may be trained using a supervised or unsupervised machine learning program or algorithm. The machine learning program or algorithm may employ a neural network, which may be a convolutional neural network (CNN), a deep learning neural network, or a combined learning module or program that learns from two or more features or feature data sets in a specific region of interest. In one embodiment, a generative adversarial neural network may be used. The machine learning program or algorithm may also include regression analysis, support vector machine (SVM) analysis, decision tree analysis, random forest analysis, K-nearest neighbor analysis, naive Bayes analysis, clustering, reinforcement learning, or other machine learning algorithms or techniques. In some embodiments, due to the processing power requirements for training the machine learning model, additional computing resources (e.g., cloud computing resources) may be used to train the selected model based on data provided by an external source (e.g., training audio data source 150). The training data may be unlabeled, or the training data may be labeled, such as manually labeled. Training of the machine learning models can continue until at least one of the machine learning models is validated and meets selection criteria for use as a predictive model for determining whether a glass breaking event has occurred. In one embodiment, the machine learning model can be validated using a second subset of the training data to determine algorithm accuracy and robustness. Such validation can include applying the machine learning model to training data in the second subset of the training data to predict whether a glass breaking event has occurred using the second subset of the training data. The machine learning model can then be evaluated to determine whether the performance of the machine learning model is sufficient based on the predictions from the validation phase. The sufficiency criteria applied may vary depending on the size of the training data available for training, the performance of previous iterations of the machine learning model, or user-specified performance requirements.
[0062] For best results, models 400A and 400B should be computationally inexpensive to allow real-time or near real-time detection of glass breaking events (e.g., being able to process and classify live audio data at the edge, meaning processing by the device itself, or being able to send audio data to the cloud for real-time processing). It is generally preferred to maximize the predictive power of models 400A and 400B within computational constraints affected by device specifications. Because glass breaking events are extremely rare and the impact of detection is large (e.g., a biomanufacturing process is stopped), models 400A and 400B preferably have particularly strong predictive power to avoid false positive events (classifying nominal ambient noise as glass breaking events). In some aspects, models 400A and 400B should achieve a false positive rate close to 0% while still maintaining a reasonable true positive rate (the rate at which the model correctly classifies real glass breaking events). These basic design criteria make the open source CNN known as "YAMNet" a suitable choice for feasibility testing.
[0063] In some respects, CNNs are well suited for machine vision applications because of their pattern recognition capabilities. As understood, CNNs differ from standard multilayer perceptrons (MLPs) in that they use convolutional layers, in which a matrix of numbers, often called filters, is convolved with the input image to generate a tensor representing a new image with an arbitrary number of channels. This new tensor can then be convolved with a new set of filters in another convolutional layer to produce yet another tensor. This process is repeated for each layer defined in the CNN. In a typical classification task, the final output of a CNN is a set of vectors representing the predicted likelihood of each class. The filters of a CNN can be trained and selected based on recognizing different patterns such as edges, corners, or shapes.
[0064] YAMNet is a CNN that utilizes the MobileNet deep separable convolutional architecture, which is an efficient convolutional network architecture that uses two hyperparameters to build low-latency models designed for mobile and embedded vision applications. More specifically, the standard convolutional filters are replaced by two alternative filters: channel-by-channel convolution with a single filter applied to each input, and 1×1 point-by-point convolution. YAMNet is an audio classification model that combines the MobileNet (deep separable CNN) architecture, which has been pre-trained on audio data to predict different audio events. In some embodiments, YAMNet may not require any feature extraction before passing the audio data into the model because the model has a feature extraction layer built into the model. The feature extraction layer can convert the audio data into a spectrogram, which is then passed into MobileNet. As previously discussed, in order to train YAMNet using an audio signal, the audio data may need to be preprocessed. One way to preprocess the audio data is by applying an STFT to the audio data and converting the output from the STFT into a spectrogram or a Mel spectrogram using a logarithmic scale (e.g., as shown in data representation 300).
[0065] According to some basic design criteria, splitting the filtering and combining steps of a typical convolution operation allows for a significant reduction in computational cost: MobileNet produces results that are almost identical to popular neural networks AlexNet, VGG16, and GoogleNet, while being a fraction of the size and computationally intensive. Therefore, as will be appreciated by those of ordinary skill in the art, the YAMNet model architecture that employs MobileNet is very useful in IoT applications (e.g., Figure 2 YAMNet can have particular technical advantages in IoT devices 210), where computational efficiency is often limited by the device footprint. More specifically, YAMNet is pre-trained on AudioSet, an ontology of 521 audio event classes, and a collection of more than 2 million manually annotated 10-second audio clips from YouTube videos. These event classes cover a wide range of noises from musical instruments to natural sounds to various animals. Classes such as "glass", "crack", "shatter", and "break" are particularly important for detecting glass breaking events.
[0066] YAMNet has two variants: the standard YAMNet model and a TensorFlow Lite model called YAMNet-lite, which is further customized for mobile applications. Figure 3The data represents 300 processed raw audio data (sampled as 16kHz mono as input, STFT is used to calculate the spectrogram, the spectrogram is mapped to 64 mel bins covering the range of 125Hz to 7500Hz, and then the logarithm of the mel spectrogram is taken (a small buffer is added to avoid zero logarithm)). 0.96 second time windows with 50% overlap (also as Figure 3 The data representation 300 is shown in Figure 3) and fed into MobileNet, which returns 521 scores ranging from 0 to 1 for each class in the AudioSet body. Importantly, the output scores are uncalibrated and ultimately unitless. For example, a score of 0.5 for a specific classifier cannot be interpreted as a 50% probability that the model has detected the corresponding classifier, nor can it be interpreted as the sum of all output scores for each class being 1.
[0067] The YAMNet-lite model behaves similarly to the standard YAMNet model, with two main differences: the model is quantized, and therefore retrained with Relu6 nonlinearity instead of ReLU activation, and the input must be 16kHz mono audio with 0.975 second fixed frames. Logically, the output of the YAMNet-lite model is a single vector of 521 classification scores. As will be appreciated by those of ordinary skill, the YAMNet-lite model is useful for optimizing IoT devices (e.g., Figure 2 There may be further technical advantages in terms of computing costs of IoT devices 210).
[0068] refer to Figure 1B , Figure 1BThe Raspberry Pi can host a series of software modules (e.g., Python code) that interface with peripheral devices (e.g., a module for continuously recording a time window (e.g., 0.975s) of 16-bit raw PCM audio processed by the YAMNet model), call various AWS clients to send text messages to subscribers, save the audio recording to an s3 bucket in the AWS cloud, and transmit MQTT messages to AWS IoT Core. One or more modules of the Raspberry Pi can also interface with an audio stream to save the audio locally to the Raspberry Pi as a backup. One or more modules of the Raspberry Pi can also interface with a USB microphone to initialize continuous audio streaming and recording capabilities. One or more modules of the Raspberry Pi can also initialize the YAMNet-lite model and the model's hyperparameters, where the model can be hosted locally on the Raspberry Pi or hosted on a Coral USB accelerator to increase processing power. One or more modules of the Raspberry Pi can also initialize connections to various AWS clients. The individual modules of the Raspberry Pi can also work together to continuously record and classify time windows (e.g., 0.975s) of live audio using the YAMNet model architecture. A communication protocol has been established via AWS to enable the device to store audio in the cloud. Additionally, if conditional heuristics are established (e.g., if the output of the YAMNet model is > 0.2 for the "glass" classifier), the device can be configured to automatically send alerts to all interested parties that a glass-breaking event may have occurred.
[0069] Models 400A and 400B provide a visual representation of the YAMNet architecture optimized for mobile audio detection applications. Given the predictive power of YAMNet's pre-trained audio classifiers for categorized audio data, models 400A and 400B can be trained via transfer learning (i.e., improving the learning of new tasks by transferring knowledge from related tasks that have already been learned). In the context of models 400A and 400B, the architecture of the YAMNet model can employ transfer learning and include additional custom trained classifiers. Model 400A is a visual representation of the YAMNet-lite model architecture, which receives two new classifiers as depicted in model 400B. Models 400A and 4000B return a 2×1 classification score vector. The "Process A" classifier of model 400B is trained on process A audio data, while the "Broken Event" classifier of model 400B is trained on glass shattering event sounds. Process A refers to audio data from an integrated downstream fill-finish process (an example of a biomanufacturing process); a representative environment for handling glass containers, and therefore, glass breakage may occur in this process.
[0070] Exemplary Glass Break Event Detection Performance
[0071] Figure 5A and Figure 5B Exemplary experimental data 500A and 500B are shown, which represent the performance of one embodiment of the presently disclosed technology in determining whether a glass breaking event occurred during a period of interest by processing audio data using a machine learning model. The machine learning model corresponding to data 500A can be trained or used via system 100A / 100B to detect glass breaking events (e.g., glass breaking events of scene 200) using the same or similar model as model 400A or model 400B that processes audio data (which can be pre-processed according to data representation 300).
[0072] In some aspects, the gain of an audio signal of audio data used by a machine learning model corresponding to data 500A or data 500B can be increased (e.g., by IoT device 210 or computing device 110). Increasing the gain of the audio signal can affect the performance of the machine learning model in predicting whether a glass breaking event occurred during a time period included in the audio data. Increasing the gain can separate the glass breaking sound from other sounds, thereby improving the ability of the machine learning model to distinguish the sounds. Determining the preferred gain amount can be performed using an iterative approach that is application specific and meets performance adequacy criteria.
[0073] As shown in data 500A, a gain level of +16 dB is selected (e.g., by an operator) for the audio signal of the audio data input to the machine learning model. Data 500A corresponds to the performance of the machine learning model in classifying a closed data set when trained on the previously described AudioSet data (training audio data). Data 500A includes a receiver operating characteristic curve (ROC) 510A of the machine learning model and a binary confusion matrix 520A with a threshold score of 0.003. As demonstrated by data 500A, the machine learning model is close to a perfect classifier. When the score threshold is ≥0.003, the true positive rate is 90% and the false positive rate is 1%. This result demonstrates the predictive performance of the machine learning model well. The main limitation of data 500A is the simulation environment of the audio data: although real biomanufacturing process glass containers are used to generate audio data, the containers are empty (e.g., not filled with biopharmaceuticals). In addition, the audio recordings were not obtained from the production line, so the ambient sounds of the environment where glass breakage events may occur cannot be accurately simulated.
[0074] Data 500B addresses some of the limitations of data 500A by predicting whether a glass breaking event has occurred for audio data representing (i) environmental sounds (e.g., laboratory sounds, equipment sounds, mechanical sounds, etc.), (ii) glass sounds (e.g., glass contacting another material that may be glass or non-glass), and (iii) training glass breaking sounds (e.g., cracking, shattering, shattering, etc.). Data 500B corresponds to a custom trained machine learning model that uses the architecture of model 400B and two classifiers (i.e., the "Process A" classifier is trained on Process A audio data, and the "Breaking Event" classifier is trained on glass breaking event noise). Data 500B demonstrates that the machine learning model performs nearly perfectly in classifying breaking events and Process A audio data, with a false positive rate of 0% when the score threshold is >0.02. In addition, the change in microphone gain is negligible. Therefore, data 500B demonstrates that the present technology can successfully distinguish glass breaking sounds from other sounds (i.e., environmental sounds and glass sounds).
[0075] Exemplary Flowchart
[0076] Figure 6is a flow chart depicting an example method 600 for identifying a glass-breaking event (e.g., a glass-breaking event of scene 200). Method 600 may be implemented by one or more components of system 100A / 100B, such as processing unit 120 when implementing GB application 130, and possibly biomanufacturing process machinery 160 (which may operate a biomanufacturing process). Method 600 may be performed as part of a process that is the same or similar to data representation 300. Method 600 may receive audio data for a time period of interest (e.g., from biomanufacturing process machinery 160 and audio sensor 162). Example method 600 may include the following elements: (1) accessing or obtaining a machine learning model trained using training audio data (block 602); (2) obtaining audio data (block 604); (3) determining whether a glass-breaking event has occurred by processing the audio data using the machine learning model (block 606); and (4) indicating that a glass-breaking event has occurred when it is determined that a glass-breaking event has occurred.
[0077] The trained machine learning model obtained (e.g., downloaded, generated, trained, etc.) or accessed (e.g., accessed to a third-party remote server hosting the machine learning model) at block 602 may have been trained (e.g., as described above) using training audio data (e.g., training audio data included in the training audio data source 150). In some aspects, the training audio data may include one or more spectrograms that may be the same or similar to the mel-spectrogram 320. The training audio data may represent (i) training environmental sounds, (ii) training glass sounds, and (iii) training glass breaking sounds. More specifically, (i) training environmental sounds may include sounds caused by mechanical operation, (ii) training glass sounds may include sounds caused by a first glass surface contacting a second glass surface or a non-glass surface, and (iii) training glass breaking sounds may include sounds caused by glass breaking or glass shattering. In some aspects, the machine learning model is a convolutional neural network or is trained using supervised learning techniques. The training audio data may include (i) environmental sound labels, (ii) glass sound labels, and (iii) breaking sound labels. In some aspects, the machine implements a biomanufacturing process (e.g., via biomanufacturing process machine 160), and the training glass sound and the training glass breaking sound are generated by one or more containers of one or more pharmaceutical products. In some embodiments, obtaining the machine learning model at block 602 includes receiving a pre-trained model (i.e., trained prior to being obtained by, for example, system 100), or generating / training the machine learning model (e.g., by system 100). The machine learning model can be obtained internally (e.g., by accessing a file / program / data / information stored locally in a computing system such as computing device 110) or externally (e.g., by receiving the machine learning model from an external source, such as receiving the model at computing device 110 via network 170).
[0078] Block 604 may include obtaining audio data for a time period of interest. The audio data may be input or provided by a user via a user interface (e.g., using a user input device 126) or by collecting the audio data as data (e.g., via a data collection unit 136). The audio data may be collected by an audio sensor (e.g., an audio sensor 162) that may be located inside, outside, or near a biomanufacturing process machine (e.g., a biomanufacturing process machine 160). In some aspects, similar to the training audio data, the audio data may include one or more spectrograms that may be the same or similar to the Mel spectrogram 320, and obtaining the audio data for a time period of interest includes generating a spectrogram of the audio data from the original audio data using a Fourier transform (e.g., SFTF). In some aspects, the audio data may be preprocessed, such as (i) as described in the data representation 300, or (ii) by iteratively adjusting the amplification gain applied to one or both of the training audio signal or the audio signal by one or more processors until a performance threshold is met, wherein the training audio signal and the audio signal correspond to the training audio data and the audio data, respectively.
[0079] Block 606 may include determining whether a glass-breaking event has occurred during the period of interest (which may include a glass-breaking event having occurred or is occurring) by processing the audio data using a machine learning model. The determination of whether a glass-breaking event has occurred may be a single binary value (e.g., 0 indicating that a glass-breaking event has not occurred, 1 indicating that a glass-breaking event has occurred), or multiple values representing a probability of a glass-breaking event occurring (e.g., a fraction representing a probability range of 0% to 100%). In some aspects, the machine learning model may be a CNN, such as YAMNet or YAMNet-lite (as described herein), a linear regressor, a random forest model, a model using support vector machine (SVM) analysis, K-nearest neighbor analysis, naive Bayes analysis, clustering or reinforcement learning, or another suitable machine learning model. In some embodiments, a statistical model (e.g., a linear regression model) or some other suitable statistical model may be used to supplement or replace a machine learning model for identifying glass-breaking events.
[0080] In some aspects, method 600 may end at block 608, which may include indicating (e.g., via a computing device, such as computing device 110) that a glass-breaking event has occurred. In some aspects, the indication may be visual (e.g., displayed on display 124), audible (e.g., played on a speaker), tactile, or some other suitable notification method. In some aspects, the notification may include electronic messaging, such as sending a text message or an email message to a user (e.g., an operator of a biomanufacturing process machine). In some aspects, the indication may be stored, possibly along with other data (e.g., operational data) related to the biomanufacturing process that may be helpful in diagnosing the cause of the glass-breaking event. In some aspects, upon identifying the glass-breaking event, block 607 may include automatically stopping operation of the biomanufacturing process machine.
[0081] In some aspects, method 600 can be performed entirely by automation, for example, by one or more processors (e.g., CPU or GPU) executing instructions stored on one or more non-transitory computer-readable storage media (e.g., volatile or non-volatile memory, read-only memory, random access memory, flash memory, electronically erasable program read-only memory, or one or more other types of memory). Method 600 can use any of the components, processes, or techniques of one or more of Figures 1 to 5.
[0082] Exemplary Flowchart
[0083] Figure 7 is a flow chart depicting an example method 700 for training a machine learning model for identifying glass breaking events (e.g., glass breaking events of scene 200). Method 700 may be implemented by one or more components of system 100A / 100B, such as processing unit 120 when implementing GB application 130, and possibly biomanufacturing process machinery 160 (which may operate a biomanufacturing process), or by a different device or system (e.g., a third party server that develops and hosts machine learning models). Method 700 may be performed as part of a process that is the same or similar to data representation 300. Method 700 may receive training audio data for a time period of interest (e.g., from training audio data source 150). Example method 700 may include the following elements: (1) obtaining training audio data (block 702), (2) classifying the training audio data into a plurality of subsets (block 704), and (3) generating a machine learning model for identifying glass breaking events using the classified subsets (block 706).
[0084] Box 702 may include obtaining training audio data (e.g., from a training audio data source 150, at a computing device via a data collection unit 132, etc.). In some aspects, the training audio data may include one or more spectrograms that may be the same as or similar to the mel-spectrogram 320. The training audio data may represent (i) training environmental sounds, (ii) training glass sounds, and (iii) training glass breaking sounds. More specifically, (i) training environmental sounds may include sounds caused by mechanical operation, (ii) training glass sounds may include sounds caused by a first glass surface contacting a second glass surface or a non-glass surface, and (iii) training glass breaking sounds may include sounds caused by glass breaking or glass shattering. In some aspects, the training audio data includes (i) environmental sound labels, (ii) glass sound labels, and (iii) breaking sound labels. In some aspects, the machine implements a biomanufacturing process (e.g., via the biomanufacturing process machine 160), and the training glass sounds and training glass breaking sounds are generated by one or more containers of one or more pharmaceutical products. In some aspects, the training audio data can be pre-processed (e.g., in the same or similar manner as data representation 300) by iteratively adjusting an amplification gain applied to a training audio signal until a performance threshold is met, where the training audio signal corresponds to the training audio data.
[0085] Block 704 may include classifying the training audio data into a plurality of subsets, each subset corresponding to a different actual result data (meaning a particular result or range of results), the subsets including (i) at least one subset representing training environmental sounds, (ii) at least one subset representing training glass sounds, and (iii) at least one subset representing training glass breaking sounds. In some aspects, classifying the training audio data into a plurality of subsets may be performed by a computing device (e.g., computing device 110 using GB application 130 and model training unit 134) based on, for example, labels associated with the training audio data.
[0086] In some aspects, method 700 can end at box 706 by using the classified subset of the training audio data to generate a machine learning model for identifying glass breaking events. In some aspects, the machine learning model can be a CNN, such as YAMNet or YAMNet-lite (as described herein), a linear regressor, a random forest model, a model using support vector machine (SVM) analysis, K-nearest neighbor analysis, naive Bayes analysis, clustering or reinforcement learning, or another suitable machine learning model. In some embodiments, a statistical model (e.g., a linear regression model) or some other suitable statistical model can be used to supplement and replace the machine learning model for identifying glass breaking events. The machine learning model can be stored using a computing device, such as computing device 110 (e.g., specifically using memory 128).
[0087] In some aspects, method 700 can be performed entirely by automation, for example, by one or more processors (e.g., a CPU or GPU) executing instructions stored on one or more non-transitory computer-readable storage media (e.g., volatile or non-volatile memory, read-only memory, random access memory, flash memory, electronically erasable program read-only memory, or one or more other types of memory). Method 600 can be performed using FIGS. 1 to 6. Figure 6 Any component, process or technique of one or more of the above.
[0088] Additional considerations
[0089] Some figures described herein show example block diagrams with one or more functional components. It will be understood that such block diagrams are for illustrative purposes, and the described and illustrated devices may have additional, fewer, or alternative components than those shown. Additionally, in various aspects, components (and the functions provided by the corresponding components) may be associated with or otherwise integrated as a part of any suitable component.
[0090] Some aspects of the present disclosure relate to non-transitory computer-readable storage media having thereon instructions / computer-readable storage media for performing various computer-implemented operations. The term "instructions / computer-readable storage media" is used herein to include any medium capable of storing or encoding a series of instructions or computer code for performing the operations, methods, and techniques described herein. The media and computer code may be media and computer code specifically designed and constructed for the purposes of various aspects of the present disclosure, or the media and computer code may be of a type known and available to those skilled in the art of computer software. Examples of computer-readable storage media include, but are not limited to: magnetic media, such as hard disks, floppy disks, and magnetic tapes; optical media, such as CD-ROMs and holographic devices; magneto-optical media, such as optical disks; and hardware devices specifically configured to store and execute program code, such as ASICs, programmable logic devices ("PLDs"), and ROM and RAM devices.
[0091] The example of computer code includes machine code such as produced by a compiler, and files containing higher-level codes executed by a computer using an interpreter or compiler. For example, Java, C++ or other object-oriented programming languages and development tools can be used to implement aspects of the present disclosure. Additional examples of computer code include encrypted codes and compressed codes. In addition, aspects of the present disclosure can be downloaded as a computer program product, which can be transmitted to a requesting computer (for example, a computer or different server computers) from a remote computer (for example, a server computer) via a transmission channel. Another aspect of the present disclosure can be implemented by replacing machine executable software instructions with hard-wired circuitry or in combination with machine executable software instructions.
[0092] As used herein, unless the context clearly indicates otherwise, the singular terms "a", "an" and "the" may include plural references. Unless otherwise expressly stated or clearly intended, this specification and subsequent claims should be understood to include one or at least one, and the singular also includes the plural. As used herein, the terms "comprise", "include", "have" or any other variation thereof are intended to cover non-exclusive inclusions. For example, a process, method, article or device comprising a series of elements is not necessarily limited to those elements, but may include other elements that are not explicitly listed or inherent to such processes, methods, articles or devices. In addition, unless expressly stated to the contrary, "or" refers to an inclusive or rather than an exclusive or. For example, conditions A or B satisfy any of the following: A is true (or exists) and B is false (or does not exist), A is false (or does not exist) and B is true (or exists), and both A and B are true (or exist).
[0093] As used herein, the terms "approximately," "substantially," "substantially," "roughly," and "about" are used to describe and explain small variations. When used in conjunction with an event or situation, these terms may refer to situations where the event or situation happens to occur as well as situations where the event or situation occurs approximately. For example, when used in conjunction with a numerical value, these terms may refer to a range of variation where the numerical value is less than or equal to ±10%, such as less than or equal to ±5%, less than or equal to ±4%, less than or equal to ±3%, less than or equal to ±2%, less than or equal to ±1%, less than or equal to ±0.5%, less than or equal to ±0.1%, or less than or equal to ±0.05%. For example, if the difference between the values is less than or equal to ±10% of the average value of the values, such as less than or equal to ±5%, less than or equal to ±4%, less than or equal to ±3%, less than or equal to ±2%, less than or equal to ±1%, less than or equal to ±0.5%, less than or equal to ±0.1%, or less than or equal to ±0.05%, then two numerical values may be considered to be "substantially" the same.
[0094] In addition, quantities, ratios, and other numerical values are sometimes presented in a range format herein. It should be understood that this range format is used for convenience and brevity, and should be flexibly interpreted to include the values explicitly specified as the limits of the range, but also include all individual values and subranges contained within the range, just as if each value or subrange was explicitly specified.
[0095] Although the technology disclosed herein has been described primarily with specific operations performed in a specific order, it should be understood that these operations can be combined, subdivided, or reordered to form equivalent technologies without departing from the teachings of the present disclosure. Therefore, unless specifically indicated herein, the order and grouping of operations are not limitations of the present disclosure.
Claims
1. A computer-implemented method for identifying glass breaking events using machine learning, The method includes: accessing or obtaining, by one or more processors, a machine learning model trained using training audio data, the training audio data representing (i) training environment sounds, (ii) training glass sounds, and (iii) training glass breaking sounds; obtaining, by the one or more processors, audio data for a time period of interest; processing, by the one or more processors, the audio data using the machine learning model to determine whether a glass breaking event occurred during the time period of interest; as well as When it is determined that the glass breaking event has occurred, the one or more processors indicate that the glass breaking event has occurred.
2. The computer-implemented method of claim 1, wherein: The training audio data and the audio data each include one or more spectrograms; and Obtaining the audio data within the period of interest includes generating a spectrogram of the audio data based on the original audio data using Fourier transform.
3. A computer-implemented method as claimed in any one of the preceding claims, wherein: (i) These training environment sounds include sounds caused by mechanical operation, (ii) the training glass sounds include sounds caused by a first glass surface contacting a second glass surface or a non-glass surface, and (iii) These training glass breaking sounds include sounds caused by glass breaking or glass shattering.
4. The computer-implemented method of claim 3, wherein: The machine implements a biomanufacturing process, and the training glass sounds and the training glass breaking sounds are produced by one or more containers of one or more pharmaceutical products.
5. The computer-implemented method of claim 3 or 4, further comprising: After identifying the glass break event, the machine is automatically stopped from operating by the one or more processors.
6. The computer-implemented method of any preceding claim, further comprising: An amplification gain applied to one or both of the training audio signals or the audio signals is iteratively adjusted by the one or more processors until a performance threshold is met, wherein the training audio signals and the audio signals correspond to the training audio data and the audio data, respectively.
7. A computer-implemented method as claimed in any one of the preceding claims, wherein: The machine learning model is a convolutional neural network.
8. A computer-implemented method as claimed in any one of the preceding claims, wherein: The machine learning model is trained using supervised learning techniques, and The audio data includes (i) an environmental sound label, (ii) a glass sound label, and (iii) a shattering sound label.
9. A computer system for identifying glass breaking events using machine learning, the computer system comprising: one or more processors; a program memory coupled to the one or more processors and storing executable instructions that, when executed by the one or more processors, cause the computer system to: accessing or obtaining a machine learning model trained using training audio data representing (i) training environment sounds, (ii) training glass sounds, and (iii) training glass breaking sounds, Get the audio data for the time period of interest, Determining whether a glass breaking event occurred during the period of interest by processing the audio data using the machine learning model, and When it is determined that the glass breaking event has occurred, it is indicated that the glass breaking event has occurred.
10. The computer system of claim 9, wherein: The training audio data and the audio data each include one or more spectrograms; and Obtaining the audio data within the period of interest includes generating a spectrogram of the audio data based on the original audio data using Fourier transform.
11. A computer system as claimed in claim 9 or 10, wherein: (i) These training environment sounds include sounds caused by mechanical operation, (ii) the training glass sounds include sounds caused by a first glass surface contacting a second glass surface or a non-glass surface, and (iii) These training glass breaking sounds include sounds caused by glass breaking or glass shattering.
12. The computer system of claim 11, wherein: The machine implements a biomanufacturing process, and the training glass sounds and the training glass breaking sounds are produced by one or more containers of one or more pharmaceutical products.
13. The computer system of claim 11 or 12, wherein: The instructions, when executed by the one or more processors, further cause the computer system to: After the glass breaking event is identified, the machine is automatically stopped from operating.
14. The computer system according to any one of claims 9 to 13, wherein: The instructions, when executed by the one or more processors, further cause the computer system to: An amplification gain applied to one or both of the training audio signals or the audio signals corresponding to the training audio data and the audio data, respectively, is iteratively adjusted until a performance threshold is met.
15. The computer system according to any one of claims 9 to 14, wherein: The machine learning model is a convolutional neural network.
16. The computer system of any one of claims 9 to 15, wherein: The machine learning model is trained using supervised learning techniques, and The audio data includes (i) an environmental sound label, (ii) a glass sound label, and (iii) a shattering sound label.
17. A computer-implemented method for training a machine learning model for identifying glass breaking events, the method comprising: obtaining, by one or more processors, training audio data; classifying, by the one or more processors, the training audio data into a plurality of subsets, each subset corresponding to different actual result data, the subsets comprising (i) at least one subset representing training environmental sounds, (ii) at least one subset representing training glass sounds, and (iii) at least one subset representing training glass breaking sounds; as well as The classified subset of the training audio data is used by the one or more processors to generate a machine learning model for identifying glass breaking events.
18. The computer-implemented method of claim 17, wherein: The training audio data includes one or more spectrograms.
19. The computer-implemented method of claim 17 or 18, wherein: (i) These training environment sounds include sounds caused by mechanical operation, (ii) the training glass sounds include sounds caused by a first glass surface contacting a second glass surface or a non-glass surface, and (iii) These training glass breaking sounds include sounds caused by glass breaking or glass shattering.
20. The computer-implemented method of claim 19, wherein: The machine implements a biomanufacturing process, and the training glass sounds and the training glass breaking sounds are produced by one or more containers of one or more pharmaceutical products.
21. The computer-implemented method of any one of claims 17 to 20, further comprising: Amplification gains applied to training audio signals corresponding to the training audio data are iteratively adjusted by the one or more processors until a performance threshold is met.
22. The computer-implemented method of any one of claims 17 to 21, wherein: The machine learning model is a convolutional neural network.
23. One or more tangible non-transitory computer readable media storing executable instructions for training a machine learning model for identifying glass break events, which when executed by one or more processors of a computer system, cause the computer system to: Obtain training audio data; Classifying the training audio data into a plurality of subsets, each subset corresponding to different actual result data, the subsets comprising (i) at least one subset representing training environment sounds, (ii) at least one subset representing training glass sounds, and (iii) at least one subset representing training glass breaking sounds; as well as A classified subset of the training audio data is used to generate a machine learning model for identifying glass breaking events.
24. One or more tangible, non-transitory computer-readable media as recited in claim 23, wherein: The training audio data includes one or more spectrograms.
25. One or more tangible, non-transitory computer-readable media as claimed in claim 23 or 24, wherein: (i) These training environment sounds include sounds caused by mechanical operation, (ii) the training glass sounds include sounds caused by a first glass surface contacting a second glass surface or a non-glass surface, and (iii) These training glass breaking sounds include sounds caused by glass breaking or glass shattering.
26. One or more tangible, non-transitory computer-readable media as recited in claim 25, wherein: The machine implements a biomanufacturing process, and the training glass sounds and the training glass breaking sounds are produced by one or more containers of one or more pharmaceutical products.
27. One or more tangible, non-transitory computer readable media as claimed in any one of claims 23 to 26, wherein: These executable instructions, when executed by one or more processors of the computer system, further cause the computer system to: Amplification gains applied to the training audio signals are iteratively adjusted until a performance threshold is met, wherein the training audio signals correspond to the training audio data.
28. The computer-implemented method of any one of claims 17 to 21, wherein: The machine learning model is a convolutional neural network.
