Federated Learning for Anomaly Detection in Industrial Equipment Using Sound

Abstract

In some example embodiments, a method may be provided, the method comprising receiving a machine learning model provided by a central server configured to provide federated learning; receiving first audio data obtained from at least one audio sensor monitoring at least one machine located at a first edge node; training the machine learning model based on the first audio data; providing parameter information to the central server to enable federated learning between multiple edge nodes; receiving an aggregated machine learning model provided by the central server; detecting an abnormal state of at least one machine. Related systems, methods, and articles of manufacture are also described.

Description

Technical Field

[0001] The present disclosure relates generally to machine learning and IoT devices. Background Art

[0002] Many organizations may rely on enterprise software applications, including, for example, enterprise resource planning (ERP) software, customer relationship management (CRM) software, etc. These enterprise software applications may provide a variety of functions, including, for example, plant operations and maintenance, invoicing, procurement, payroll management, time and attendance management, recruitment and onboarding, learning and development, performance and compensation, workforce planning, equipment tracking, etc. Some enterprise software applications may be hosted by cloud computing platforms so that the functions provided by the enterprise software applications can be remotely accessed by multiple end users at computing devices. For example, enterprise software applications may be available as cloud-based services (including, for example, software as a service (SaaS)). Summary of the invention

[0003] In some example embodiments, a method may be provided, the method comprising: receiving, at a first edge node, a machine learning model provided by a central server, the central server being configured to provide federated learning between a plurality of edge nodes including the first edge node; receiving, at the first edge node, first audio data obtained from at least one audio sensor, the at least one audio sensor monitoring at least one machine located at the first edge node; training the machine learning model at the first edge node, the training being based on the first audio data; in response to the training, providing, by the first edge node, parameter information to the central server so as to enable federated learning between a plurality of edge nodes including the first edge node; receiving, by the first edge node, an aggregated machine learning model provided by the central server in response to the federated learning; detecting, by the first edge node including the aggregated machine learning model, an abnormal state of at least one machine, the abnormal state being detected based on second audio data received from the at least one audio sensor; and reporting, by the first edge node, the abnormal state to the central server.

[0004] In some variations, one or more of the features disclosed herein including the following features may be optionally included in any feasible combination. The parameter information may include configuration information about a machine learning model trained by a first edge node. The configuration information may include one or more weights of the trained machine learning model. The first audio data may be preprocessed by filtering the first audio data at least based on the arrival time to at least partially remove other audio data associated with other machines. The first audio data may be preprocessed by at least transforming the first audio data to the frequency domain. The machine learning model may include at least one of a convolutional neural network or a generative adversarial network. The aggregated machine learning model may include an average of the parameter information provided by the first edge node and other parameter information provided by other edge nodes in the plurality of edge nodes. The aggregated machine learning model may be generated without access to the first audio data at the first edge node or other audio data at other edge nodes by the central server. The training may also include providing the first audio data as an input to the machine learning model, the first audio data including timestamp information and label information indicating whether an anomaly exists. The first audio data may correspond to training data for training the machine learning model. The second audio data may represent operational data without label information indicating whether an anomaly exists.

[0005] A non-transitory computer program product (i.e., a physically embodied computer program product) storing instructions is also described, and the instructions, when executed by one or more data processors of one or more computing systems, cause at least one data processor to perform the operations described herein. Similarly, a computer system that may include one or more data processors and a memory coupled to the one or more data processors is also described. The memory may temporarily or permanently store instructions that cause at least one processor to perform one or more of the operations described herein. In addition, each method may be implemented by one or more data processors within a single computing system or distributed in two or more computing systems. Such computing systems may be connected and may exchange data and / or commands or other instructions, etc., via one or more connections (including connections through a network (e.g., the Internet, a wireless wide area network, a local area network, a wide area network, a wired network, etc.), via direct connections between one or more of a plurality of computing systems, etc.).

[0006] The details of one or more variations of the subject matter described herein are set forth in the accompanying drawings and the description below.Other features and advantages of the subject matter described herein will be apparent from the description and drawings, and from the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate certain aspects of the subject matter disclosed herein and, together with the description, help explain some of the principles associated with the disclosed embodiments.

[0008] Figure 1A-1B An example of a system according to some implementations of the current subject matter is shown;

[0009] Figure 2A-2C depicts examples of sensors, such as acoustic sensors, according to some implementations of the current subject matter;

[0010] Figure 3 depicts an example of a machine learning model configured to detect anomalies and determine when to perform maintenance based on audio sensor data in accordance with some implementations of the current subject matter;

[0011] Figure 4 depicts examples of user interfaces according to some implementations of the current subject matter;

[0012] Figure 5A-5B illustrates an example of a process according to some implementations of the current subject matter; and

[0013] Figure 6 is another example of a system according to some implementations of the current subject matter. DETAILED DESCRIPTION

[0014] Factories and other types of facilities and industries, such as mining, oil and gas, etc., may include machinery, such as pumps, motors, fans, valves, slides, actuators, and / or other types of machines. From time to time, these machines may require maintenance in the form of replacement, adjustment, etc. If maintenance is performed too frequently (such as before it is actually needed), maintenance can lead to unnecessary downtime. However, if maintenance is not performed frequently enough, there is an increasing chance that the machine will fail unexpectedly, resulting in unplanned downtime.

[0015] Machine learning can be used to learn when to provide maintenance to a machine. For example, a machine learning (ML) model can learn to determine when maintenance should be performed on a machine (e.g., predict that a failure has occurred or predict that a failure is about to occur). In some cases, sensors can be used to monitor the machine and provide sensor data to the ML model so that the ML model learns to predict when the machine needs maintenance to avoid, for example, a failure. When the ML model is trained, the ML model can detect anomalies that indicate that the machine needs maintenance.

[0016] However, in the case of machine learning-based solutions, sensor data from one or more of the machines being monitored may result in a huge amount of data and may (if a remote cloud platform is used to host the ML model) require ultra-reliable, low-latency communications to couple to the remote cloud platform. And due to the appropriateness and / or private nature of the sensor data being collected from the machine, the end user may be reluctant to share the raw sensor data with the remote cloud server or share the sensor data with other end users with the same or similar type of machine. In this way, the ML model may be hosted at the "edge" of the network (e.g., where the factory or monitored machine is located). However, using only sensor data available locally at the factory or machine may provide a machine learning model that is trained with an insufficient dataset and is therefore less robust than a machine learning model that benefits from a larger, more robust, and larger training dataset from multiple end user locations.

[0017] In some example embodiments, a federated learning model is provided that uses a central server that maintains a master machine learning model and provides a copy of the master machine learning model to each of the edge nodes (such as, for example, a factory where the machine is located as described in the previous example). At each of the edge nodes, a copy of the master machine learning model is used locally as a local machine learning model. For example, at a given edge node, sensor data (obtained locally from at least one sensor monitoring the machine) is used to locally train the local machine learning model. Next, each of the edge nodes can provide feedback to the central server, such as parameters of the local machine learning model (e.g., weights, gradients, and / or other parameters for the configuration of the local machine model), but the edge nodes do not share local sensor data. The central server can then aggregate the parameter information received from the edge nodes to update the master machine learning model. The parameter information refers to parameters that define the configuration of the ML model, such as the weights of a neural network. This updated master machine learning model (also referred to herein as an aggregated learning model) can be replicated again and shared with the edge nodes (to repeat the process). In federated learning, the central server can pool the learning that occurs at each of the machine learning models of the edge nodes without the central server having access to the local sensor data at the given edge node.

[0018] Figure 1A An example of a system 100 is depicted in accordance with some example embodiments. The system 100 may include a central server 182 that may be implemented in a cloud platform 180. The central server may be configured to control a federated learning process at each of the edge nodes, which in this example are located at the plant 112A and the plant 112B (although other types of locations may also host edge nodes).

[0019] The central server 182 may be coupled to a central data store 184. In some embodiments, the central server may provide a copy of the master ML model 186 to each of the edge nodes, which serves as a local ML model 106A-C at each of the edge nodes. These local ML models may be one or more of: a pre-configured default ML model (e.g., an ML model that has been at least partially trained); an initialized local ML model (e.g., an ML model that has not yet been trained, so the ML model may have random or initialized settings as parameters); or an updated copy of the master ML model.

[0020] As the local edge nodes at the plants 112A and 112B train their local ML models 106A-C, the central server 182 may also receive parameter information, such as ML model parameter information or neural network parameter information (e.g., weights, gradients, and / or other information about the local ML model or neural network being trained), from each edge node. As described above, the edge nodes share parameter information describing the configuration of the local ML model, but the edge nodes do not share local sensor data with the central server. The central server may process (e.g., aggregate, combine, average, etc.) the parameter information received from the edge nodes. The central server may use the processed parameter information to update the master ML model 186. The central server may then repeat the process by providing (e.g., publishing, sending, etc.) the updated master ML model 186 to each of the edge nodes, which is again used as the local ML model 106A-B at each of the edge nodes. This process may be repeated until the ML models converge (or reach a threshold accuracy for determining when a given machine is maintained). In short, the central server is responsible for aggregating parameter information from each of the edge nodes in order to generate a master ML model (which takes into account the training or learning that occurs across the edge nodes). And, the master ML model is trained with the central server, which has sensor data output generated by the sensors at the edge nodes.

[0021] The central data store 184 may store one or more master ML models (as well as other ML models for different types of machines), parameter information, and / or other metadata.

[0022] In some embodiments, the central data storage device may store ML models for specific types of machines. For example, the central data storage device may store multiple different ML models for different types of machines. In this case, the ML model may be selected by the central server 182 based on the machine being monitored at the edge node. For example, a first machine learning model may be selected for a fan (or a specific type or model of fan) monitored as machine 102A, a second machine learning model may be selected for a pump (or a specific type or model of pump) monitored as machine 102B, and a third local machine learning model may be selected for another pump, and so on.

[0023] The cloud platform 180 may be implemented as a public cloud platform (e.g., Amazon Web Services, Azure, etc.) and / or via a private cloud (e.g., a cloud platform operated exclusively for a single organization). The cloud platform may be coupled to the first factory 112A and the second factory 112B via a network 150B (e.g., the Internet and / or any other type of communication medium). Figure 1A In the example of FIG. 1 , edge networks 150A and 150B are depicted. The edge network may correspond to an access network, such as a wireless local area network (WLAN), a WiFi network, a 5G local private cellular network, and / or other types of networks (including ultra-reliable, low-latency edge networks). In this example, the edge node at the factory is thereby coupled to the edge network. Although Figure 1A A cloud platform 180 is depicted, but other types of processor platforms may be used.

[0024] exist Figure 1A In the example of , the first plant 112A may include one or more machines, such as machine 102A. Machine 102A may correspond to any type of machine, such as a pump, an actuator, a drill, a fan, a generator, and / or any other type of machine that may be monitored by a sensor.

[0025] The machine 102A may be monitored by one or more sensors, such as sensor 104A. For example, sensor 104A may include a sensor that measures, derives, and / or collects data about the machine, and the sensor data may be used by the local ML model 106A (e.g., for training (during a training phase) and / or predicting (during an operation phase) when maintenance should be performed on the machine 102A monitored by sensor 104A). Sensor 104A may include a sound sensor (e.g., such as an acoustic sensor, a microphone, etc.), a temperature sensor, a pressure sensor, a transducer, and / or any other type of sensor. The sensor may be a wired sensor coupled to a computing device (such as a training node 105A hosting the local machine learning model 106A). Alternatively or additionally, the sensor may be a wireless sensor (e.g., an IoT sensor) that is wirelessly coupled to the computing device.

[0026] In some embodiments, the training node 105A can receive sensor data, such as sound data, from the sensor 104A, save the sensor data to the data storage device 108A, train the local machine learning model 106A with the sensor data, predict anomalies with the local ML model 106A, and communicate with the factory server 110A (e.g., provide parameter information, queries about the availability of updated ML models, anomaly detection information, etc.). In some embodiments, the training node 105A hosts at least the machine learning model 106A. The training node 105A can also track whether sensor data exists (e.g., audio sensor data from the sensor 104A can be used to train the local ML model 106A or generate an anomaly score indicating an anomaly at the machine 102A). The training node can also track (e.g., query or receive notification about) the availability of (an updated machine learning model at) the central server.

[0027] In some embodiments, sensor 104A includes an audio sensor, such as a microphone, that captures sounds from one or more machines at a location such as factory 112A. In addition, sensor 104A may include multiple audio sensors, such as an acoustic array. In a location such as a factory with many different sound sources, an acoustic array may enable sound localization of the source location of the received sound, so that the direction and distance from the sensor to the source location of the sound may be determined. For example, an acoustic array may locate sounds originating from the location of machine 102A. This localization may allow sounds from machine 102A to be processed, while sounds (or sensor data) from other locations in the factory are filtered out or ignored. Sound localization may be performed using time division of arrival (TDOA). TDOA may be used to filter sounds and triangulate the location of the sound source, although other techniques may also be used.

[0028] The factory server 110A may be configured to receive sensor data from the sensor 104A, coordinate local training of the local ML model 106A with the training node 105A, and / or communicate with the central server 182 (e.g., receive the ML model, provide parameter information of the ML model, receive an updated ML model, etc.). The factory server may send parameter information (e.g., weights, etc.) of the local ML model to the central server, and / or query the central server for an updated ML model. In addition, the factory server may receive anomaly scores and / or anomaly alerts, and may forward the anomaly scores and / or alerts to other computing devices so that the anomaly scores / alerts may be presented at a user interface to indicate an anomaly at a given machine (e.g., maintenance is required). For example, the local ML model 106A (when being trained and / or during training) may issue an anomaly alert indicating (based on the sensor data from the sensor 104A) that the machine 102A may require maintenance. The alert may be presented on a user interface of a computing device coupled to the factory server 110A or the cloud platform 180. The alarm can also be used as a label for the sensor data, so that the sensor data associated with the time of the abnormal alarm can be used to further train the local ML model 106A. In some cases, the user interface presenting the abnormal alarm can request confirmation that the abnormal alarm is indeed accurate or that the abnormal alarm is wrong (e.g., a user at the user interface selects a user interface element to confirm or reject the abnormal alarm of the machine). This confirmation can also be provided with a label to further train the local ML model. Although the training node 105A and the factory server 110A are described as independent devices, the training node 105A can be implemented as part of the factory server 110A.

[0029] The data storage device 108A may be used to store sensor data (which may include sound data collected by the sensor 104A), local ML models, and other metadata.

[0030] In some embodiments, the factory 112A may include a plurality of machines 102A, etc., that are monitored. In this case, each machine, such as machine 102A, may have one or more sensors 104A, such as sound sensors and / or other types of sensors. These sound sensors, such as microphones, may receive and record sounds (or acoustic sensor data) for each machine. In addition, each machine may be assigned a machine learning model that is trained to determine when the machine should be maintained based on sensor data (such as sound sensor data) received by the machine learning model.

[0031] Figure 1AAlso depicted is a factory 112B, which may also include machines 102B-C being monitored by sensors 104B-C to enable training of corresponding ML models 106B-C at training nodes 105B-C. The data storage device 108B and the factory server 110B may operate in the same or similar manner as described above with respect to the data storage device 108A and the factory server 110A.

[0032] Figure 1B A system 199 is depicted according to some example embodiments. System 199 is similar in some respects to system 100, but shows some additional features. For example, Figure 1B Scorers (labeled "Anomaly Scorers") 122A-B, model managers 114A-B, aggregator 190, dashboard service 195A, dashboard UI 195B, dashboard repository 195C, and model service 192 are depicted.

[0033] The scorers 122A-B (labeled as anomaly scorers) each calculate an anomaly score. For example, the machine learning model 106A can be trained based on sensor data. The sensor data can be collected by the sensor 104A and then stored in the local data storage device 108A with a time stamp. During training, the sensor data can include one or more labels, such as a label indicating an anomaly (e.g., a machine failure or other machine problem that can indicate a failure or a precursor to a failure) and the time (e.g., a timestamp) when the anomaly occurred. The labeled timestamp data can then be used as reference data to train the local ML model 106A.

[0034] For example, the scorer 122A may also be notified when new sensor data (e.g., for training or detecting anomalies) is available. The scorer may also be notified when the ML model is updated at the central server. The scorer may also score sensor data (e.g., audio data) using local ML models and aggregated ML models. The scorer may send a heartbeat message (e.g., indicating an anomaly score for the machine) to the central server 182 or other device so that the heartbeat message can be processed (e.g., generating an alert for a user interface).

[0035] The factory server 110A may include an audio retriever that provides a representation state transfer (REST) ​​endpoint for receiving a request for audio sensor data (or other types of sensor data). The request may be received from a user interface or other device. The audio retrieval may retrieve sensor data (e.g., an audio clip) from the data storage device 108A in response to the request and respond to the request with sensor data (such as, audio data). The factory server 110A may include an audio feeder that is configured to notify other components of the system 199 (or 100), such as a scorer, trainer, etc., that there is new sensor data (e.g., an audio clip) available for processing.

[0036] exist Figure 1B In the example of , the factory server 110A includes a model manager 114A. The model manager can be notified of a new local machine learning model available at a training node; send 124 parameter information (e.g., connection weights) of the new local machine learning model to the model aggregator 190; receive 126 a new aggregated machine learning model from the model aggregator; notify the scorer of the new local machine learning model and / or the aggregated machine learning model for scoring; and / or notify the training node of the local machine learning model to stop training (off).

[0037] When trained, the local machine learning model 106A may receive sensor data from the sensor 104A as input and output an indication of whether an anomaly exists. The scorer 122A may forward the anomaly indication (obtained from the local machine learning model) in a message 120. The message 120 may be in the form of a heartbeat message 120 indicating an identification of the machine (e.g., identification of the machine 102A), a timestamp (e.g., date and / or time), and / or an indication of the anomaly (e.g., which may indicate the type of anomaly). For example, the local machine learning model 106A may receive sound data (collected by the sensor 104A) and output an indication that the machine 102A has fan blades that require maintenance immediately or within the next 24 hours. The scorer 122A may forward the indication in a heartbeat message 120 to the central server 182, and specifically, to the user interface service 195A that generated the alert, for presentation on a user interface 195B (labeled as a dashboard UI) that is being viewed by an end user 197 assigned to (or responsible for) the machine 102A. Dashboard repository 195C provides a repository of user interface elements for use by user interface 195B.

[0038] In some embodiments, the heartbeat message 120 sent to the aggregator 190 includes one or more of: an anomaly score from the local ML model; an anomaly score from the aggregated ML model; an identification of the version of the local ML model and / or the aggregated ML model; a sensor data identifier associated with the anomaly (e.g., the identifier can be used to search for sensor data at the data storage device 108A); and a timestamp. In some embodiments, the machine learning model generates and provides an anomaly score as an output. The machine learning model generates (based on the sensor data provided as input to the machine learning model) an anomaly score. The anomaly score represents a likelihood, probability, or other indication of whether the machine is in an abnormal (e.g., faulty or pre-faulty) state.

[0039] exist Figure 1B In the example of , machine 102A is being monitored by a sensor (such as sensor 104A) that is configured to monitor at least sound. For example, sensor 104A can generate a signal representing the sound emitted by machine 102A. This sensor data (in this example, sound sensor data, or more simply, sound data) can be provided to training node 105A. The training node can process the sound data from sensor 104A and machine 102A to train a local machine learning model 106A. This training can be continuous, or occur irregularly. The result of this process will create / update a local model in the edge container.

[0040] exist Figure 1B In the example of , the model manager 192 (labeled “aggregate model service”) stores the ML models, tracks updates to the ML models, provides updated ML models to edge nodes, and stores the ML models in the central data storage device 184.

[0041] exist Figure 1BIn the example of , the system 199 includes an aggregator 190. The aggregator can provide aggregation of parameter information provided by edge nodes. For example, the aggregator can receive (as shown in 124) parameter information from an edge node (e.g., model manager 114A) in which one or more local machine learning models monitor one or more specific machines. As described above, each of the local machine learning models can be assigned to monitor a specific type of machine (e.g., a pump, a specific type of pump, or a specific model of pump). When the aggregator 190 receives parameter information from the edge node, the aggregator can combine the parameter information. For example, in the case of ML model weights, the aggregator can combine the weights by averaging the weights and then adding the averaged weights to update the master ML model 186. This updated master ML model can also be referred to as an aggregated ML model. The aggregator 190 can send the aggregated ML model to the model manager 192, the centralized data storage device 184, and / or the edge node (as shown in 126). The aggregator 190 can also provide metadata of the aggregated ML model (e.g., version information of the aggregated model, etc.) to the UI dashboard service 195A.

[0042] Aggregator 190 can also notify edge nodes when the master model is updated (by the aggregate model service) to include updated or aggregated weights. In other words, when the aggregate model service 192 generates an updated aggregated model, the parameter information of the updated model is published as an update to the edge nodes, so the edge nodes can update their local ML models.

[0043] As described above, system 199 may include user interface service (labeled dashboard service) 195A. User interface service processes actions and / or alerts that result in display on a user interface, such as a dashboard presented on a computing device where an end user, such as a reliability or maintenance engineer, can view the status of a machine. User interface service 195A may generate alerts, such as visual, audio, and / or tactile alerts, so that the end user is alerted that the machine requires maintenance. The alerts may be communicated to the end user via the user interface.

[0044] As described above, user interface service 195A may provide a user interface, such as a dashboard at 195B. For example, the dashboard may also include an alert about an anomaly at machine 102A. The dashboard may also include audio associated with the anomaly at machine 102A. The dashboard may also provide a graph of the machine status based on the heartbeat message. The graph may include an X-axis representing time and a Y-axis representing an anomaly score. The higher the anomaly score value, the greater the probability that the machine is anomaly.

[0045] The dashboard may provide a representational state (REST) ​​endpoint for reading device master data and configuration settings from a data store, receiving heartbeat messages (e.g., from an anomaly scorer), and / or receiving versions / metadata of newly built aggregated machine learning models from aggregator 190. Dashboard 195A may also serve as (e.g., cause to be presented) an end-user user interface (e.g., to present device master data, present visualizations of trends for anomaly scores for local and aggregated machine learning models from heartbeat messages, retrieve audio clips from edge nodes, and / or present visualizations).

[0046] In some embodiments, as described above, sensor 104A may collect sound from a machine. The data may be recorded and stored in a data storage device, such as data storage device 108A. In addition, the sound data may be tagged with a timestamp indicating when a fault occurred. For example, a sensor may record a time-varying sound from a pump. When a fault occurs at a given time, the sound data may be tagged with a timestamp of the given time. The tagged data may be used to train a machine learning model. In some example embodiments, sensor 104A may include a plurality of sensors, such as an array of sound sensors configured to locate sound.

[0047] Figure 2A An example of an array of microphones 202A-H is depicted. The array enables sound localization. For example, sound recordings can be collected from microphone array 200 (e.g., microphones 202A-H). Figure 2A In the example of , microphones of the same type are used and are spaced equidistantly from adjacent microphones, such as Figure 2A As shown. Figure 2A A particular number of microphones is described, but other numbers of microphones may be used.

[0048] Figure 2B Depicted are the distances and directions from microphone array 200 to four different types of machines 204A-D. Microphones 202A-H may (e.g., continuously and / or intermittently) record sounds in a noisy factory environment, which are segmented into given lengths (e.g., 10 seconds) (so each segment includes 8 separate channels for each segment).

[0049] Figure 2C Some of the microphones 220A-E of the array 200 are depicted to illustrate the sound localization provided by multiple microphones. The time difference of arrival (TDOA) from each of the microphones can be used to determine the direction to and distance to a sound source.

[0050] Sound collected from one or more sensors (e.g., microphones 220A-H) can be stored in a digital format including a timestamp to enable association with machine failures. In addition, the sound can also be processed in other ways. For example, a transform such as a Fourier transform can be used to convert the time domain sound signal to a frequency domain signal, as shown in 220A-B. For example, when a failure occurs at a given time, the frequency domain and / or time domain sound data can be marked with the time of the failure at a given machine (e.g., a timestamp) to train a machine learning model.

[0051] Figure 3 An example ML model 300 is depicted in accordance with some example embodiments. The ML model 300 includes a convolutional neural network (CNN), although other types of ML models (e.g., generative adversarial networks) may also be used. The ML model includes a first max pooling layer 302, followed by a first convolutional layer 304, followed by a second max pooling layer 306, followed by a second convolutional layer 308, followed by a third max pooling layer 310, followed by a third convolutional layer 312, followed by a fourth max pooling layer 314, and a dense layer 316. An input 350A of the ML model 300 corresponds to sensor data used as features. An output 350B of the ML model 300 corresponds to whether an anomaly is present at the machine.

[0052] For example, during the training phase of the ML model 300, sensor data is provided as input at 350A. The sensor data may include sensor data associated with a machine that is operating normally and sensor data associated with a machine that has anomalies. In addition, the sensor data may have time information (e.g., a timestamp). Figure 3In the example of , the sensor data corresponds to time-domain sound information captured by a sensor 104A (e.g., a microphone) that monitors the sound from a machine 102A (a fan in this example). For example, in the case of a fan, there are different types of industrial fans, so training is performed for a specific subset of fans (and / or a specific model of fan). Sensor 104A collects sound (e.g., audio) data that changes over time, and during this time the machine operates normally (e.g., without anomalies) and operates abnormally (e.g., with anomalies). The sensor data includes examples of audio data of the fan in a normal state and examples of audio data in an abnormal state. The normal and abnormal sound sensor data are then presented as input at 350A (along with labels indicating an abnormal state and / or a normal state). For example, 1016 normal audio files and 359 abnormal audio files can be looped at input 350A until the ML model learns (e.g., configures its weights, etc.) to provide the correct output 350B (e.g., when the audio file is normal, it is classified as normal at the output, and when the audio file corresponds to an abnormal state, the output is classified as abnormal). Output 350B may include a value (eg, a probability or likelihood that input data 350A is an anomaly).

[0053] exist Figure 3 In the example of , the audio sensor data is pre-processed before being provided at input 350A. For example, the audio data can be converted to a frequency domain representation before being provided at input 350A. To illustrate, the audio file can be windowed, and a fast Fourier transform is performed on each window, and the y-axis is converted to a mel scale so that each window corresponds to a frequency spectrum (x-axis) and a mel scale (y-axis). Figure 3 A first spectrogram 360A for audio sensor data of a machine operating normally and a second spectrogram 360B for audio sensor data of a machine operating abnormally are depicted. As described above, these spectrograms 360A-B are provided at input 350A until the CNN converges its weights (e.g., internal parameters) to correctly classify the training data provided at the input.

[0054] After training the ML model, the factory server 110A can forward parameter information of the ML model to the central server 182. And after training the ML model 300, the ML model 300 can be used as a local ML model to classify the sensor data 104A from the machine 102A to detect anomalies. In an example embodiment, the CNN 300 can obtain an accuracy of greater than about 98.6% in detecting anomalies in sensor data (such as audio sensor data).

[0055] Figure 4An example of a user interface 400 is depicted in accordance with some example embodiments. The user interface 400 may be presented to an end user (e.g., an engineer) to assess the health of the machine 104A as well as other machines. If the local ML model 106A detects an anomaly, an alert 402 (in this example, "!", although other types of alerts may also be generated) will be indicated on the user interface.

[0056] For example, an edge node or device, such as node 112A, can determine an anomaly score. The anomaly score can be provided to central server 182 via a heartbeat message. Based on the anomaly score, user interface 400 can highlight or indicate whether a machine, such as machine 102A in this example, is in an abnormal state. In addition, selecting machine 102A via user interface 400 can provide additional information about the machine and / or the anomaly. For example, selecting machine 102A at user interface 400 can present an audio view of machine 102A. Audio view 410 presents anomaly scores for machine 102A at regular intervals.

[0057] In the example of audio view 410, a local score 414 and an aggregate score 412 are plotted. The local score 414 represents a score calculated relative to a local ML model that is built based only on locally available sensor data (such as audio data from sensor 104A). The aggregate score 412 represents a score calculated relative to an aggregated ML model that is built by a central server in a federated manner based on learning across multiple edge nodes. Over time, the aggregated ML model performs better than the local ML model because the aggregated ML model is built on a larger data set from multiple devices. In this example, the edge node is able to operate using a local machine learning model (which is trained using only local sensor data) and an aggregated machine learning model (which is generated based on federated learning and aggregation provided by the central server 182).

[0058] The audio view 410 displays local anomaly scores 414 (y-axis) generated over time (x-axis), and aggregate anomaly scores 412 (y-axis) generated over time (x-axis). Figure 4 In the example of FIG. 4 , a triangular element (e.g., 415) represents an anomaly alert. An anomaly alert indicates when an anomaly score exceeds a threshold score 418 and may therefore be associated with an abnormal event at the machine being monitored. In some embodiments, element 415 is selectable to obtain additional information of the anomaly (e.g., anomaly type, time, machine location, and sensor data associated with the anomaly, such as audio data).

[0059] Figure 5A An example of a process 500 is depicted in accordance with some example embodiments.

[0060] At 502, an edge node may receive a local ML model. For example, a computing device at a first location may receive an ML model to be used locally at the first location from a central server. For example, the central server 182 may provide a copy of the master ML model 186 to each of the plurality of edge nodes 112A-C, which serves as a local ML model 106A-C at each of the edge nodes. The local ML model may include a pre-configured default ML model (e.g., an ML model that has been at least partially trained); an initialized local ML model (e.g., an ML model that has not yet been trained, so the ML model may have random or initialized settings as parameters); or an updated or aggregated ML model (which is a copy of the master ML model).

[0061] At 504, the edge node may receive audio data from an audio sensor monitoring at least one machine. For example, machine 102A may be monitored by a sensor 104A, such as an audio sensor (e.g., a microphone), to monitor the state of the machine and detect an abnormal state of machine 102A, so that when an abnormal state occurs (e.g., a failure or obstruction failure), maintenance can be performed on machine 102A. The audio sensor data may be preprocessed to determine the location of the sound. For example, the audio array may determine the source location of the sound so that only the sound from machine 102A is processed to detect the anomaly (while other machines and noise may be filtered out). Alternatively or additionally, the audio sensor data may be preprocessed to transform the audio data from the time domain to the frequency domain using, for example, a Fourier transform (e.g., a fast Fourier transform or other type of transform). In addition, the audio data is data collected locally at an edge node such as a local to the factory 112A, and the audio data may not be shared with other end users and / or the central server 182 during ML model training.

[0062] At 506, the edge node may train a local ML model based on the audio sensor data received at 504. For example, sensor 102 may generate audio sensor data from monitored machine 102A over time. The audio sensor data may include a timestamp and one or more tags indicating whether the audio sensor data at a given timestamp is abnormal. The audio sensor data (which may be pre-processed as described above and / or filtered for the location of the machine using an audio array) may be looped through ML model 106A (see also above regarding Figure 3 ) until the ML model learns to provide correct outputs, such as anomaly scores or other types of outputs that indicate the probability or likelihood of whether the input sensor data corresponds to an anomaly.

[0063] At 508, the edge node may provide parameter information of the local ML model to the central server that is performing federated learning across multiple edge nodes. For example, the computing device at the plant 112A may provide parameter information of the local ML model to the central server 182 to allow the central server to learn in a federated manner across multiple edge node ML models. As described above, the central server may aggregate the parameter information to generate an updated ML model (also referred to as an aggregated ML model). This updated ML model may be shared with the edge nodes.

[0064] At 510, the edge node may receive an updated ML model, wherein the updated ML model includes aggregated learning provided by the central server. The updated ML model may then be used as a local ML model at the edge node (such as at a computing device at the plant 112A).

[0065] In some embodiments, during a training phase of the local ML model, processes 502-510 repeat at 512 (e.g., until the local ML model converges to within a threshold level of accuracy to provide anomaly detection). In some embodiments, the edge node can use the local ML model during an operational phase to detect an abnormal state of the machine at 514. For example, the local ML model can be used to generate an anomaly score (e.g., an indication of the likelihood 350B that the audio data indicates that the machine is in an abnormal state) and report the score to other computing devices as a heartbeat message, an alarm, etc.

[0066] At 520, the local ML model can generate an anomaly score as sensor data, such as audio data generated by sensor 104A. In response to the anomaly score exceeding a threshold, an alert can be generated, which can be sent to other computing devices such as central server 182, presented on a user interface, etc. The threshold can be predetermined, configured by the edge node, configured by the central server, configured by the end user, and / or learned by the ML model.

[0067] In process 500, the audio data used in the training phase 502-510 may be the same as or different from the audio data used during the operation phase 514-522. During the training phase, the audio data may be training audio data that includes a label indicating an anomaly (e.g., a first audio sample may have a label indicating an anomaly at the machine, and a second audio sample may indicate that the state is not abnormal) to enable machine learning model training. During the operation phase, the machine learning model is trained, so the audio data provided to the machine learning model may not be labeled (because detecting anomalies is the task of the trained ML model). However, the operation phase audio data may also be used for training of the machine learning model. For example, if the machine learning model indicates an anomaly, and the anomaly is proven to be true or confirmed, the edge node may use the operation phase audio data to subsequently train the machine learning model.

[0068] Figure 5B An example of a process 599 is depicted in accordance with some example embodiments.

[0069] At 552, the central server may provide the local ML model to one or more edge nodes. For example, the central server 182 may provide the local ML model to each of the one or more edge nodes 112A-C. Figure 1A , the central server can provide a copy of the master ML model (which can be an initial version of the ML model, a predefined version of the ML model, or an updated / aggregated ML model).

[0070] At 554, the central server may receive parameter information of the local ML model to enable federated learning. For example, the central server 182 may receive the local ML model parameter information at 124 (e.g., Figure 1B ) receives parameter information, such as weights, from edge nodes 112A and 112B.

[0071] At 556, the central server may aggregate the parameter information to generate an updated ML model (also referred to as an aggregated ML model). The central server may aggregate the parameter information from the edge nodes by combining or averaging the parameter information. For example, the weight information provided at 124 may be averaged.

[0072] The central server may provide the updated / aggregated ML model to one or more edge nodes at 558. For example, the updated or aggregated ML model (generated based on the combined or averaged parameter information) may be provided at 126 to edge nodes where the updated or aggregated ML model may be used as a local ML model.

[0073] In some embodiments, processes 552-558 repeat during the training phase of the local ML model (at 560). In some embodiments, the edge node can use the local ML model (or aggregated local ML model) to detect anomaly scores during the operation phase and provide the anomaly scores to the central server at 562. The anomaly scores can be included in the heartbeat message at 120, as described above, or also transmitted to the central server in other formats. If the anomaly score exceeds a threshold, an alert can be generated. Reference Figure 4 In the example of , when the anomaly score exceeds the threshold 418, this can generate an alert, which can be presented on the user interface at 402, or can also be presented or transmitted to the end user in other ways.

[0074] In some implementations, the current subject matter can be configured to be implemented in system 600, such as Figure 6 For example, including Figure 1A , 1B and Figure 3 , any component described or depicted herein may be included in the system 600. To further illustrate, the factory server, training node, sensor, anomaly scorer, model manager, aggregator, dashboard service, model service, etc. may be implemented using a physical system such as the system 600. The system 600 may include a processor 610, a memory 620, a storage device 630, and an input / output device 640. Each of the components 610-640 may be interconnected using a system bus 650. The processor 610 may be configured to process instructions for execution within the system 600. In some embodiments, the processor 610 may be a single-threaded processor. In alternative embodiments, the processor 610 may be a multi-threaded processor or a graphics processor unit (GPU). The processor 610 may also be configured to process instructions stored on the memory 620 or the storage device 630, including receiving or sending information through the input / output device 640. The memory 620 may store information within the system 600. In some embodiments, the memory 620 may be a computer-readable medium. In alternative embodiments, the memory 620 may be a volatile memory unit. In some embodiments, the memory 620 may be a non-volatile memory unit. The storage device 630 can provide mass storage for the system 600. In some embodiments, the storage device 630 may be a computer-readable medium. In alternative embodiments, the storage device 630 may be a floppy disk device, a hard disk device, an optical disk device, a magnetic tape device, a non-volatile solid-state memory, or any other type of storage device. The input / output device 640 may be configured to provide input / output operations to the system 600. In some embodiments, the input / output device 640 may include a keyboard and / or a pointing device. In alternative embodiments, the input / output device 640 may include a display unit for displaying a graphical user interface.

[0075] The systems and methods disclosed herein can be embodied in various forms, including, for example, data processors (such as computers that also include a database), digital electronic circuits, firmware, software, or combinations thereof. In addition, the above-mentioned features and other aspects and principles of the disclosed embodiments can be implemented in various environments. Such environments and related applications can be specially constructed to perform various processes and operations according to the disclosed embodiments, or they can include general-purpose computers or computing platforms that are selectively activated or reconfigured by code to provide necessary functions. The processes disclosed herein are not inherently related to any particular computer, network, architecture, environment, or other device, and can be implemented by appropriate combinations of hardware, software, and / or firmware. For example, various general-purpose machines can be used with programs written according to the teachings of the disclosed embodiments, or it may be more convenient to build special-purpose devices or systems to perform the required methods and techniques.

[0076] The systems and methods disclosed herein may be implemented as a computer program product, i.e., a computer program tangibly embodied in an information carrier (e.g., in a machine-readable storage device or in a propagated signal) for execution by a data processing device (e.g., a programmable processor, a computer, or multiple computers) or for controlling its operation. The computer program may be written in any form of programming language, including compiled or interpreted languages, and it may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. The computer program may be deployed to execute on one computer, or on multiple computers located at one site or distributed across multiple sites and interconnected by a communication network.

[0077] As used herein, the term "user" may refer to any entity, including a person or a computer.

[0078] Although ordinal numbers, such as first, second, etc., can relate to order in some cases; however, as used herein, ordinal numbers do not necessarily imply order. For example, ordinal numbers can be used only to distinguish one item from another item. For example, to distinguish a first event from a second event, but need not imply any time ordering or fixed reference system (such that the first event in one paragraph of the specification can be different from the first event in another paragraph of the specification).

[0079] These computer programs, which may also be referred to as programs, software, software applications, application components or codes, include machine instructions for programmable processors and may be implemented in high-level procedural and / or object-oriented programming languages ​​and / or assembly / machine languages. As used herein, the term "machine-readable medium" refers to any computer program product, device and / or equipment, such as disks, optical disks, memories and programmable logic devices (PLDs), for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as machine-readable signals. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor. A machine-readable medium may store such machine instructions non-temporarily, such as in a non-temporary solid-state memory or magnetic hard drive or any equivalent storage medium. A machine-readable medium may alternatively or additionally store such machine instructions in a temporary manner, such as in a processor cache or other random access memory associated with one or more physical processor cores.

[0080] To provide interaction with a user, the subject matter described herein can be implemented on a computer having a display device, such as a cathode ray tube (CRT) or liquid crystal display (LCD) monitor, for displaying information to a user; and a keyboard and pointing device, such as a mouse or trackball, through which a user can provide input to the computer. Other types of devices can also be used to provide interaction with a user. For example, the feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including but not limited to sound, voice, or tactile input.

[0081] The subject matter described herein may be implemented in a computing system that includes a back-end component (e.g., one or more data servers), or includes a middleware component (e.g., one or more application servers), or includes a front-end component (e.g., one or more client computers with a graphical user interface or a web browser through which a user can interact with an implementation of the subject matter described herein), or any combination of such back-end, middleware, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include, but are not limited to, a local area network ("LAN"), a wide area network ("WAN"), and the Internet.

[0082] A computing system may include clients and servers. Clients and servers are typically, but not exclusively, remote from each other and typically interact through a communication network. The relationship of client and server arises through computer programs running on the respective computers and having a client-server relationship to each other.

[0083] The embodiments set forth in the foregoing description do not represent all embodiments consistent with the subject matter described herein. Instead, they are merely some examples consistent with the relevant aspects of the described subject matter. Although some variations have been described in detail above, other modifications or additions are possible. In particular, in addition to the features and / or variations set forth herein, further features and / or variations may also be provided. For example, the above embodiments may be directed to various combinations and subcombinations of the disclosed features and / or combinations and subcombinations of several further features disclosed above. In addition, the logical flows depicted in the accompanying drawings and / or described herein do not necessarily require the specific order shown or sequential order to achieve the desired results. Other embodiments may be within the scope of the following claims.

Claims

1. A method comprising: Maintaining a master machine learning model at a central server; receiving, at a first edge node, a copy of the master machine learning model to be used locally as a machine learning model provided by the central server, the central server being configured to provide federated learning among a plurality of edge nodes including the first edge node; receiving, at the first edge node, first audio data obtained from at least one audio sensor that monitors at least one machine located at the first edge node; At the first edge node, training the machine learning model, the training being based on the first audio data; In response to the training, the first edge node provides parameter information to the central server, so as to enable joint learning among a plurality of edge nodes including the first edge node; aggregating, by the central server, parameter information received from the edge nodes to update the main machine learning model; In response to the federated learning, receiving, by the first edge node, a copy of the updated master machine learning model provided by the central server; detecting, by the first edge node including a copy of the updated master machine learning model, an abnormal state of the at least one machine, the abnormal state being detected based on second audio data received from the at least one audio sensor; and The first edge node reports the abnormal state to the central server.

2. The method of claim 1, wherein the parameter information comprises configuration information about a machine learning model trained by the first edge node, and wherein the configuration information comprises one or more weights for the trained machine learning model.

3. The method according to claim 1, wherein: The first audio data is pre-processed by filtering the first audio data based at least on arrival time to at least partially remove other audio data associated with other machines.

4. The method according to claim 1, wherein: The first audio data is preprocessed by at least transforming the first audio data into a frequency domain.

5. The method according to claim 1, wherein: The machine learning model includes at least one of a convolutional neural network or a generative adversarial network.

6. The method according to claim 1, wherein: The updated main machine learning model includes an average of parameter information provided by the first edge node and other parameter information provided by other edge nodes among the multiple edge nodes.

7. The method according to claim 6, wherein: The updated master machine learning model is generated without the central server accessing the first audio data at the first edge node or other audio data at the other edge nodes.

8. The method according to claim 1, wherein: The training also includes providing the first audio data as input to the machine learning model, the first audio data including timestamp information and label information indicating whether there is an anomaly.

9. The method according to claim 8, wherein: The first audio data corresponds to training data for training the machine learning model, and wherein the second audio data represents operational data without label information indicating whether an abnormality exists.

10. A system comprising: at least one processor; and at least one memory including program code, which, when executed by the at least one processor, causes operations including: Maintaining a master machine learning model at a central server; receiving, at a first edge node, a copy of the master machine learning model to be used locally as a machine learning model provided by the central server, the central server being configured to provide federated learning among a plurality of edge nodes including the first edge node; receiving, at the first edge node, first audio data obtained from at least one audio sensor that monitors at least one machine located at the first edge node; At the first edge node, training the machine learning model, the training being based on the first audio data; In response to the training, the first edge node provides parameter information to the central server, so as to enable joint learning among a plurality of edge nodes including the first edge node; aggregating, by the central server, parameter information received from the edge nodes to update the main machine learning model; In response to the federated learning, receiving, by the first edge node, a copy of the updated master machine learning model provided by the central server; detecting, by the first edge node including a copy of the updated master machine learning model, an abnormal state of the at least one machine, the abnormal state being detected based on second audio data received from the at least one audio sensor; and The first edge node reports the abnormal state to the central server.

11. The system according to claim 10, wherein: The parameter information includes configuration information about a machine learning model trained by the first edge node, and wherein the configuration information includes one or more weights for the trained machine learning model.

12. The system according to claim 10, wherein: The first audio data is pre-processed by filtering the first audio data based at least on arrival time to at least partially remove other audio data associated with other machines.

13. The system according to claim 10, wherein: The first audio data is preprocessed by at least transforming the first audio data into a frequency domain.

14. The system according to claim 10, wherein: The machine learning model includes at least one of a convolutional neural network or a generative adversarial network.

15. The system according to claim 10, wherein: The updated main machine learning model includes the parameter information provided by the first edge node and an average of other parameter information provided by other edge nodes among the multiple edge nodes.

16. The system of claim 15, wherein: The updated master machine learning model is generated without the central server accessing the first audio data at the first edge node or other audio data at the other edge nodes.

17. The system according to claim 10, wherein: The training also includes providing the first audio data as input to the machine learning model, the first audio data including timestamp information and label information indicating whether there is an anomaly.

18. The system of claim 10, wherein: The first audio data corresponds to training data for training the machine learning model, and wherein the second audio data represents operational data without label information indicating whether an abnormality exists.

19. A non-transitory computer readable storage medium comprising program code, which, when executed by at least one processor, causes operations comprising: Maintaining a master machine learning model at a central server; receiving, at a first edge node, a copy of the master machine learning model to be used locally as a machine learning model provided by the central server, the central server being configured to provide federated learning among a plurality of edge nodes including the first edge node; receiving, at the first edge node, first audio data obtained from at least one audio sensor that monitors at least one machine located at the first edge node; At the first edge node, training the machine learning model, the training being based on the first audio data; In response to the training, the first edge node provides parameter information to the central server, so as to enable joint learning among a plurality of edge nodes including the first edge node; aggregating, by the central server, parameter information received from the edge nodes to update the main machine learning model; In response to the federated learning, receiving, by the first edge node, a copy of the updated master machine learning model provided by the central server; detecting, by the first edge node including a copy of the updated master machine learning model, an abnormal state of the at least one machine, the abnormal state being detected based on second audio data received from the at least one audio sensor; and The first edge node reports the abnormal state to the central server.

20. The non-transitory computer-readable storage medium of claim 19, wherein: The parameter information includes configuration information about a machine learning model trained by the first edge node, and wherein the configuration information includes one or more weights for the trained machine learning model.

Images