Data processing and anomaly detection

Abstract

There is described a computer-implemented method to monitor operational performance of a system based on sensor data associated with the operation of the system. Received sensor data is processed to provide a plurality of input signals for an Artificial Neural Network, ANN. This processing involves transforming the sensor data into a plurality of components that are characteristic of the operation of the system and normalised with respect to activity values within the ANN, each component having a corresponding component value, and for each component, determining a difference value between the normalised component value and a respective feedback value from the ANN and generating an input signal representative of the difference value. By monitoring activity associated with the ANN, anomalous activity indicative of a departure from normal operation of the system is detected.

Description

[0001] DATA PROCESSING AND ANOMALY DETECTION

[0002] Technical Field

[0003] This disclosure relates to processing sensor data related to operation of a system to detect occurrences of anomalous operation of the system.

[0004] Background

[0005] Many systems employ sensors that are used either to control the system, for example as part of a feedback loop, or to monitor the operation of the system or a combination of both controlling and monitoring the system.

[0006] Sensor data from the sensors may indicate anomalous operation of the system. In other words, sensor data may identify when the operation of the system departs from historically normal operation. Such departures may be indicative of a gradual drift in operation, for example due to a gradual degradation of moving components, or may be indicative of a more serious degradation that may, if uncorrected, develop into a catastrophic failure of the system.

[0007] This disclosure describes a novel technique for monitoring sensor data from a system to identify departures from normal operation of the system.

[0008] Summary

[0009] According to a first aspect of the disclosed technology, there is provided a computer-implemented method to analyse, or analyse and identify, or predict, operational performance of a system based on sensor data associated with the operation of the system. Received sensor data is processed to provide a plurality of input signals for an Artificial Neural Network, ANN. This processing involves transforming the sensor data into a plurality of components that are characteristic of the operation of the system and normalised with respect to activity values within the ANN, each component having a corresponding component value, and for each component, determining a difference value between the normalised component value and a respective feedback value from the ANN and generating an input signal representative of the difference value. By monitoring activity associated with the ANN, anomalous activity indicative of a departure from normal operation of the system is detected.

[0010] In some implementations, the sensor data includes time-varying data streams from multiple sensor modalities, for example position sensing, audio sensing, video sensing and temperature sensing. The ANN can take into account interrelationships between sensor readings from different technologies when assessing whether or not there is anomalous activity.

[0011] The sensor data may be received from the system in real time or near real time. In some embodiments, activity signals corresponding to detection of an anomaly are analysed and a corresponding output signal generated. This output signal may be output to the system as a control signal for one or more actuators or sensors within the system such that some form of feedback loop incorporating the system and / or its environment is generated. Alternatively, the output signal may be an alert signal, for example providing at least one of an audible alert and a visual alert.

[0012] In some embodiments, the actuator may comprise an actuated sensor.

[0013] In some embodiments, the one or more actuators include, for example, a motor.

[0014] In some embodiments, the network output is used to modulate or calibrate an actuatable sensor.

[0015] In some embodiments, the feedback from the ANN to the ingestion unit regulates the information received by the ANN in the form of a negative feedback loop which is not connected to an actuator.

[0016] A second aspect of the disclosed technology relates to a computer- implemented method for training a time-varying artificial neural network (ANN) to process a data stream which comprises receiving a data stream from one or more sensors, processing the received data stream to derive at least one stream of components or features, analyzing the stream of components or features as they are ingested into the ANN model, and monitoring internal states adopted by the ANN model responsive to ingesting the stream of components or features.

[0017] In some embodiments, the monitoring generates at least one indication of a capability of the time-varying ANN model to process ingested data. In some embodiments, the method further comprises extracting diagnostic data representing characteristics of internal states adopted by the time-varying model, analyzing characteristics of internal states over time, and determining when the ANN model is trained based on characteristics of transient internal states when non- anomalous data is input.

[0018] In some embodiments, each stream of components or features is derived from a received multi-variate data stream and analyzed using a first multi-variate analysis computer model on ingestion to the ANN model.

[0019] In some embodiments, each stream of components or features is derived by applying a data transform model, such as a Fourier transform model, to the received data stream.

[0020] In some embodiments, the ANN model adopts transient states as it updates connections between source and target nodes, internal weights applied by source nodes, or which nodes act as source or target nodes based on input data features.

[0021] In some embodiments, processing the multi-variate data stream comprises generating multiple streams of features and separately inputting each stream into the ANN model, with each stream potentially processed in parallel.

[0022] In some embodiments, the processing of the received data stream to derive components or features is dependent on characteristics of internal states adopted by the ANN model when processing previously received data.

[0023] In some embodiments, the time-varying ANN comprises a time-varying model architecture.

[0024] In some embodiments, one or more nodes of the ANN are configured to spontaneously generate activity independently from components or features input, in other words ingested, into the ANN.

[0025] In some embodiments, the time-varying activity in the ANN causes it to adopt different states, with the ANN considered trained when activity follows a stable trajectory path.

[0026] In some embodiments, the multi-variate data comprises multi-modal data. Examples of multi-modal data comprise image data, video data, text data, audio data. More specifically, a mix of different types of data may be provided in one or more ingested data streams in some embodiments. The different types of data may represent different physical conditions, physical characteristics and / or physical phenomena affecting a sensed real-world entity. Examples of different types of data include, but are not limited to: speech data, non-speech audio data such as vibrational data, temperature data, ambient light data, humidity data, pressure data, colour data, depth data, black and white image or video data, greyscale image data, colour data, an event-based video data stream.

[0027] In some embodiments, the model is trained in real-time or near real-time as the data stream is fed into the model.

[0028] A third aspect of the disclosed technology relates to a system for training a time-varying artificial neural network (ANN) to process a data stream and comprises receiving a data stream comprising one or more streams of sensed components or features from one or more sensors, analysing the stream of components or features as they are ingested into the ANN model, and monitoring internal states adopted by the ANN model responsive to ingesting the stream of components or features.

[0029] In some embodiments, the system further comprises processing the received data stream to derive or extract the at least one stream of components or features.

[0030] A fourth aspect of the disclosed technology relates to a computer program product configured to perform the method aspect or any embodiments of the method aspect disclosed herein.

[0031] A fifth aspect of the disclosed technology comprises a system for anomaly detection comprises means for receiving a data stream from sensors, processing the data stream as it is ingested into a time-varying artificial neural network (ANN) model, monitoring internal states of the ANN model, extracting diagnostic data, analysing characteristics of internal states, determining when the ANN model is trained, inputting new data streams, comparing internal activity representations, and determining if the new data stream contains anomalous data based on the comparing.

[0032] The system may further comprise means to process the data stream to derive a plurality of component or feature streams from the data stream.

[0033] The means to process the data stream may further comprise means for analysing the derived component or feature streams.

[0034] The system may further include means for adjusting the component or feature stream processing and re-training the ANN model. The ANN model may have a time- varying architecture and nodes capable of spontaneous activity generation, allowing it to adapt to different states and follow stable trajectory paths when trained.

[0035] A sixth aspect of the disclosed technology comprises a system for anomaly detection, comprising: means for receiving a data stream comprising data from one or more sensors; means for processing the received data stream to derive at least one stream of features, each stream of features being ingested along a separate channel into a timevarying artificial neural network (ANN) computer model; means for analysing the at least one stream of features as they are ingested into the time-varying ANN computer model; means for monitoring one or more internal states adopted by the time-varying ANN computer model responsive to ingesting of the at least one stream of features; means for extracting diagnostic data comprising data representing one or more characteristics of each of the one or more internal states adopted by the time-varying ANN computer model; means for analysing, using an ANN state analysis model, one or more characteristics of a plurality of internal states adopted by the time-varying ANN computer model over a first period of time as it processes input comprising the at least one stream of features of the received data stream; means for determining, from one or more characteristics of a plurality of transient internal states adopted by the ANN computer model over a period of time when non-anomalous data is input to the model, when the ANN computer model is trained; means for inputting a new data stream into the trained time-varying ANN computer model; means for determining, in a time-step of the ANN computer model, for each of one or more spatial locations in the ANN computer model, one or more representations of internal activity at the one or more spatial locations in the ANN computer model; means for comparing, for each time-step in which one or more representations of internal activity at one or more spatial locations of the ANN computer model were determined, at least one of: (a) at least one representation of internal activity at a spatial location of the ANN computer model derived from the input new data stream in that time-step with at least one representation of internal activity at that spatial location in the ANN computer model trained on non-anomalous data; and

[0036] (b) at least one spatial location where a representation of internal activity is generated in the ANN computer model in that time-step derived from the input new data stream with at least one spatial location where a representation of internal activity in the ANN computer model trained on non-anomalous data was previously generated; and means for determining the input new data stream being processed in that timestep by the ANN computer model comprises anomalous data based on the comparing.

[0037] The disclosed technology offers several technical advantages:

[0038] 1. Adaptive processing: The time-varying artificial neural network (ANN) can adapt its internal states and processing based on the input data streams, allowing it to handle dynamic and evolving data patterns.

[0039] 2. Multi-stream feature analysis: By deriving multiple streams of features from the input data and ingesting them through separate channels, the system can capture and analyze different aspects of complex, multi -variate data simultaneously.

[0040] 3. Real-time anomaly detection: The continuous monitoring of internal states and comparison with trained models enables real-time or near real-time detection of anomalies in the input data stream.

[0041] 4. Flexible architecture: The time-varying model architecture allows the ANN to adjust its structure and connections based on the input data and generated activity, enhancing its ability to learn and adapt to new patterns.

[0042] 5. Spontaneous activity generation: The ability of nodes to generate spontaneous activity independently from input features may help in exploring new patterns and improving the model's robustness.

[0043] 6. Parallel processing: The system's ability to process multiple streams of features in parallel can improve computational efficiency and reduce processing time for complex data sets.

[0044] 7. Self-assessment: The monitoring of internal states provides a means for the system to assess its own capability to process ingested data, potentially allowing for self-optimization.

[0045] 8. Continuous learning: The system can be trained in real-time or near realtime as new data is fed into the model, allowing it to continuously improve and adapt to changing data patterns.

[0046] 9. Multi-modal data handling: The ability to process multi-variate and multimodal data makes the system versatile and applicable to a wide range of complex real-world scenarios.

[0047] 10. Diagnostic capabilities: The extraction and analysis of diagnostic data from internal states provide insights into the model's behavior and training progress, aiding in model optimization and troubleshooting.

[0048] 11. Robustness: Sensory drifts may be due to environmental factors, for example, outside weather may affect a temperature sensor that is monitoring physical entity such as the heat given off by a motor or when there is wear and tear in the mechanics of a machine. These are not anomalies but changing environmental conditions which the model is able to adapt to given its continuous learning capability and flexible architecture.

[0049] Brief Description of the Drawings

[0050] Various implementations will now be described, by way of example, with reference to the accompanying diagrams in which:

[0051] Figure 1 is a block diagram schematically showing the main components of a system monitor;

[0052] Figure 2 is a block diagram schematically showing functional components of a an implementation of the system monitor of Figure 1;

[0053] Figure 3 is a flow chart schematically showing operations performed by the system monitor of Figure 1;

[0054] Figure 4 is a flow chart schematically showing in more detail operations performed by the system monitor of Figure 1; and

[0055] Figure 5 shows representations of activity within a artificial neural network forming part of the system monitor of Figure 1;

[0056] Figure 6 schematically shows a first selection of input channels during training and after training of the artificial neural network; Figure 7 schematically shows a second selection of input channels during training and after training of the artificial neural network;

[0057] Figure 8 shows the variation of monitored activity following the occurrence of anomalies;

[0058] Figure 9 is a block diagram schematically showing the main components of a system monitor in which an ingestion module performs spectral decomposition;

[0059] Figure 10 schematically shows the main components of an artificial neural network forming part of the system monitor of Figure 1;

[0060] Figure 11 schematically shows the main components of a node forming part of the artificial neural network illustrated in Figure 10;

[0061] Figure 12 is a block diagram schematically showing a system in which the system monitor is external to sensory data sources and an external system;

[0062] Figure 13 is a block diagram schematically showing a system in which the system monitor is internal to sensory data sources and an external system;

[0063] Figure 14 is a block diagram of another example apparatus for processing data, for example, for implementing an embodiment of a method according to the disclosed technology, showing the relationships between various components including processing circuitry, memory, and communication units according to some embodiments of the disclosed technology; and

[0064] Figure 15 is a block diagram of an example distributed system for processing data, for example, for implementing an embodiment of a method according to the disclosed technology, the figure indicating a connection between ingestion apparatus and a remote processing system through a transmit / receive communications unit according to some embodiments of the disclosed technology; and

[0065] FIG. 16 is a block diagram of an example system apparatus for processing multi-modal sensory data using an embodiment of a method according to the disclosed technology, which illustrates the integration of a processing apparatus in a system with a plurality of sensors and optionally actuatable components according to some embodiments of the disclosed technology. Detailed Description

[0066] System Overview

[0067] Figure 1 shows a schematic block diagram of a system monitor 100 that processes sensor data from an external system (not shown in Figure 1) to provide information regarding the operation of the external system. As shown, the system controller 100 includes an input module 102, and ingestion module 104, an artificial neural network (ANN) model 106 and an ANN model activity monitor 110. In operation, during an initial phase the system monitor 100 processes the sensor data from the external system to configure the ANN model to correspond to a normal mode of operation. Following the initial phase, the ANN model activity monitor 110 is able to identify changes in the activity in the ANN model 106 that are indicative of a departure from the normal operation of the external system, which will hereafter be referred to as an anomalous event, and output anomalous event information.

[0068] The anomalous event information can have a variety of uses. For example:

[0069] • the anomalous event information can be used to generate a control signal for the external system to alter the operation of the external system; or

[0070] • the anomalous event information can be presented to a user or a user system to provide information regarding the state of operation of the external system.

[0071] When the anomalous event information is being used to generate a control signal for the external system, the control signal may be input to one or more actuators within the external system to alter the operation of the external system. The altering of the operation of the external system may involve varying operating parameters of the external system, or in a more extreme case terminating operation of the external system, for example to enable maintenance work to be carried out on the system when the anomalous event information is associated with the possibility of a catastrophic breakdown of the external system.

[0072] When the anomalous event information is presented to a user or a user system, the anomalous event information can be used to provide an indication of a possible malfunction of the external system. The sensor data may either be received in real time from an operational external system, or may be historical sensor data that has been stored as a data file. When the sensor data is being received by the system monitor from the external system in real time, the anomalous event information may trigger an alert signal being provided to a user, e.g. an audible alarm signal or a visible alarm signal, indicating possible malfunction of the external system. When the sensor data is historical sensor data, the anomalous event information may be used to investigate a historical malfunction of the external system. A machine operator, can also interpret the alarm signal as a warning, for example of a non-critical anomaly and if no action is deemed required allow the system monitor to adapt its understanding of normal operation to take the non-critical anomaly into account. An example of a non- critical anomaly is sensor drift.

[0073] The input module 102 receives sensor data for the external system, either in real time from the external system or from a data storage device. This sensor data is typically in the form of one or more time-varying data streams. Each stream of data 103 can be multi -variate. In some examples, the input module 102 receives a plurality of streams of data 103 which may include data streams derived from different types or modalities of sensors, such as thermal sensors, acoustic sensors, video sensors, position sensors or any other modality of sensor, including sensors such as accelerometers, gyroscopes, magnetometers.

[0074] The input module forwards the sensor data 103 to an ingestion module 104, which processes the sensor data 103 to generate a plurality of transformed signals 105 which are configured for provision to the ANN model 106. More particularly, the ingestion module 104:

[0075] • processes the streams of data 103 to transform the streams of data 103 into streams of features that characterise the operation of the external system,

[0076] • normalises the streams of features to be suitable for processing by the ANN model 106, and

[0077] • modifies the streams of features in dependence upon feedback 107 from the ANN model 106.

[0078] As will be described in more detail hereafter, the ANN model 106 learns to adapt the feedback that is used to modify the stream of features such that the values input to the ANN model 106 are kept low in the course of normal operation. Accordingly, the stream of features input to the ANN model 106 can be considered to define a multi- dimensional problem space, with the ANN model 106 being configured to maintain the location of the input to the ANN model 106 close to an origin point within the problem space. The features are determined such that different operational problems of the external system are mapped to different positions within the problem space.

[0079] As an example, one of the streams of data 103 may include multiple frequency components, with different operational problems of the external system mapping to characteristically different changes in the multiple frequency components. In such an example, the stream of data is transformed into multiple feature streams using a spectral decomposition technique, with each feature stream corresponding to a respective different frequency component. This spectral decomposition may utilise Fourier analysis, for example using a Fast Fourier Transform or any similar technique. More generally, the transformation of the data streams into feature streams is determined by analyzing the data streams from the external system to identify attributes which very during operation of the external system. This may be done automatically or by using a combination of automatic processing and manual input.

[0080] Each of the plurality of transformed signals 105 is received by the ANN model 106 via its own separate data channel, such that the ANN model 106 can process the received steams of features both separately and in combination. In other words, each stream of features may influence the internal states of the ANN model 106, and differences, correlations, anti-correlations or other relationships between two or more respective streams of features can also influence the internal state of the ANN model 106. The ANN model 106 has a plurality of nodes that are interconnected by edges or links. As the transformed signals 105 flow into input nodes the transformed signals 105 modify activity output by those nodes to the other nodes of the ANN model 106, which then output activity to other nodes within the ANN model 106 based on the weighted input activity. Output nodes of the ANN model 106 provide the feedback values 107 to the ingestion module 104, with the feedback values being based on the activity propagating throughout the ANN model 106. The internal state of the ANN model 106, in terms of the structure of the interconnections between nodes and the activity propagating along the interconnections, is therefore determinative of the performance of the ANN model 106. The model activity monitor 110 monitors activity signals within the ANN model 106 to determine whether the external system is operating normally or whether there is an anomalous event. It is not necessary that every activity signal in the ANN 106 is monitored. In one example, the monitored activity signals correspond to the activity signals output by a subset of the nodes within the ANN model 106. In another example, the monitored activity signals may correspond to the transformed signals

[0081] 105 input to the ANN model 106.. Real-time, or near real-time, monitoring of the internal state of the ANN model 106, by the model activity monitor 110, can provide valuable insights about the development and performance of the ANN model 106 and if the stream of information received at an input module 102 is representative of an anomaly. The model activity monitor 110 may be able to detect an anomaly even where the presence of anomalous behaviour is not clear from the raw data provided in the stream of information.

[0082] Where the system controller 100 receives sensor information from an external apparatus and provides output information to control the external apparatus, the system controller may be designed to minimise a difference between the received sensor information and predetermined set-point information, corresponding to a desired state of operation of the external apparatus. To achieve this, the ANN model

[0083] 106 may seek to minimize the activity within its nodes. In an ideal case, the sensor data may remain consistent over time, such that the ANN model 106 can develop into a state of minimum activity, that maintains the desired state of operation of the external apparatus. However, in other scenarios, the sensor data may deviate over time. Such deviations may be characteristic of changes in the external apparatus, such as due to gradual wear and tear, or may arise from gradual drift in the performance of the sensors that is not representative of changes in the external apparatus. The ANN model 106 can gradually modify its activity levels over time to account for these gradual changes in sensor data. In other scenarios, an anomalous change in the behaviour of the external apparatus may occur that does not correspond to expected wear and tear or sensor drift. In some examples, such anomalous changes may be extremely difficult to detect by conventional means as such anomalies may only be apparent based on subtle changes in the relationships between multiple streams of features, although such changes may be readily detected by monitoring the internal states of the ANN model 106.

[0084] Figure 2 shows a schematic block diagram of an example implementation of a system controller of Figure 1. Features of Figure 2 that correspond to those of Figure 1 have been given like reference numerals and will not necessarily be described further here.

[0085] As shown in Figure 2, the node activity data 108 that is provided to the model activity model monitor 110 is also provided to the ingestion module 104 as the feedback data. In response to receiving transformed signals 105, that provide streams of features, an ANN model 106 generates node activity data 108 that is representative of the internal state of the ANN model 106. The node activity data 108 is provided to a node activity model monitor 110 and is also provided to an ingestion module 104. Advantageously, the ingestion module 104 can use the node activity data 108 to modify the derivation of the transformed signals 105 from the one or more streams of data 103. For example, the ingestion module 104 may reduce the amplitude of a particular stream of features 105 in response to the node activity data 108 indicating that the particular stream of features 105 is relatively unimportant. Alternatively, one or more of the links between nodes could be removed altogether to remove the influence of certain streams of features that are identified as being less important. In this way, the ingestion module 104 may be re-configured to provide streams of features 105 that concentrate on the most relevant data that the ANN model 106 requires. This may speed up the evolution of the internal state of the ANN model 106 to minimize activity during a normal phase of operation of the external apparatus. Consequently, when an anomaly occurs, the ANN model 106 may be especially sensitive to the deviation from normal sensor data being received at the input module 102 and thereby enable the rapid identification that an anomaly has occurred and, in some examples, may also be able to identify the type of anomaly and how to respond to the anomaly.

[0086] Figure 3 shows a flow chart of a method 300 for operating the system controller 100. Sensor data associated with the operation of the system is received, at 302, and the received sensor data is processed, at 304. This processing involves transforming the sensor data into a plurality of components (or features) that are characteristic of the operation of the system and normalised with respect to values within the ANN. For each component, a difference value Is determined between the normalised component value and a respective feedback value from the ANN, and a plurality of input signals are generated with each input signal being representative of a corresponding difference value.

[0087] The plurality of input signals are provided, at 306, to the ANN model via respective different data channels. The ANN model processes the input signals and over time the internal state of the ANN model varies, whether settling down to a steady state, undergoing a gradual change in response to expected drifts in sensor data over time, or due to an anomalous change in sensor data. These internal states of the ANN model are monitored, at 308, to detect anomalous activity. The internal state of the ANN can be monitored by monitoring at least some of the activity propagating along edges between nodes within the ANN. Following detection of anomalous activity, an output signal is produced that is associated with the anomalous activity. As discussed above, the output signal may involve one or more control signals for supply to respective actuators within the system, or to a user or user system as an alert signal.

[0088] Figure 4 shows a flow chart of a method 400 for operating a system controller. At a first step 402 data representing streams of features is input into a trained ANN model. Here, that the ANN model is trained means that its internal state has achieved a consistent steady state, having ingested a sufficient amount of data representative of an external apparatus being in a normal, non-anomalous, state of operation. At a second step, the streams of features are processed by the ANN model, which determines the internal states of the nodes and links that make up the ANN model. These internal states can be considered to exist at different spatial locations, as the nodes and links can be considered to occupy separate spatial locations.

[0089] At a third step 406, the internal state of the ANN model for a current time step, that may relate to an anomaly, can be compared against internal states of the ANN model for one or more previous time steps that relates to non-anomalous sensor data. The comparison can be based on comparing the activity levels at each different node and or link. At a fourth step 408, the method 400 determines whether the data for the present time step is representative of an anomaly or not. This determination may be based on one or more nodes or links, or at least a threshold number of nodes or links, satisfying a threshold condition that indicates a deviation away from the normal operational state of the ANN model responding to a non-anomalous stream of sensor data. If the streams of features of the present time step are determined to represent anomalous behaviour of the external apparatus, then a suitable control signal can be generated to initiate appropriate action to respond to the anomaly.

[0090] Figure 5 shows charts 500 representing internal activity within an ANN model. Horizontal axes 502 show time, while vertical axes 504, 506, 508 show amplitude. A first signal 510 shows an activity amplitude at a first node or neuron A. A second signal 512 shows an activity amplitude at a second node or neuron B. A third signal 514 shows a combination of the first signal 510 and the second signal 512, which is simply the difference between the first signal 510 and the second signal 512. Most of the third signal 514 shows a low and constant amplitude, however, a portion 516 of the third signal shows a clear deviation away from normal behaviour. This portion 516 reveals an anomaly in the underlying sensor data. While the anomaly is revealed based on a very simple calculation of taking the difference between two node activity levels, this difference may depend on a complex combination of interrelationships between different streams of features based on the underlying sensor data. Detection of this anomaly may therefore have been impractical, or even impossible, in real-time or near real-time, without the methods and systems disclosed herein.

[0091] Figure 6 shows a schematic timeline diagram 600 that depicts different phases of the training and operation of an ANN model as disclosed herein. Time is depicted on the horizontal axis. The timeline diagram 600 shows six separate data channels 105A-F via which separate streams of features can be provided to the ANN model from an ingestion module. All six data channels 105A-F are used to provide streams of features to the ANN model during an initial training phase 612 between a first time tO and a second time tT. By monitoring the internal activity of the ANN model during this time 612 it is possible to determine that a sufficiently stable set of internal states has been achieved, such that the model can be considered adequately trained on data assumed to be anomaly free.

[0092] The timeline diagram 600 shows a second phase 614, after time tT and until subsequent time tl during which a subset of the channels D, E and F are used to provide signals 105D-F to the ANN model. Then, during a third phase 616, between time tl and a subsequent time t2, a different subset of channels A, B and C are used to provide signals 105A-C to the ANN model. This illustrates the flexibility that the present disclosure provides to enable a system controller to adapt to changing conditions. Based on changes in the internal state of the ANN model, which can be fed back to the ingestion module as discussed above in relation to Figure 3, the streams of features provided to the ANN model can be changed in real-time or near real-time. The second phase 614 and the third phase 616 may be relevant to detecting different types of anomalies that may appear at different times, or in different states of operation of the external apparatus being monitored and controlled by a system controller of the present disclosure.

[0093] Figure 7 shows an alternate schematic timeline diagram 700. Again, time is depicted on the horizontal axis. During a first phase 712, extending from time tO until a subsequent time tT, the ANN model, receives streams of features 105D-F on channels D, E and F. This provides sufficient training data, that is free of anomalies, to train the ANN model. After time tT, at time tl, during a second phase 714, further streams of features 105D-F are input into the ANN model using channels D, E and F. Having been sufficiently trained, the ANN model continues to maintain a consistent set of internal states. Thus, the monitoring concludes that no anomaly has occurred during the second phase 714.

[0094] The alternate timeline diagram 700 also shows a third phase 716, which begins after time tl at a later time t2. During the third phase 716 streams of features are provided on channels A, B and C, which were not previously used to train the ANN model. Consequently, there is a risk that monitoring of the internal states of the ANN model during the third phase 716 may provide a false positive identification of an anomaly. In such a scenario, it may be necessary to temporarily suspend identification of anomalies until the ANN model has ingested sufficient data from channels A, B and C to have achieved a steady state, from which anomaly identification can reliably be performed.

[0095] Figure 8 shows a chart 800 illustrating the training of an ANN model and subsequent identification of two different anomalies in series. Time is shown on horizontal axes 802 while an amplitude of activity within the ANN model is shown on a vertical axis 804. The chart 800 shows three time periods: a first training phase 810, a second training phase 812 and a third training phase 814. During the first training phase 810, the ANN model is trained on anomaly free data and a first activity level 820 falls monotonically to a low level as activity in the ANN model diminishes to a low steady state. During the second training phase 814, the ANN model ingests data representative of a first type of anomaly and consequently a second activity level 822 in the ANN model rapidly increases to a peak, thereby enabling the identification of the first type of anomaly as having occurred. As the ANN model continues to receive further data during the second training phase 814, the second activity level 822 diminishes to a low level as the ANN model adapts to the first type of anomaly. During the third training phase 814, the ANN model ingests data representative of a second type of anomaly, different than the first type of anomaly. Consequently, a third activity level 824 rises rapidly to a peak, enabling identification of the second type of anomaly as having occurred. Subsequently, the third activity level 824 falls to a relatively low level as the ANN model adjusts to the second type of anomaly. In this way, system controllers of the present disclosure can detect and adapt to a variety of different types of anomalies in an entirely autonomous way.

[0096] Ingestion Module

[0097] Figure 9 shows a system controller 900 in. Features corresponding to those of Figures 1 and 3 have been given like reference numerals and will not necessarily be further described here.

[0098] Ingestion module 104 receives a stream of data 103 representative of two different sensor modalities. For example, sensor modality A may be a voltage sensor while sensor modality B may be a vibration sensor. Signals derived from the voltage sensor are processed by a first filter 902a. Signals derived from the vibration sensor are processed by the second filter 902b. The first filter 902a provides a single stream of feature data 105a via a first channel A to the ANN model 106. In this example, the first filter 902a may remove high frequency components of the voltage signal and provide a stream of features corresponding to amplitudes of lower frequency bands. The second filter 902b splits the vibration sensor signals into two parts, a lower frequency feature stream 105b provided by channel B to the ANN model 106 and a higher frequency feature stream 105c provided by channel C to the ANN model 106.

[0099] Having received the streams of features 105a-c, the ANN model processes the features and generates node activity data 108 representative of the activity levels of nodes and / or links within the ANN model 106. The node activity data 108 is fed back to the ingestion module 104 where it is used to modify the processing of the stream of data 103 to generate the stream of features 105a-c. For example, the high frequency cut-off applied to the voltage sensor data may be reduced if the node activity data 108 indicates that only the lowest frequency voltage data is particularly relevant to controlling the external apparatus in an optimized way. By modifying the performance of the ingestion module 104, the stream of features provided to the ANN model 106 may render the ANN model 106 more sensitive to anomalies and thereby enable the detection of anomalies at an earlier stage, thus enabling corrective action to be taken before damage or other problems with the performance of an external apparatus arises.

[0100] ANN Model

[0101] An example of the ANN Model 106 will now be described in more detail with reference to Figures 10 and 11. In this example, the ANN model 106 is a random network including a plurality of nodes (which may also be referred to as artificial neurons) that are configured into three sets, in particular a set of input nodes 27, a set of basic nodes 29 and a set of output nodes 31.

[0102] While three input nodes 27a-27c are shown in Figure 10 for ease of explanation, typically there is one input node for each input signal channel and accordingly there is one input node corresponding to each component. Similarly, while three output nodes 33a-33c are shown in Figure 2 for ease of illustration, the number of output nodes is a design choice dependent on the number of feedback values required by the ingestion module 104 and the number of activity values required by the model activity monitor 110. While four basic nodes 29 are shown in Figure 2 for ease of illustration, typically there will be many more basic nodes, for example in the range of ten to five thousand.

[0103] All the nodes of the ANN are labelled either “excitatory” (Exc in Figure 10) or “inhibitory” (Inh in Figure 10). More particularly, in this example all the input nodes 27 and all the output nodes 31 are excitatory nodes while a subset of the basic nodes (represented in Figure 10 by the basic nodes 29a and 29d) are excitatory nodes with the remainder being inhibitory nodes (represented in Figure 10 by the basic nodes 29b and 29c). The difference between an excitatory node and an inhibitory node is that signals received by a recipient node from an excitatory node generally contribute to increasing the activity of the recipient node whereas signals received by a recipient node from an inhibitory node generally contribute to reducing the activity of the recipient node, as will explained in more detail hereafter.

[0104] In this example, nodes within the same set and having the same label are configured to have a predefined number of output edges. Accordingly, each of the input nodes 27 has a single input edge connected to the ingestion module 104 and a first predefined number of output edges interconnecting the input node with basic nodes 29 and output nodes 31. Each of the excitatory basic nodes 29 has a second predefined number of output edges interconnecting the excitatory basic node with other basic nodes 29 and output nodes 31. Each of the inhibitory basic nodes 29 has a third predefined number of output edges interconnecting the inhibitory basic node 29 with other basic nodes 29 and output nodes 31. Each of the output nodes 31 has a single output edge connected to respective one of a set of signal generators 33a-33c, which form the control signal generator of Figure 1, and input edges as mentioned above.

[0105] In this example, the edges within the ANN 15 are initially assigned in accordance with a probabilistic function to allow direct paths through respective populations of basic nodes 29 from an input node 27 to an output node 31, and to add edges connecting the basic nodes 29 of one population to the basic nodes 29 of another population, either directly or via other basic nodes, permitting activity in the one population of basic nodes 29 to interact with the other population of basic nodes 29, and vice versa. More generally, the insertion of each of the plurality of directed edges is specified by probabilities to a subset of the plurality of nodes for each source node when the ANN is constructed, such that pre-established relationships between sensors and actuators are emphasized in the resulting connectivity.

[0106] Figure 11 schematically shows the processing of signals received by a recipient node 41 from multiple excitatory nodes, represented in Figure 11 by three excitatory nodes 43a-43c and hereafter referred to as excitatory nodes 43, and multiple inhibitory nodes, represented in Figure 11 by two inhibitory nodes 45a-45b and hereafter referred to as inhibitory nodes 45. It will be appreciated that the actual number of excitatory nodes 43 and inhibitory nodes 45 in practical implementations will generally be significantly higher.

[0107] The activity signals received by the recipient node 41 from the excitatory nodes 43 and the inhibitory nodes 45 are input to respective different input functions 47a-47e. For each input function 47, the output y is determined in a periodic manner according to the function y(prev_y, x):=MAX(prev_y*label_specific_decay, x) where x is the value of the input to the input function 47, prev_y is a value corresponding to previous output from the input function 47, and label specific delay is a parameter between zero and one that may have different values when the input signal being processed is from an excitatory node and when the input signal being processed is from an inhibitory node. The effect of the input function is that if there is a reduction in the value x of the input of the input function 47 that results in the value x decaying faster than the decay of the previous output of the input function 47 corresponding to the value of the label specific decay parameter, then the value of the decayed previous output y is used in preference to the value of the input x to the input function as the output y of the input function y. This has the effect of reducing high-frequency signals which assists in determining a solution. Such high frequency signals may be generated within recurrent artificial neural networks because there is always a risk of creating positive feedback loops that saturate the network activity, rendering it unresponsive to actual sensory input, and although such positive feedback loops can be at least partially quenched by the inhibitory nodes, the inhibitory quenching lags the build-up of excitatory activity thereby creating high frequency self-amplifying transients. By smoothing the activity of the individual neurons using a decay function for its activity, such self-amplifying transients can be reduced or even avoided, thereby allowing the network activity to focus on determining control signals that reduce the difference between the sensor data and the set point data.

[0108] The value y of the output from each input function 47 is then input to a respective weight function 49, where the value y is multiplied by a weight w corresponding to the edge via which the input signal for that input function 47 was received by the recipient node 41.

[0109] The outputs of the weight functions 49 for signals received from excitatory nodes 43 are then input to a first combiner function 51a, which sums the outputs together to generate a sum L_exc, where:

[0110] L_exc = Xy*w over all the outputs corresponding to excitatory inputs.

[0111] Similarly, the outputs of the weight functions 49 for signals received from inhibitory nodes 45 are then input to a second combiner function 51b, which sums the outputs together to generate a sum L_inh, where:

[0112] L_inh = Xy*w over all the outputs corresponding to inhibitory inputs.

[0113] The values of the parameters L_exc and L_inc are output by the first combiner 51a and the second combiner 51b respectively and input to a base function 53, which determines the magnitude of the activity signal output by the recipient node 41 to other nodes. In this example, the output x of the base function is determined by the expression: x := MAX(0, L_exc - (L_inh / 2)).

[0114] This expression mitigates against the possibility of positive feedback loops being present within the ANN 15, with the L_inh parameter being a determining factor for the rate at which activity in the ANN 15 is reduced. It will be appreciated that variations to this expression can be made while achieving the same effect.

[0115] The output x of the base function 53 is input to an output function 55 which propagates the output x along the output edges of the node 41. In this example, the output function 55 introduces a latency to the propagation of the output x, with a latency value being specified for each output edge. In this example, the latency values are specified in a random manner.

[0116] Introducing latencies to the propagated signals introduces non-linearities into the artificial neural network, which allows the activity of different nodes in the artificial neural network to be differentiated. In this way, the time-varying signal in each node is more unique, thereby increasing the number of options to find solutions in the network by amplifying the weights of the edges from those nodes. This assists in the artificial neural network converging to a robust solution.

[0117] The base function 53 also outputs the L_exc parameter and the L_inh parameter to a learning function 57 which adjusts the weights corresponding to a subset of the input edges so as to reduce activity in the ANN 15 over time and bring the ANN into a stable solution. More particularly, the learning function 57 is a local learning function which adjusts weights for input edges to the corresponding node based on parameters associated with that node.

[0118] For each of the weights being modified by the learning function 57, if the corresponding input edge connects to an excitatory node and the value of the output y of the corresponding input function is greater than the value of the L_exc parameter, then the learning function 57 increases that weight w by an amount dw that may be expressed as: dw += MAX(0, y - 1 + s - L_exc)*rate where s is a suitability parameter for the node 41 and rate is a learning rate, which is a scalar value used to control the size of dw, and if the value of the L_exc parameter is greater than a threshold value T, then the learning function 57 reduces that weight w by an amount dw that may be expressed as: dw -= MAX(O, L_exc - T)*rate.

[0119] Similarly, if the input edge corresponding to a weight being modified by the learning function 57 connects to an inhibitory node then if the output y of the corresponding input function is greater than the value of the L_inh parameter, then the learning function 57 increases that weight w by an amount dw that may be expressed as: dw += MAX(0, y - 1 + s - L_inh)*rate and if the value of the L_inh parameter is greater than a threshold value T, then the learning function 57 diminishes that weight w by an amount dw given by the expression: dw -= MAX(0, L inh - T)*rate.

[0120] As the condition for increasing a weight is dependent on the output y and the condition for reducing a weight is dependent on the summation L_exc, L_inh, it is possible for both conditions to be satisfied in which case the weight is adjusted by the final value of dw after addition and subtraction.

[0121] The suitability parameter s of the node 41 is modified in dependence on changes to the length of a vector V = (y, L_exc, L_inh). In particular, if dV is zero or negative, suggesting that one or both of L_exc and L_inh is decreasing, then the suitability s is increased, for example by a fixed amount, whereas if dV is positive, suggesting an increase in one or both of L_exc and L_inh, then the suitability s is diminished, for example by a fixed amount.

[0122] Increasing the suitability s has the effect that for the same difference between the output y for an edge and L_exc when the node 41 is an excitatory node, or the same difference between the output y for an edge and L_inh when the node 41 is an inhibitory node, the weight w corresponding to that edge can be potentiated by a greater amount. If, however, such an increase in weight results in worse performance of the system and the input y, V will increase over time, leading to the suitability reducing.

[0123] If a positive feedback loop develops in the ANN 15, then the output y corresponding to an edge forming part of the positive feedback loop will grow. The resultant increase in activity results in the suitability s of edges associated with that positive feedback loop diminishing, thereby reducing or eliminating any increase in the weight w for those edges. In this example, in the event that the output y for an edge exceeds the threshold T (for example y > 1) and the weight w cannot be potentiated, then that edge may be randomly assigned a different endpoint node or that edge may be removed and another edge randomly inserted elsewhere in the ANN 15, thereby assisting to break any positive feedback loop.

[0124] The operation of the ANN 15 described above results in the weights for edges being modified until the dw reaches zero for all nodes and the values of the sensed parameters match the values of the corresponding set point data. If the values of L_exc and L_inh are also stably under the threshold T, then all the pathways between the input nodes and the output nodes form part of a negative feedback control system.

[0125] While in the illustrated example the system controller utilises software routines, it will be appreciated that at least some of these software routines may alternatively be implemented by hardware, and that there may be performance benefits in so doing.

[0126] FIG. 12 is an block diagram of an example apparatus for processing data, for example, for implementing an embodiment of a method according to the disclosed technology, illustrating the interconnections between processing circuitry, input unit, memory, and output unit, as well as external data sources and systems according to some embodiments of the disclosed technology; and

[0127] FIG. 13 is a block diagram of another example apparatus for processing data, for example for implementing an embodiment of a method according to the disclosed technology, focusing on internal components and their connections to internal sensory data sources and systems according to some embodiments of the disclosed technology; and

[0128] FIG. 14 is a block diagram of another example apparatus for processing data, for example, for implementing an embodiment of a method according to the disclosed technology, showing the relationships between various components including processing circuitry, memory, and communication units according to some embodiments of the disclosed technology; and

[0129] FIG. 15 is a block diagram of an example distributed system for processing data, for example, for implementing an embodiment of a method according to the disclosed technology, the figure indicating a connection between ingestion apparatus and a remote processing system through a transmit / receive communications unit according to some embodiments of the disclosed technology; and

[0130] FIG. 16 is a block diagram of an example system apparatus for processing multi-modal sensory data using an embodiment of a method according to the disclosed technology, which illustrates the integration of a processing apparatus in a system with a plurality of sensors and optionally actuatable components according to some embodiments of the disclosed technology.

[0131] Applications

[0132] By way of example only, various applications of a system monitor as described above will now be described.

[0133] Robotic Systems

[0134] Robotic systems, such as robots with articulated limbs (e.g. biped or quadruped robots), are known. Other robotic systems may comprise autonomous propulsion systems for movement which do not use articulated limbs. By way of example, the disclosed technology may be used for monitoring the operation of systems such as vehicles, for example, cars and heavy duty vehicles, trains, aircraft, surface vessels or submersibles which do not use articulated limbs for movement, including unmanned airborne vehicles (e.g. drones) or unmanned underwater vehicles or parts thereof. Another example of a system or system component which may be monitored using the disclosed technology, is a motor. The disclosed technology may monitor, for example, one or more of the rotary position, vibrations, the power input, the sound and the temperature of the motor to identify and / or analyse anomalous operation. Optionally, the disclosed technology may be configured to control the actuation of a system component such as a valve or the like for flow regulation or to regulate the speed of a propulsion system.

[0135] For an example such as biped and quadruped robots, the articulated limbs include motors and sensors, and there have previously been successful attempts to train such robotic systems to walk. In an application of the system monitor described above to robots with articulated limbs, the sensor data may convey positional information for different locations on the robotic system. This positional information may, for example, be the distance of each sensed location above a datum point such as the ground. The system monitor may detect anomalous operation indicative of, for example, a loss of balance, and optionally either perform corrective action or issue an alert signal as discussed above.

[0136] By way of another example, for unmanned airborne systems and unmanned underwater systems, navigation can be an issue as environmental factors such as wind or water currents can affect navigation in an unpredictable manner. In an application of the system monitor described above to unmanned airborne or underwater vehicles, the sensor data could be produced by positional sensors, for example utilising a suitable global positioning system, and actuators supplying drive signals to a steering mechanism. The system monitor may, for example, identify anomalous activity indicating a gradual change in wind or water currents, and provide control signals to the steering mechanism or identify more serious anomalous activity, for example caused by interference by an external object, and output an alert signal.

[0137] A Vision System

[0138] A vision system may include a multi-pixel camera mounted on a motor-driven platform to enable the multi-pixel camera to track movement of an object. Each pixel of the multi-pixel camera may input a sensor signal to the system monitor, and the system monitor may determine normal operation as corresponding to the image of the object being present in the central pixels of the camera. The system monitor can then connect anomalous activity, which could relate to a inaccuracy in the tracking or obstruction of part of the image, and optionally output a control signal to the motor- driven platform or an alert signal. Such a vision system may have utility on a vehicle such as an automobile or an airborne vehicle, e.g. a drone, either tracking movement of other vehicles or movement of the vehicle relative to stationary objects.

[0139] An Auditory System

[0140] A cochlear implant can be used to amplify audio signals to produce output signals that can be more readily perceived by a person with hearing difficulties. In an application, one sensor detects the time-varying audio signal in the environment and inputs the sensed audio data into the system controller, while another sensor detects the time-varying output signal from the cochlear implant and inputs the sensed output data into the system controller.

[0141] A pre-processing function in the system monitor decomposes the sensed audio data into individual spectral components using, for example, Fast Fourier Transform (FFT) or the Short-Term Fourier Transform (STFT). Similarly, the pre-processing function decomposes the sensed output data into the same individual spectral components using, for example, Fast Fourier Transform (FFT) or the Short-Term Fourier Transform (STFT). The pre-processing function is then able to calculate a gain value for each individual spectral component based on the corresponding decomposed sensed signal data and output signal data. In an initial phase, the disclosed technology establishes normal operation for the cochlear implant. Following this, the disclosed technology can identify anomalous operation and optionally either output a control signal to vary the gain of the cochlear implant or an alert signal.

[0142] Telecommunications Network

[0143] A telecommunications network, such as a wireless communications network, has multiple network parameters which may have an impact on performance. The performance of such a telecommunications network can be characterized by multiple parameters, including parameters associated with noise, for example bit error rate, data rates and data volumes. The exploration of the network parameters to achieve desired physical performance characteristics, which may for example be determined so as to satisfy one or more service level agreement, is challenging. In an application, the physical performance characteristics of the telecommunications network are measured, and the resultant sensor data corresponding to the measurements is input to the system monitor. Following an initial phase during which the system monitor establishes normal operation, the system monitor can identify anomalous operation and optionally either output a control signal or an alert signal..

[0144] For example, if the aim is meet desired service levels for one or more users of a wireless communications network by optimizing signal coverage and quality, this may be achieved by providing control signals to hardware actuators for one or more of the following: adjusting the tilt angle of one or more antennas in some embodiments; adjusting the power levels of base station transmitters or user equipment to manage interference and ensure adequate signal strength; adjusting the direction of an antenna beam, also known as beam forming, dynamically towards specific areas or users for better coverage and capacity; adjusting small cells and relays to enhances network capacity and remove dead zones in high-density under-served areas; dynamically reconfiguring distributed antenna systems for better load balancing and coverage; and optimizing spatial streams for higher network throughput by adjusting the configuration of multiple input multiple output, MIMO, antennas.

[0145] For example, to optimise signal coverage and quality in a physical network, the disclosed technology may be used to provide control signals to software actuators configured to actuate physical components, for example, one or more of the following: a software actuator for a self-organization network which automatically optimizes network coverage, capacity, and reduces interference by dynamically reconfiguring the network responsive to receiving control signals according to the disclosed technology; a software actuator for adjusting thresholds for handover decisions and / or adjusts other handover optimization parameters responsive to receiving control signals according to the disclosed technology to minimize cell drops and improve user experience; a software actuator for dynamic spectrum allocation, DSA, which causes frequency bands to be adjusted responsive to receiving control signals according to the disclosed technology based on real-time traffic demand and interference conditions; a software actuator for carrier aggregation which combines multiple frequency bands to increase user throughput and overall network capacity responsive to receiving control signals according to the disclosed technology; a software actuator for network slicing responsive to receiving control signals according to the disclosed technology to create virtual networks optimized for specific use cases, for example, loT, video streaming, etc. a software actuator for traffic offloading responsive to receiving control signals according to the disclosed technology to redirect cellular traffic to Wi-Fi or other networks to reduce congestion on that cellular network; a software actuator for implementing load balancing responsive to receiving control signals according to the disclosed technology to redistribute traffic across cells or frequency layers to avoid overloading; and a software actuator for QoS (quality of service) tuning responsive to receiving control signals according to the disclosed technology which prioritizes certain types of traffic, e.g. Video streaming, VoIP, based on service agreements.

[0146] Another example of a system or system component which may be monitored using the disclosed technology comprises a network traffic management optimiser configured to maximise available uplink connectivity in a wireless. For example, the system may be used to configure multiple network resources to optimise one or more of: network capacity, network connectivity, spectrum allocation, and the like.

[0147] Other examples of use cases of the disclosed technology in a networking context include but are not limited to: predictive analytics, e.g. for faults or congestion for proactive or pre-emptive action, e.g. load-balancing, proactive or pre-emptive interference management, energy efficiency, resource scheduling and allocation, content caching, anomaly detection and / or correction management. The disclosed technology may accordingly be used in conjunction with a variety of different sensors, including but not limited to sensors configured to sense physical properties, for example: temperature, humidity, proximity, ultrasound, light including one or more or all of ambient visible light, infra-red light, ultra-violet light, pressure, acceleration, colour, touch, level, position, hall effect, tilt, vibration, gas, chemical(s), vibration.

[0148] Such physical properties may be sensed using sensor systems comprising one or more of the following types of sensors, which is not intended to be a complete list: optical sensors, image sensors, temperature sensors, depth imaging sensors, event imaging sensors, gyroscopic sensors, position sensors, speed sensors, accelerometers, chemical sensors, pressure sensors, electromagnetic field sensors, magnetic field sensors, spectral sensors, electrical current or voltage sensors.

[0149] Actuators may comprise pneumatic actuators, hydraulic actuators, electric actuators, linear actuators, rotary actuators, piezoelectric actuators, magnetic actuators, mechanical actuators, electric motors, solenoids, thermal actuators e.g. heaters or heat-sinks, valves, diaphragm actuators, stepper motors etc.

[0150] In some examples, a robotic limb actuator comprise a plurality of different types of actuators, for example, a combination of linear actuators and rotary actuators.

[0151] The above examples are to be understood as illustrative examples only. Further examples are envisaged. Furthermore, equivalents and modifications not described above may also be employed without departing from the scope of the invention, which is defined in the accompanying claims.

[0152] The content of this description further includes the following numbered clauses.

[0153] Clause 1. A computer-implemented method for training a computer-model comprising at least one time-varying artificial neural network, ANN, to process a data stream, comprising: receiving a data stream comprising data from one or more sensors for ingestion into the time-varying ANN; processing the received data stream to derive at least one stream of features, each stream of features being ingested along a separate channel into the time-varying ANN computer model; analysing, the at least one stream of features as they are ingested into the timevarying ANN computational model; and monitoring one or more internal states adopted by the time-varying ANN computer model responsive to ingesting of the at least one stream of features, wherein the time-varying ANN computational model adopts one or more internal states as it processes one or more or all of the at least one ingested data streams, each ingested data stream comprising at least one feature derived from the multi-variate data which differs from at least one feature derived from the multi-variate data in another ingested data stream.

[0154] Clause 2. The computer-implemented method of clause 1, wherein the monitoring generates at least one indication of a capability of the time-varying ANN model to process ingested data.

[0155] Clause 3. The method of clause 1 or clause 2, wherein the method further comprises: extracting diagnostic data comprising data representing one or more characteristics of each of the one or more internal state adopted by the time-varying model; analysing using a ANN state analysis model one more characteristics of a plurality of internal states adopted by the time-varying model over a first period of time as it processes input comprising the at least one streams of features of the received data stream; and determining, from one or more characteristics of a plurality of transient internal states adopted by the ANN computer model over a period of time when non- anomalous data is input to the model, when the ANN computer model is trained.

[0156] Clause 4. The method of any one of clauses 1 to 3, wherein the received data stream comprises a multi-variate data stream, and wherein each of the at least one stream of features is derived from the received multi-variate data stream. For example, and wherein the at least one stream of features is analysed using a first multi-variate analysis computer model on ingestion to the time-varying computational ANN model. Clause 5. The method of any one of clauses 1 to 4, wherein each of the at least one stream of features is derived by applying a data transform model to the received data stream.

[0157] Clause 6. The method of clause 5, wherein the transform model is a Fourier transform model.

[0158] Clause 7. The method of any one of the previous clauses, wherein the ANN model adopts a plurality of transient states as the model updates one or more connections between each source node and each target node in the ANN, wherein a source node outputs activity within the ANN and a target node receives activity within in the ANN, wherein the one or connections are updated based on the data features input to the time-varying computational ANN model.

[0159] Clause 8. The method of any one of the previous clauses, wherein the ANN model adopts a plurality of transient states as the ANN model updates internal weights applied by source nodes of the ANN model when processing data input to the ANN model based on one or more data features in the input data to the time-varying computational ANN model.

[0160] Clause 9. The method of any one of the previous clauses , wherein the ANN model adopts a plurality of transient states as the model updates which nodes in the ANN act as source nodes for internal activity within the ANN and which nodes in the ANN act as target nodes receiving data from source nodes in the ANN model based on the data features input to the time-varying computational ANN model.

[0161] Clause 10. The method of any one of clauses 1 to 9, wherein processing the multi-variate data stream to derive at least one stream of features comprises: generating a plurality of streams of features; separately inputting each stream of features of the plurality of streams of features into the time-varying computational model.

[0162] Clause 11. The method of clause 10, wherein each stream of features derived from the multi-variate stream is processed in parallel by the ANN computational model.

[0163] Clause 12. The method of any one of the previous clauses, wherein the processing of the received data stream to derive at least one stream of features is dependent on at least one characteristic of the plurality of internal states adopted by the time-varying ANN model when processing previously received multi-variate data.

[0164] Clause 13. The method of clause 12, wherein the processing of the received data stream includes performing a frequency decomposition, wherein the frequency decomposition for each channel the receive data stream is split into is dependent on the activity of nodes in the time-varying ANN.

[0165] Clause 14. The method of any one of the previous cluses, wherein the timevarying ANN comprises a time-varying model architecture.

[0166] Clause 15. The method of any one of the previous clauses, wherein one or more nodes of the ANN are configured to spontaneously generate activity within the ANN, wherein spontaneous activity comprises activity which is generated in a processing timestep by a node independently from any activity generated from the at least one stream of features input to the ANN in that timestep or previous timesteps.

[0167] Clause 16. The method of clause 15, wherein one or more nodes of the ANN are configured to generate activity in a timestep within the ANN from the at least one stream of features input to the ANN in that timestep or previous timesteps.

[0168] Clause 17. The method of either clause 15 or clause 16, wherein the timevarying ANN comprises a time-varying model architecture which is time-varying depending on the activity generated within the ANN.

[0169] Clause 18. The method of clause 17, wherein the time-varying activity generated within the ANN causes the ANN to adopt a plurality of different states, wherein when the activity in the ANN follows a stable trajectory path within the ANN model architecture, the ANN is trained on the received multi -variate data.

[0170] Clause 19. The method of any of the preceding clauses, wherein the multivariate data comprises multi-modal data.

[0171] Clause 20. The method of any one of the preceding clauses, wherein the model is trained in real-time or near real-time on data as the data stream is fed into the model.

[0172] Clause 21. A computer-implemented method of processing a data stream, the method comprising: inputting the data stream into a time-varying ANN computer model trained on a non-anomalous data stream using a method according to any one of clauses 1 to 20; determining in a time-step of the ANN model, for each of one or more spatial locations in the ANN computer model, one or more representations of internal activity at the one or more spatial locations in the ANN computer model, wherein the internal activity includes activity derived from the data stream input in that time-step or a previous time-step; comparing, for each time-step in which one or more representations of internal activity at one or more spatial locations of the ANN model were determined, one or both of: at least one representation of internal activity at a spatial location of the ANN model derived from the input data stream in that time-step with at least one representation of internal activity at that spatial location in the ANN model trained on non-anomalous data; and at least one spatial location where a representation of internal activity is generated in the ANN model in that time-step derived from the input data stream with at least one spatial location where a representation of internal activity in the ANN model trained on non-anomalous data was previously generated; and, determining the input data stream being processed in that time-step by the ANN comprises anomalous data based on the comparing.

[0173] Clause 22. The method of clause 21, wherein the method further comprises: adjusting the processing of the data stream to derive at least one stream of features to modify the separation of features into separate streams, each stream being input along a separate channel into the time-varying ANN computer model.

[0174] Clause 23. The method of clause 22, wherein the method further comprises, based on the adjusted channel inputs to the time-varying ANN computer model, retraining the ANN using non-anomalous data.

[0175] Clause 24. A system for anomaly detection, comprising: means for receiving a data stream comprising data from one or more sensors; means for processing the received data stream to derive at least one stream of features; means for analyzing the at least one stream of features; means for monitoring one or more internal states adopted by a time-varying artificial neural network (ANN) computer model responsive to ingesting the at least one stream of features; means for comparing representations of internal activity in the ANN model derived from the input data stream with representations of internal activity in the ANN model trained on non-anomalous data; and means for determining the input data stream comprises anomalous data based on the comparing.

[0176] Clause 25. A system for anomaly detection, comprising: means for receiving a data stream comprising data from one or more sensors; means for processing the received data stream to derive at least one stream of features, each stream of features being ingested along a separate channel into a timevarying artificial neural network (ANN) computer model; means for analyzing the at least one stream of features as they are ingested into the time-varying ANN computer model; means for monitoring one or more internal states adopted by the time-varying ANN computer model responsive to ingesting of the at least one stream of features; means for extracting diagnostic data comprising data representing one or more characteristics of each of the one or more internal states adopted by the time-varying ANN computer model; means for analyzing, using an ANN state analysis model, one or more characteristics of a plurality of internal states adopted by the time-varying ANN computer model over a first period of time as it processes input comprising the at least one stream of features of the received data stream; means for determining, from one or more characteristics of a plurality of transient internal states adopted by the ANN computer model over a period of time when non-anomalous data is input to the model, when the ANN computer model is trained; means for inputting a new data stream into the trained time-varying ANN computer model; means for determining, in a time-step of the ANN computer model, for each of one or more spatial locations in the ANN computer model, one or more representations of internal activity at the one or more spatial locations in the ANN computer model; means for comparing, for each time-step in which one or more representations of internal activity at one or more spatial locations of the ANN computer model were determined, at least one of:

[0177] (a) at least one representation of internal activity at a spatial location of the ANN computer model derived from the input new data stream in that time-step with at least one representation of internal activity at that spatial location in the ANN computer model trained on non-anomalous data; and

[0178] (b) at least one spatial location where a representation of internal activity is generated in the ANN computer model in that time-step derived from the input new data stream with at least one spatial location where a representation of internal activity in the ANN computer model trained on non-anomalous data was previously generated; and means for determining the input new data stream being processed in that timestep by the ANN computer model comprises anomalous data based on the comparing.

[0179] Clause 26. The system of clause 25, further comprising: means for adjusting the processing of the new data stream to derive at least one stream of features to modify the separation of features into separate streams, each stream being input along a separate channel into the time-varying ANN computer model; and means for re-training the ANN computer model using non-anomalous data based on the adjusted channel inputs to the time-varying ANN computer model.

[0180] Clause 27. The system of clause 25 or clause 26, wherein the time-varying ANN computer model comprises a time-varying model architecture.

[0181] Clause 28. The system of any one of clauses 25 to 27, wherein one or more nodes of the ANN computer model are configured to spontaneously generate activity within the ANN computer model, wherein spontaneous activity comprises activity which is generated in a processing timestep by a node independently from any activity generated from the at least one stream of features input to the ANN computer model in that timestep or previous timesteps. Clause 29. The system of clause 28, wherein the time-varying ANN computer model comprises a time-varying model architecture which is time-varying depending on the activity generated within the ANN computer model.

[0182] Clause 30. The system of clause 29, wherein the time-varying activity generated within the ANN computer model causes the ANN computer model to adopt a plurality of different states, wherein when the activity in the ANN computer model follows a stable trajectory path within the ANN model architecture, the ANN computer model is trained on the received data stream.

[0183] Clause 31. A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform a method for training a computer-model comprising at least one time-varying artificial neural network (ANN) to process a data stream, the method comprising: receiving a data stream comprising data from one or more sensors for ingestion into the time-varying ANN; processing the received data stream to derive at least one stream of features, each stream of features being ingested along a separate channel into the time-varying ANN computer model; analyzing the at least one stream of features as they are ingested into the timevarying ANN computational model; and monitoring one or more internal states adopted by the time-varying ANN computer model responsive to ingesting of the at least one stream of features, wherein the time-varying ANN computational model adopts one or more internal states as it processes one or more or all of the at least one ingested data streams, each ingested data stream comprising at least one feature derived from the multi-variate data which differs from at least one feature derived from the multi-variate data in another ingested data stream.

[0184] Clause 32. The non-transitory computer-readable storage medium of clause 31, wherein the monitoring generates at least one indication of a capability of the time-varying ANN model to process ingested data.

[0185] Clause 33. The non-transitory computer-readable storage medium of clause 31 or clause 32, wherein the method further comprises: extracting diagnostic data comprising data representing one or more characteristics of each of the one or more internal state adopted by the time-varying model; analyzing using an ANN state analysis model one or more characteristics of a plurality of internal states adopted by the time-varying model over a first period of time as it processes input comprising the at least one streams of features of the received multi-variate data stream; and determining, from one or more characteristics of a plurality of transient internal states adopted by the ANN computer model over a period of time when non- anomalous data is input to the model, when the ANN computer model is trained.

[0186] Clause 34. The non-transitory computer-readable storage medium of any one of clauses 31 to 33, wherein each of the at least one stream of features is derived from a received multi-variate data stream, and wherein the at least one stream of features is analyzed using a first multi-variate analysis computer model on ingestion to the timevarying computational ANN model.

[0187] Clause 35. The non-transitory computer-readable storage medium of any one of clauses 31 to 34, wherein each of the at least one stream of features is derived by applying a data transform model to the received data stream.

[0188] Clause 36. The non-transitory computer-readable storage medium of clause 35, wherein the transform model is a Fourier transform model.

[0189] Clause 37. The non-transitory computer-readable storage medium of any one of clauses 31 to 36, wherein the ANN model adopts a plurality of transient states as the model updates one or more connections between each source node and each target node in the ANN, wherein a source node outputs activity within the ANN and a target node receives activity within in the ANN, wherein the one or connections are updated based on the data features input to the time-varying computational ANN model.

[0190] Clause 38. The non-transitory computer-readable storage medium of any one of clauses 31 to 37, wherein the ANN model adopts a plurality of transient states as the ANN model updates internal weights applied by source nodes of the ANN model when processing data input to the ANN model based on one or more data features in the input data to the time-varying computational ANN model. Clause 39. The non-transitory computer-readable storage medium of any one of clauses 31 to 38, wherein the ANN model adopts a plurality of transient states as the model updates which nodes in the ANN act as source nodes for internal activity within the ANN and which nodes in the ANN act as target nodes receiving data from source nodes in the ANN model based on the data features input to the time-varying computational ANN model.

[0191] Clause 40. The non-transitory computer-readable storage medium of any one of clauses 31 to 39, wherein processing the multi -variate data stream to derive at least one stream of features comprises: generating a plurality of streams of features; separately inputting each stream of features of the plurality of streams of features into the time-varying computational model.

[0192] Clause 41. The non-transitory computer-readable storage medium of clause 40, wherein each stream of features derived from the multi-variate stream is processed in parallel by the ANN computational model.

[0193] Clause 42. The non-transitory computer-readable storage medium of any one of clauses 31 to 41, wherein the processing of the received data stream to derive at least one stream of features is dependent on at least one characteristic of the plurality of internal states adopted by the time-varying ANN model when processing previously received multi-variate data.

[0194] Clause 43. The non-transitory computer-readable storage medium of clause 42, wherein the processing of the received data stream includes performing a frequency decomposition, wherein the frequency decomposition for each channel the receive data stream is split into is dependent on the activity of nodes in the timevarying ANN.

[0195] Clause 44. The non-transitory computer-readable storage medium of any one of clauses 31 to 43, wherein the time-varying ANN comprises a time-varying model architecture.

[0196] Clause 45. The non-transitory computer-readable storage medium of any one of clauses 31 to 44, wherein one or more nodes of the ANN are configured to spontaneously generate activity within the ANN, wherein spontaneous activity comprises activity which is generated in a processing timestep by a node independently from any activity generated from the at least one stream of features input to the ANN in that timestep or previous timesteps

Claims

CLAIMS1. A computer-implemented method to analyse operational performance of a system based on sensor data associated with the operation of the system, the method comprising: receiving the sensor data; processing the sensor data to provide a plurality of input signals for an Artificial Neural Network, ANN, wherein the processing comprises: transforming the sensor data into a plurality of components that are characteristic of the operation of the system and normalised with respect to activity values within the ANN, each component having a corresponding component value; and for each component, determining a difference value between the normalised component value and a respective feedback value from the ANN and generating an input signal representative of the difference value; providing the plurality of input signals to the ANN; and monitoring activity associated with the ANN following provision of the plurality of input signals to detect anomalous activity indicative of a departure from normal operation of the system.

2. The computer-implemented method of claim 1, wherein the sensor data comprises a plurality of data streams that are characteristic of the operation of the system and the transforming comprises normalising values of the data streams with respect to activity values within the ANN.

3. The computer-implemented method of claim 1 or claim 2, wherein the sensor data comprises at least one data stream having multiple components of the plurality of components that are characteristic of the operation of the system, and the transforming comprises transforming each of the at least one data stream into the multiple components.

4. The computer-implemented method according to claim 3, wherein the at least one data stream includes a data stream for which different frequencyranges correspond to different components of the plurality of components that are characteristic of the operation of the system, and the transforming comprises a spectral decomposition of the data stream.

5. The computer-implemented method of any preceding claim, wherein the sensor data comprises data streams from multiple sensor modalities.

6. The computer-implemented method of claim 5, wherein the multiple sensor modalities include two or more of position sensing, audio sensing, video sensing and temperature sensing.

7. The computer-implemented method of any preceding claim, wherein the ANN has a plurality of nodes interconnected by a plurality of edges, and the ANN is configured to adjust weights associated with edges of the ANN to reduce the difference values corresponding to the input signals.

8. The computer-implemented method of claim 7, wherein the adjusting of weights comprises each node using a local learning rule to adjust the weights for one or more of the plurality of edges for which the node receives activity values output by other nodes.

9. The computer-implemented method of claim 7 or claim 8, comprising an initial training phase during which the weights associated with edges of the ANN are adjusted until the monitored activity of the ANN satisfies a condition whereby the monitored activity after the initial training phase represents the normal operation of the system.

10. The computer-implemented method of claim 9, comprising monitoring the variation of activity of nodes within the ANN with variations in each of the input signals during the initial training phase to determine whether or not learning is being performed within the ANN in respect of all the input signals.

11. The computer-implemented method of claim 10, wherein if it is determined that learning is not being performed with respect to one or more input signals, then a subset of the input signals is input to the ANN.

12. The computer-implemented method of claim 9, wherein subsets of the input signals are separately input to the ANN during the initial training phase, each subset comprising one or more of the input signals.

13. The computer-implemented method of any preceding claim, wherein the sensor data is received from the system in real time or near real time.

14. The computer-implemented method of any preceding claim, further comprising: analysing the monitored activity signals corresponding to a detection of an anomaly; and generating an output signal based on the analysis of the monitored activity signals.

15. The computer-implemented method of claim 14, wherein the output signal is output to the system as a control signal for one or more actuators within the system.

16. The computer-implemented method of claim 14, wherein the output signal is an alert signal.

17. The computer-implemented method of claim 16, wherein the alert signal is one of an audible alert and a visual alert.

18. A system controller, configured to analyse operational performance of a system based on sensor data associated with operation of the system, comprising: a processor, configured to receive and process the sensor data and to provide a plurality of input signals for an Artificial Neural Network, ANN, wherein the processor is further configured to: transform the sensor data into a plurality of components that are characteristic of the operation of the system and normalised with respect to activity values received from within the ANN, each component having a corresponding component value;for each component, determine a difference value between the normalised component value and a respective feedback value received from the ANN and generate an input signal representative of the difference value; and receive and monitor activity signals associated with the ANN to detect anomalous activity indicative of a departure from normal operation of the system.

19. The system controller of claim 18, wherein the sensor data comprises a plurality of data streams that are characteristic of the operation of the system and the processor is configured to transform the sensor data by normalising values of the data streams with respect to the activity values received from within the ANN.

20. The system controller of claim 18 or claim 19, wherein the sensor data comprises at least one data stream having multiple components of the plurality of components that are characteristic of the operation of the system and the processor is configured to transform the sensor data by transforming each of the at least one data stream into the multiple components.

21. The system controller of claim 20, wherein the at least one data stream includes a data stream for which different frequency ranges correspond to different components of the plurality of components that are characteristic of the operation of the system and the processor is configured to determine a spectral decomposition of the data stream.

22. The system controller of any of claims 18 to 21, wherein the sensor data comprises data streams from multiple sensor modalities.

23. The system controller of claim 22, wherein the multiple sensor modalities include two or more of position sensing, audio sensing, video sensing and temperature sensing.

24. The system controller of any of claims 18 to 23, further comprising the ANN, wherein the ANN has a plurality of nodes interconnected by a plurality of edges, and the ANN is configured to adjust weights associated with edges of the ANN to reduce the difference values corresponding to the input signals.

25. The system controller of claim 24, wherein the ANN is configured to adjust the weights by, for each node, using a local learning rule to adjust the weights for one or more of the plurality of edges for which the node receives activity values output by other nodes.

26. The system controller of claim 24 or claim 25, wherein the ANN is configured during an initial training phase during which the weights associated with edges of the ANN are adjusted until the monitored activity signals of the ANN satisfy a condition whereby the monitored activity signals after the initial training phase represent the normal operation of the system.

27. The system controller of claim 26, wherein the processor is configured to monitor a variation of activity of nodes within the ANN with respect to variations in each of the input signals during the initial training phase to determine whether or not learning is being performed within the ANN with respect to all the input signals.

28. The system controller of claim 27, wherein if the processor determines that learning is not being performed with respect to one or more of the input signals, then a subset of the input signals is input to the ANN.

29. The system controller of claim 26, wherein the processor is configured to provide subsets of the input signals separately to the ANN during the initial training phase, each subset comprising one or more of the input signals.

30. The system controller of any of claims 18 to 29, wherein the sensor data is received from the system in real time or near real time.

31. The system controller of any of claims 18 to 30, wherein the processor is configured to: analyse the monitored activity signals corresponding to a detection of an anomaly; and generate an output signal based on the analysis or the monitored activity signals.

32. The system controller of claim 31, wherein the output signal is output to the system as a control signal for one or more actuators within the system.

33. The system controller of claim 31, wherein the output signal is an alert signal.

34. The system controller of claim 33, wherein the alert signal is one of an audible alert and a visual alert.

35. A robotic system comprising a system controller according to any of claims 18 to 34.

36. A vision system comprising a system controller according to any of claims 18 to 34.

37. An auditory system comprising a system controller according to any of claims 18 to 34.

38. A telecommunications network comprising a system controller according to any of claims 18 to 34.

Images