Apparatus and method for evaluation of sensor data via undersampling of sensor data

Abstract

An apparatus and method for evaluation of sensor data can include creating an undersampling of the data via a randomization selection process and subsequently evaluating that data with a trained machine learning model to determine or detect at least one condition or parameter based on the sensor data. The processing can be performed by the sensor itself, a remote device, or a computer device communicatively connected to the one or more sensors collecting the data to be evaluated. In some embodiments, the creation of the undersampled portion of sensor data can be performed by one device and the subsequent evaluation of the data can be performed by another device. In other embodiments, the undersampling and subsequent evaluation can all be performed by the same device.

Description

[0001] APPARATUS AND METHOD FOR EVALUATION OF SENSOR DATA VIA UNDERSAMPLING OF SENSOR DATA

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS

[0003] The present application claims priority to U.S. Provisional Patent Application No. 63 / 617,463, which was filed on January 4, 2024. The entirety of this provisional patent application is incorporated by reference herein.

[0004] FIELD

[0005] The present innovation relates to apparatuses and methods for evaluation of sensor data. The sensor data can be any type of 1-D data (e.g. periodic data, non-pcriodic data, labeled sensor data, non-labeled sensor data, etc.) In some embodiments, the sensor data can include, for example, sensor data used to collect data related to operation of a process, operation of a machine (e.g. rotating industrial and manufacturing machines, spectral analysis of sensor data, etc.) or deployment of a vehicle (e.g. a drone). Some embodiments can be configured to help process such data in a way that can allow for detection of a condition being monitored with reduced bandwidth and processing requirements. Other embodiments can be configured to help process such sensor data in a way that can allow for monitoring of a condition or situation with reduced bandwidth and processing requirements. Such features can be provided via edge devices as well as in cloud processing implementations. Embodiments of the apparatus can include a computer device, and can also include a communication system that can include multiple devices (e.g. a sensor, and edge device and / or a cloud based device, etc.).

[0006] BACKGROUND

[0007] Sensors are often used to monitor a process or device. For example, multiple sensors can be used to help monitor operation of a drone, vehicle, industrial machine, or a chemical process. Such sensor data is often evaluated in a way in which all the data collected by the sensor is ultimately received and analyzed to detect a condition or determine a status of a process or device.

[0008] SUMMARY

[0009] The processing required by systems that may utilize all the data collected by one or more sensors (e.g. an array of sensors, a single sensor, etc.), can require a significant amount of (i) data infrastructure costs (e.g. computational time, power, and bandwidth, etc.) and (ii) human capital costs for analyzing large volumes of data. For example, as more and more data is collected from one or more sensors (e.g. a single sensor, an array of sensors, etc.) deployed to monitor different types of assets, there can be a significant increase in the (i) data infrastructure (edge computing power, cloud computing power, computation time, latency, bandwidth availability and costs, etc.) and, (ii) human capital time and costs to perfomi analytics on all this data. We have determined that an updated process and apparatus for evaluating sensor data can be needed to help significantly reduce the data infrastructure and human capital time and cost requirements without sacrificing accuracy of a determination based on that sensor data.

[0010] Embodiments of our apparatus and method can be employed in conjunction with evaluation of data collected from only a single sensor or in conjunction with the evaluation of data collected from an array of sensors (e.g. two sensors, more than two sensors, etc.). The sensors can include, for example, temperature sensors, humidity sensors, pressure sensors, proximity sensors, level sensors, accelerometers, gyroscope sensors, gas sensors, infrared sensors, or other types of sensors. We believe some embodiments can be used advantageously in conjunction with more high frequency sampling sensors like vibration sensors, sound sensors, or power sensors that can be employed to facilitate undersampling of a large volume of sensor data to provide improved processing efficiencies and reduced bandwidth requirements.

[0011] Other embodiments can be used in conjunction with other types of sensors. In one other advantageous embodiment, an embodiment can be utilized on conjunction with spectroscopic sensors to facilitate undersampling in time to provide an ability to identify a specific condition with less than usual exposure time.

[0012] Embodiments can be configured so that the sensor, an edge device communicatively connected to the sensor, and / or a remote device (e.g. a cloud based server having a processor connected to non-transitory memory and at least one transceiver unit, an array of such cloud based servers, etc.) can receive a randomly distributed portion of the data collected by the sensor(s) and process that randomly distributed portion of the data to determine at least one condition or parameter of a process, a device, a machine, or a vehicle. The sensor data that can be processed can be any type of 1-D data, for example (e.g. periodic data, non-periodic data, labeled sensor data, non-labeled sensor data, etc.). The randomly distributed portion of the sensor data can be less than 50% of the collected sensor data, less than 30% of this data, or even less than 10% of this data in some embodiments. For example, some embodiments can be configured to only utilize 5%-l 5% of the collected sensor data or 5%-20% of the collected sensor data. Other embodiments may only utilize between 5% and 30% of the collected sensor data. Yet other embodiments may only utilize between 5% and 50% of the sensor data. The specific portion of the collected sensor data (e.g. less than 50% and greater than 5%, less than 40% and greater than 5%, less than 20% and greater than 5%, etc.), that may be utilized can vary for a specifically adapted embodiment to account for a specific implementation and the level of accuracy required by that implementation. In some testing we have performed, we have found that undersampling of data that is between 5% and 20% of the collected data can provide substantial improvement and / or provide results that can be considered a best result for a particular situation.

[0013] Embodiments can be utilized in numerous different types of industrial applications or other uses. For example, embodiments can be utilized for monitoring remote high-Capex rotating industrial machines like wind turbines, gas turbines, motors, pump, centrifuges, or heating ventilation and air conditioning (HVAC) type devices. As another example, embodiments can be utilized in conjunction with spectral analysis of radio frequency-based data from satellites, unmanned aerial vehicles, unmanned underwater vehicle, or drones. Embodiments can apply a random sampling processing scheme that can provide a mechanism for offloading processing demands for such devices to a more powerful remote computer unit, for example. As yet another example of an embodiment that can be applied in an industrial or commercial setting, embodiments can be utilized in agriculture. For instance, embodiments can be utilized to monitor or evaluate nitrogen content in a farm, moisture content, and / or crop health based on evaluation of spectral data obtained by one or more sensors, which is a type of data that can be decomposed into a set of frequencies, where each frequency means some physical condition / fault / health, etc. Yet other embodiments can be utilized in conjunction with ocean awareness or other type of body of water awareness (lake, river, sea, etc.). In such applications, embodiments can be configured for obj ect detection, mammal passing, fish passing, animal monitoring, and / or atmospheric condition monitoring.

[0014] Yet other embodiments can be utilized in predictive maintenance situations. For example, an embodiment can be utilized in conjunction with evaluating predictive maintenance for the International Space Station in which an objective would be to keep utility equipment like a fan, motor, etc. running. Embodiments can be utilized to pemiit lightweight edge analytics to be utilized to provide better computational efficiency and lower power requirements.

[0015] Embodiments can be configured so that the sensor itself, an edge device (e.g. intermediate computer device communicatively connected to the sensor device), or a remote computer device (e.g. a cloud based server etc.) can reduce the collected sensor data to an undersampling of the entirety of the sensor data set so that only a portion of not more than 60% and greater than 5% of the collected sensor data is utilized. The utilized portion of data can be randomized sampling of the data in accordance with uniform or Gaussian randomization schemes, for example. Other randomization schemes may also be utilized in other embodiments (e.g. Bernoulli or other suitable randomization scheme that may be utilized). The randomized undersampling portion of the sensor data can be fed to a neural network or other type of machine learning model run on the device (e.g. sensor, intermediate device, remote device, etc.) to evaluate the data and detect a condition or parameter being monitored by the sensor data. In some embodiments, the machine learning model can be a trained model that was trained via a machine learning algorithm that is at least partially pre-defined by code stored in a non-transitory memory or other type of non-transitory computer readable medium that can be executed by a processor of a device.

[0016] The evaluation of the data can be configured to utilize regression and / or classification processing. The neural network or other type of machine learning model can be trained via a machine learning algorithm that can use simulated data and / or empirical real-world data collected from sensors for training of the machine learning model. In some embodiments, the neural network or other type of machine learning model may only be trained via a machine learning algorithm that utilizes simulated sensor data.

[0017] In some embodiments, a lightweight sensing and data analysis apparatus can be provided that includes (i) an edge device (typically a single board computer) for collecting undersampled measurements, and one of: (ii) a lightweight compute device on the edge device using shiftinvariant undersampled networks, OR (iii) the edge device configured for transmitting the collected undersampled measurements based on a positional encoding scheme (e.g. fixed seed, etc.) to at least one remote processing element configured to: (a) receive randomly sampled data from the edge device; (b) store the randomly sampled data in a data store; and (c) run a shiftinvariant undersampled network to perform a classification or regression analysis. Such embodiments can be configured so that when only a label is needed for evaluation of the collected undersampled measurements, the lightweight computer device is utilized and otherwise, the at least one remote processing element is utilized. The utilization of the at least one remote processing element can be configured to preserve more information via latent representations learned in the network, depending on the task to be solved and the utility / life cycle of the information (label vs. more information - improvement, history, prediction, etc.). More undersampled measurements can help improve the cloud model over time, which can be deployed on the edge for better labels or inference determinations.

[0018] Some embodiments of the machine learning model that may be utilized can include a neural network, a support vector machine (SVM), a k-Nearest Neighbors (kNN) machine, or a logistic regression machine.

[0019] In a first aspect, an apparatus for evaluation of sensor data is provided. Embodiments of the apparatus can include a computer device integrated with and / or communicatively connectable to at least one sensor to receive sensor data from the at least one sensor. The computer device can have a processor connected to a non-transitory computer readable medium and at least one transceiver unit. The computer device can be configured to undersample the sensor data received from the sensor for evaluation of the undersampled sensor data. In a second aspect, the computer device can also be configured to evaluate the undcrsamplcd sensor data via a neural network or machine learning model defined in code of the non-transitory computer readable medium.

[0020] In a third aspect, the computer device can also be configured to communicatively connect to a remote computer device for sending the undersampled sensor data to the remote computer device. The remote computer device can be configured to evaluate the undersampled sensor data via a neural network or machine learning model defined in code of a non-transitory computer readable medium of the remote computer device. In some embodiments, the remote computer device can be a server or an array of servers that may host at least one telecommunications service, for example.

[0021] In a fourth aspect, the at last one sensor can include at least one of: a temperature sensor, a humidity sensor, a pressure sensor, a proximity sensor, a level sensor, an accelerometer, a gyroscope sensor, a gas sensor, an infrared sensor, a microphone, a sound sensor, a vibrational sensor, a power sensor, radio frequency sensor, distributed fiber optics sensing (DFOS) device, Human Activity Recognition (HAR) device, thermal radiation sensor, ultrasound sensor, or a light sensor.

[0022] In a fifth aspect, the computer device can be an edge device. For instance, the computer device can have a relatively small processing capacity, can be a laptop computer, a tablet, or other type of edge device or another type of lightweight edge device.

[0023] In a sixth aspect, the computer device can be integrated with the at least one sensor. For example, the computer device can be a component of a sensor or sensor assembly and may communicate data to at least one remote computer device.

[0024] In a seventh aspect, the computer device can be a remote computer device. For instance, the computer device can be a server, a workstation, a desktop computer, or other type of computer device that may be communicatively connected to the at least one sensor via a network connection or other type of communicative connection.

[0025] In an eighth aspect, computer device can be configured to undersample the sensor data by forming a randomization portion of the sensor data that includes less than or equal to 60% of the sensor data, less than or equal to 40% of the sensor data, or between 5% and 20% of the sensor data. In other embodiments, the sensor data may be undersampled by randomization of a portion of the sensor data that is less than or equal to 50% of the sensor data, less than 20% of the sensor data, less than 10% of the sensor data, or less than or equal to 5% of the sensor data. The undcrsamplcd sensor data can also be greater than 0% of the sensor data while also being less than the entirety of the sensor data (e.g. less than 50% of the sensor data, less than 60% of the sensor data, less than 20% of the sensor data, less than 10% of the sensor data, etc.).

[0026] In a ninth aspect, the computer device can be configured to undersample the sensor data via a randomization process, Gaussian randomization process or a uniform randomization process defined in code stored in the non-transitory computer readable medium.

[0027] In a tenth aspect, the apparatus of the first aspect can include one or more features of the second aspect, third aspect, fourth aspect, fifth aspect, sixth aspect, seventh aspect, eighth aspect and / or ninth aspect. Other embodiments may also include other elements or features. Examples of such elements or features can be appreciated from the exemplary embodiments discussed herein, for instance.

[0028] In an eleventh aspect, a method for evaluation of sensor data can be provided. Embodiments of the method can include collecting sensor data from at least one sensor, selecting a portion of the sensor data for evaluation via undersampling of the collected sensor data; and at least one of: (i) transmitting the selected portion of the sensor data for the evaluation of the sensor data; and / or (ii) evaluating the selected portion of the sensor data.

[0029] In a twelfth aspect, the evaluating of the selected portion of the sensor data can include feeding the selected portion of the sensor data to an input layer of a neural network defined in code of a non-transitory computer readable medium of a computer device for evaluation of the sensor data.

[0030] In a thirteenth aspect, the computer device can be communicatively connected to the at least one sensor or can be integrated with the at least one sensor.

[0031] In a fourteenth aspect, the method can be performed such that the computer device is remote from an edge device that receives the sensor data from the at least one sensor and performs the selecting of the portion of the sensor data, the edge device also performing the transmitting of the selected portion of the sensor data for the evaluation of the sensor data, the transmitting of the selected portion of the sensor data for the evaluation of the sensor data comprising the edge device transmitting the selected portion of the sensor device to a remote computer device, the remote computer device performing the evaluating of the selected portion of the sensor data. In a fifteenth aspect, the evaluating of the selected portion of the sensor data can include classification of a condition based on the selected portion of the sensor data or performing a regression analysis of the selected portion of the sensor data to determine a condition or parameter.

[0032] In a sixteenth aspect, the portion of the sensor data that is selected for evaluation via undersampling of the collected sensor data can be less than or equal to 60% of the collected sensor data, less than or equal to 40% of the collected sensor data, or is between 5% and 20% of the collected sensor data. For example, the portion of the sensor data that is selected for evaluation can be less than 50%, less than 30%, less than 20%, less than 10% or less than 5% of the sensor data.

[0033] In a seventeenth aspect, the method can also include performing a randomization for the selected portion of the sensor data for evaluation of the selected portion of the sensor data.

[0034] In an eighteenth aspect, the method of the eleventh aspect can include one or more features of the twelfth aspect, thirteenth aspect, fourteenth aspect, fifteenth aspect, sixteenth aspect, and / or seventeenth aspect. Embodiments may also include other features or elements. Examples of such elements or features can be appreciated from the exemplary embodiments discussed herein, for instance. Embodiment of the apparatus can be utilized in conjunction with an embodiment of the method or can be configured to implement an embodiment of the method as well.

[0035] In a nineteenth aspect, a lightweight sensing and data analysis apparatus is provided. The apparatus can include (i) an edge device for collecting undersampled measurements. The apparatus can also include (ii) a lightweight computer device on the edge device using shiftinvariant undersampled networks. Alternatively, the edge device can be configured for transmitting the collected undersampled measurements based on a positional encoding scheme to at least one remote processing element configured to: receive randomly sampled data from the edge device, store the randomly sampled data in a data store, and run a shift-invariant undersampled network to perform a classification or regression analysis.

[0036] In some embodiments, the lightweight computer device can be a computer device having a relatively non-powerful processor. Examples of a lightweight computer device can include a laptop computer, a smartphone, a tablet, a smartwatch, a smart speaker, a personal digital assistant device, or a personal computer. In a twentieth aspect, the lightweight sensing and data analysis apparatus can be configured so that when only a label is needed for evaluation of the collected undcrsamplcd measurements, the lightweight computer device is utilized and otherwise, the at least one remote processing element is utilized. In some embodiments, the utilization of the at least one remote processing element can be configured to preserve more information via latent representations learned in the network.

[0037] In some embodiments, the positional encoding scheme can be a fixed seed process. Other embodiments may utilize other types of positional encoding schemes.

[0038] Other details, objects, and advantages of the invention will become apparent as the following description of certain present preferred embodiments thereof and certain present preferred methods of practicing the same proceeds.

[0039] BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Exemplary embodiments of our apparatus and method for evaluation of sensor data and embodiments of methods of making and using the same are shown in the accompanying drawings. It should be appreciated that like reference numbers used in the drawings may identify like components.

[0041] Figure 1 is a block diagram of a first exemplary embodiment of an apparatus for evaluation of sensor data.

[0042] Figure 2 is a block diagram of a second exemplary embodiment of an apparatus for evaluation of sensor data.

[0043] Figure 3 is a schematic view of a machine learning model architecture (e.g. neural network, etc.) that can be implemented in the exemplary embodiment of the apparatus for evaluation of sensor data shown in Figures 1 and 2.

[0044] Figure 4 is a graph illustrating test accuracy with a percentage of undersampling that was obtained from testing of an exemplary embodiment of our apparatus for evaluation of sensor data from a bearing fault data set having a driving end shaft speed of 1730 revolutions per minute (rpm) and a fault diameter of 0.007 inches (0.1778 mm).

[0045] Figure 5 is a graph illustrating test accuracy with a percentage of undersampling that was obtained from testing of an exemplary embodiment of our apparatus for evaluation of sensor data using a bearing fault data set having a driving end shaft speed of 1797 revolutions per minute (rpm) and a fault diameter of 0.021inchcs (0.5334 mm).

[0046] Figure 6 is a graph illustrating test accuracy with a percentage of undersampling that was obtained from testing of an exemplary embodiment of our apparatus for evaluation of sensor data having a shaft unbalance at a shaft speed of 17 Hz.

[0047] Figure 7 is a first exemplary confusion matrix plot for the bearing fault data set having the driving end shaft speed of 1730 revolutions per minute (rpm) and a fault diameter of 0.007 inches (0.1778 mm). For Figure 7, class 0 is normal, class 1 is outer race defect, class 2 is a ball defect and class 3 is an inner race defect.

[0048] Figure 8 is first exemplary confusion matrix plot for the bearing fault data set having the driving end shaft speed of 1797 revolutions per minute (rpm) and a fault diameter of 0.021 inches (0.5334 mm). For Figure 8, class 0 is normal, class 1 is outer race defect, class 2 is a ball defect and class 3 is an inner race defect.

[0049] Figure 9 is a first exemplary confusion matrix plot for the sensor data having the shaft unbalance at a shaft speed of 17 Hz. For Figure 9, class 0 is normal, class 1 is unbalanced shaft data.

[0050] Figure 10 is a graph illustrating undersampled sensing based regression data in which there was a less than 15% prediction error for a signal having two frequencies and also noise using an exemplary embodiment of our apparatus for evaluation of sensor data for evaluation of sensor data from synthetically generated signals having 2-5 frequencies each, in the range of 20- 20,000 Hz, with a sampling frequency of 44,100 Hz.

[0051] Figure 11 is a graph illustrating undersampled sensing based regression data in which there was a less than 15% prediction error for a signal having three frequencies and also noise using an exemplary embodiment of our apparatus for evaluation of sensor data for evaluation of sensor data from synthetically generated signals having 2-5 frequencies each, in the range of 20- 20,000 Hz, with a sampling frequency of 44,100 Hz.

[0052] Figure 12 is a graph illustrating undersampled sensing based regression data in which there was a less than 15% prediction error for a signal having four frequencies and also noise using an exemplary embodiment of our apparatus for evaluation of sensor data for evaluation of sensor data from synthetically generated signals having 2-5 frequencies each, in the range of 20- 20,000 Hz, with a sampling frequency of 44,100 Hz. Figure 13 is a graph illustrating undersampled sensing based regression data in which there was a less than 15% prediction error for a signal having five frequencies and also noise using an exemplary embodiment of our apparatus for evaluation of sensor data for evaluation of sensor data from synthetically generated signals having 2-5 frequencies each, in the range of 20- 20,000 Hz, with a sampling frequency of 44,100 Hz.

[0053] Figure 14 is a graph illustrating a loss function from training performed on an exemplary embodiment of a neural network for an exemplary embodiment of an apparatus for evaluation of sensor data. The training was performed using a simulated signal with a frequency and noise. The trained embodiment was then utilized for predictions using real-world vibration calibration signal data.

[0054] Figure 15 is a graph illustrating the evaluation of a real -world multi-modal signal from a vibration calibration signal of 159 Hz performed by the embodiment of the apparatus for evaluation of sensor data trained as indicated in Figure 14. The median for the evaluation was a signal of 144.5 Hz in this embodiment’s use of undersampling of the sensor data to evaluate the data.

[0055] Figure 16 is a graph illustrating the loss function for training performed on an exemplary embodiment of a neural network for an exemplary embodiment of an apparatus for evaluation of sensor data. The training was performed using a simulated signal with a frequency and noise. The trained embodiment was then utilized for predictions using real- world sound calibration signal data.

[0056] Figure 17 is a graph illustrating the evaluation of a real-world multi-modal signal from a sound calibration signal of 945 Hz performed by the embodiment of the apparatus for evaluation of sensor data trained as indicated in Figure 16. The median for the evaluation was a signal of 947.7 Hz in this embodiment’s use of undersampling of the sensor data to evaluate the data.

[0057] Figure 18 is a flow chart illustrating a first exemplary embodiment of a process for forming a selected portion of sensor data that is an undersampling of the collected sensor data that can be utilized in the embodiments of the apparatus for evaluation of sensor data shown in Figures 1-2.

[0058] Figure 19 is a flow chart illustrating a first exemplary embodiment of a process for defining a neural network or other type of machine learning model that can be run on a computer device 3 or remote computer device 20 for evaluation of the selected portion of the sensor data (e.g. the undersampled portion of the sensor data) that can be utilized in the embodiments of the apparatus for evaluation of sensor data shown in Figures 1-2.

[0059] Figure 20 is a chart illustrating results from conducted testing.

[0060] Figure 21 is a graph illustrating computational improvements that were obtained in conducted testing.

[0061] Figure 22 is a graph illustrating computational improvements that were obtained in conducted testing.

[0062] Figure 23 is a graph illustrating classified, or labeled, data that was synthesized for use in undersampling of that data for conducted testing and evaluation.

[0063] Figure 24 is a graph illustrating a first twenty windows for window selection utilized during the testing conducted based on the data of Figure 23.

[0064] Figure 25 is a graph illustrating classified, or labeled, data that was synthesized for use in undersampling of that data for conducted testing and evaluation.

[0065] Figure 26 is a graph illustrating a first four windows for window selection utilized during the testing conducted based on the data of Figure 25.

[0066] Figure 27 is a graph illustrating unclassified, or unlabeled, data that was synthesized for use in undersampling of that data for conducted testing and evaluation.

[0067] Figure 28 is a graph illustrating unclassified, or unlabeled, data that was synthesized for use in undersampling of that data for conducted testing and evaluation.

[0068] Figure 29 is a graph illustrating a variation of test accuracy with percent data used (% data used) for the CWRU bearing fault dataset for different experimental testing work that was conducted using this dataset.

[0069] Figure 30 is a confusion matrix for the CWRU dataset for different experimental testing work that was conducted using this dataset.

[0070] Figure 31 is a graph illustrating accuracy that was observed in the transfer learning on a 4-class classification problem and a 10-class classification problem for conducted experimental work.

[0071] Figure 32 is a graph illustrating the variation of test accuracy with % data used for the experimental work that utilized a machinery fault simulator (MFS) dataset.

[0072] Figure 33 is a graph illustrating the variation of test accuracy with % data used in connection with UCI HAR dataset experimental work. Figure 34 is a graph illustrating variation of test accuracy with % data used in connection with PAMAP2 Physical Activity Monitoring dataset experimental work.

[0073] Figure 35 is a graph illustrating a variation of test accuracy with % data used for 12 experiments done varying the number of frequencies.

[0074] Figure 36 is a graph illustrating inference results utilizing an embodiment of a trained device in experimental work conducted on sound data.

[0075] Figure 37 is a graph illustrating inference results utilizing an embodiment of a trained device in experimental work conducted on vibration data.

[0076] Figure 38 is a flow chart illustrating second exemplary embodiment of a process for forming a selected portion of sensor data that is an undersampling of the collected sensor data that can be utilized in the embodiments of the apparatus for evaluation of sensor data shown in Figures 1-2.

[0077] DETAILED DESCRIPTION

[0078] Referring to Figures 1-3, an apparatus 1 for evaluation of sensor data can be configured as a telecommunication apparatus that includes one or more communication devices or computer devices communicatively connected to other devices via at least one network 13. For example, there can be one or more sensors 9 that are communicatively connected to a computer device 3 within a first network 13a. Each sensor 9 can be a temperature sensor, humidity sensor, pressure sensor, proximity sensor, level sensor, accelerometer, gyroscope sensor, gas sensor, infrared sensor, microphone, sound sensor, vibrational sensor, light sensor, radio frequency sensor, distributed fiber optics sensing (DFOS) device, Human Activity Recognition (HAR) device, thermal radiation sensor, ultrasound sensor, or other types of sensor.

[0079] The first network 13a can be an enterprise network, a local area network, a wireless local area network, or other type of network. The computer device 3 can be communicatively connected to a remote computer device 20 (e.g. a cloud based server, an array of cloud based servers that host at least one service, etc.). The remote computer device 20 can be included within a second network 3b (e.g. a local area network, a wide area network, etc.). The communicative connected between the computer device 3 and the remote computer device 20 can include a third network 13c. The third network can be the internet or other very large wide area network, for example. In some embodiments, the third network 13c can be an intermediary network between the first and second networks 13a and 13b that can all be components of a larger fourth network 13d. For example, the first, second, and third networks can be different inter-connected local area networks of a larger enterprise network 13d. In such a situation, the first and second networks 13a and 13b can be communicatively connected via the fourth network 13d and the third network 13c (shown in broken line) may not be needed or used.

[0080] The computer device 3 can include a processor 4 connected to a non-transitory computer readable medium 6 and at least one transceiver unit 5. The non-transitory computer readable medium 6 can be a type of non-transitory memory (e.g. solid state drive, hard drive, flash memory, etc.). The non-transitory computer readable medium 6 can have at least one application 7 stored thereon that has code that can be run by the processor 4. The code can define at least one method that is performed by the computer device 3 when the processor 4 runs the code of the application 7. There can also be one or more other data stores 2 on the non-transitory computer readable medium 7 (e.g. at least one database, files, etc.).

[0081] The computer device 3 can also be communicatively connected to one or more input device 8a and / or one or more output devices 8b. Such devices can also include an input / output device, such as a touch screen display. Example of input devices 8a can include pointer devices, keyboards, buttons, a microphone, or a touch screen display. Example of output devices 8b can include a printer, a speaker, a display, or a touch screen display. Other types of input / output devices that can be communicatively connected to the computer device 3 can include a laptop computer, smartphone, tablet, or personal computer that may be communicatively connected to the computer device 3 (e.g. via an application programming interface (API), local area network connection, and / or other type of communicative connection).

[0082] The remote computer device 20 can be a workstation, server, or array of servers. In some embodiments, the remote computer device 20 can be structured as a computer device 3 that includes a processor 4 connected to a non-transitory computer readable medium 6 and at least one transceiver unit 5. The non-transitory computer readable medium 6 can be a type of non- transitory memory (e.g. solid state drive, hard drive, flash memory, etc.). The non-transitory computer readable medium 6 can have at least one application 7 stored thereon that has code that can be run by the processor 4. The code can define at least one method that is performed by the computer device 3 when the processor 4 runs the code of the application 7. There can also be one or more other data stores 2 on the non-transitory computer readable medium 7 (e.g. at least one database, fdes, etc.).

[0083] The remote computer device 20 can also be communicatively connected to one or more input device 8a and / or one or more output devices 8b. Such devices can also include an input / output device, such as a touch screen display. Example of input devices 8a can include pointer devices, keyboards, buttons, a microphone, or a touch screen display. Example of output devices 8b can include a printer, a speaker, a display or a touch screen display. Other types of input / output devices that can be communicatively connected to the remote computer device 20 can include a laptop computer, smartphone, tablet, or personal computer that may be communicatively connected to the remote computer device 20 (e.g. via an application programming interface (API) and / or other type of communicative connection).

[0084] In some embodiments, the computer device 3 can be remote from the one or more sensors to which it can be communicatively connected. An example of such an arrangement is shown in Figure 1. In other embodiments, the computer device 3 can be included within a sensor 9. An example of such an arrangement is shown in Figure 2. In such an embodiment, the computer device 3 can be integrated with the sensor 9 and the sensor can be communicatively connected to the remote computer device 20 via at least one network. In some situations, the sensor 9 in such an embodiment can be in the first network 13a and be communicatively connected to the remote computer device 20 that is positioned in a second network 13b via another network 13 (e.g. the internet, an enterprise network having the first and second networks 13a and 13b as local area network components to that larger network, etc.).

[0085] In situations where the sensor and computer device 3 are integrated or the sensor 9 and computer device 3 are separate devices that are communicatively connected, the computer device 3 can be configured to collect and store the sensor data. A selected portion of this sensor data 9 can then be selected for analysis at the computer device 3 or be sent to the remote computer device 20 for further evaluation. This selected portion can be a randomized undersampling of all the received sensor data that is between 5% and 50% of the total sensor data obtained from the sensor 9 (e.g. in some situations, it may be greater than 5% of the data and less than 15% of the data, in other situations it may be more than 10% of the data and not more than 50% of the sensor data, in yet other situations it may be greater than 8% of the data and less than or equal to 30% of the sensor data, etc. ). The selected portion of the sensor data to be evaluated can be considered an undcrsampling of the data because it utilizes significantly less than an entirety of the sensor data obtained by the one or more sensors within a pre-selected time period (e.g. within a time period of 2 minutes, 5 minutes, an hour, a day, a week, or other suitable time period that accounts for the parameter(s) to be evaluated by the data collected from the sensor(s)).

[0086] An exemplary undersampling process that can be utilized to select the undersampled portion of the sensor data can be appreciated from Figures 18 and 38.

[0087] As may be seen from Figure 38, in some embodiments of a process for selection and / or use of a portion of sensor data to be evaluated can include using a random sampling scheme to generate positional encoding for the sensor data. A random distribution scheme can then be utilized based on a desired percentage or portion of undersampling that is to be performed. For example, if only 10% or between 10%-30% of the sensor data is to be utilized in the undersampling, a random distribution scheme can be utilized based on such a desired level of undersampling of the sensor data (e.g. use of less than 50% of the sampling data, etc.).

[0088] Then, the dataset (X) can be sampled according to the indices of the random distribution scheme. Depending on the type of problem to be addressed via the undersampled data and analysis of that data, a decision can then be made as to whether to preserve phase information and / or amplitude scaling associated with the data.

[0089] Thereafter, a series of labels can be generated corresponding to the undersampled dataset for training of the neural network or form making an inferenced determination about the sensor data. Alternatively, a machine learning algorithm of a machine learning model can be trained for subsequent deployment as the trained machine learning model for implementation of the undersampling process for data inference processing without labeling being generated.

[0090] After a machine learning model has been sufficiently trained to provide a desired or preselected accuracy level for undersampling of data to make inferenced determinations based on the undersampled data, the code of the machine learning model can be deployed for implementation by one or more devices running the deployed code of the trained machine learning model.

[0091] For example, in some embodiments of a process for selection of the portion of the sensor data to be evaluated (as shown in Figure 18), the process can include fixing the seed for generating random numbers in a first step (wherein the fixing of the seed is selected for the positional encoding scheme) and generating random numbers based on a pre-selected desired undcrsampling amount (c.g. a desired percent, or desired %, of undcrsampling) in a second step. The dataset of sensor data from the pre-selected time period can then be sampled according to the indices of the Gaussian random samples generated via the random sampling scheme of the second step. In other embodiments, another type of random sampling scheme can be utilized instead of Gaussian.

[0092] Depending on the type of problem, one or more parameters of interest, phase information and / or amplitude scaling may be preserved from the sampled data set. In some embodiments, labels can then be generated corresponding to the undersampled dataset. The labels can also be used in training of a neural network or to continue training of a neural network after the neural network has been defined and undergone initial training. In other embodiments, labels may not be needed and training can occur without use of labels (e.g. in an unsupervised manner, for example).

[0093] In embodiments where the remote computer device 20 receives the selected portion of the sensor data, an application 7 can define a neural network or other type of machine learning model that may be run to evaluate the portion of the sensor data received from the computer device 3. In embodiments where the computer device 3 integrated into the sensor 9 or communicatively connected to the sensor 9 can evaluate the selected portion of the sensor data, the application 7 of the computer device 3 can define a neural network or other type of machine learning model that can evaluate the selected portion of the sensor data. An example of the machine learning model architecture defined by the application 7 or code stored in the non- transitory computer readable medium of the computer device can be appreciated from Figures 3 and 19.

[0094] For instance, the code of the application 7 can define multiple layers of a fully connected neural network that can utilize the undersampled portion of the sensor data as input to the neural network for evaluation. A LeakyReLu or tanh function can be defined as the activation function between layers of the network. The hyperparameter including the number of iterations, regularization of parameters, learning rate, number and depth of layers, and the activation function can be determined for defining the neural network and training it. A desired number of regression variables at an output layer of the neural network can be defined for a regression problem to be solved via the evaluation of sensor data to be performed by the neural network. A cross-entropy loss for the required number of classes can be defined for a classification problem to be solved via the evaluation of sensor data to be performed by the neural network. An Adam optimizer and square root of LI -loss for a loss function can be defined as well for defining the neural network evaluation to be performed.

[0095] As may best be appreciated from Figure 3, the neural network can be defined so that there are a number of layers (e.g. four layers of the example of Figure 3). The randomized sampling of sensor data can be selected from a Gaussian randomization process as shown in Figure 3 for providing as input to an input layer of the defined neural network. The network may then process that data through various layers to perform a classification evaluation of the data based on pre-defined classifications. For a regression evaluation, the layers can be defined to utilize the selected random sampled portion of the sensor data to evaluate the sensor data.

[0096] The output layer of the neural network can be defined to output a classification of the preselected classification options based on the performed evaluation of the undersampled sensor data that was processed in accordance to the defined parameters (e.g. as defined per the process of Figure 19, for example, etc.).

[0097] The neural network can be defined to evaluate the undersampled portion of the sensor data to provide a task-driven sensing process that can be focused on obtaining accurate representations of the sensor data from the undersampled measurement data of the undersampled sensor data to solve different downstream tasks rather than focusing on acquiring high-fidelity data for accurate sensor data reconstruction. In such approaches, the neural network can be configured for evaluation of the undersampled sensor data to perform a classification function based on the sensor data or perform a regression evaluation from the undersampled sensor data. The utilized sensor data can be multi-modal temporal sensor data from a pre-selected time period or other type of sensor data from a pre-selected time period that is not the entirety of the sensor data collected in that time period.

[0098] In embodiments where the data may be unlabeled, the data can undergo clustering to define different classes of data, or clusters of data, for use in labeling the data based on the results of the clustering. After the data is clustered and labeled, the data can be further evaluated via undersampling for training of a defined neural network using the labeling of data defined via the clustering. Any type of 1-D sensor data can be evaluated via embodiments of our process and system. The trained machine learning model may then be deployed for being run on other devices to evaluate sensor data via the undcrsampling process.

[0099] Testing we have conducted has confirmed that embodiments can be provided so that a trained machine learning model can perform undersampling to provide accurate evaluations of the undersampled data, as discussed further below.

[0100] Figure 3 illustrates an example of a defined neural network as an example of a machine learning model that can utilize undersampling. The exemplary embodiment of the machine learning model shown in Figure 3 is configured as a shift- invariant undersampled network (SIUN) to preserve signal information from highly undersampled (e.g. very few) sensor measurements. An exemplary architecture of the SIUN is described in Figure 3 and was defined via the exemplary neural network formation process shown in Figure 19. The training of the SIUN that was performed in the testing showed that any type of undersampling of data can be used for training of the SIUN via labeled data or unlabeled data regardless of whether the data is periodic type data or non-periodic type data, and provide substantial computation efficiency improvements in training the SIUN for accurate classification of the data.

[0101] For instance, the architecture of the SIUN is shown in Figure 3, highlighting the implementation of undersampling, positional encoding, and the feed forward block on different problems like classification, anomaly detection, transfer learning, and regression. The weights and biases are learned using backpropagating errors by formulating appropriate loss functions: Cross entropy loss for classification, and LI loss (Ll)1 / x, x = 2n, n = positive integer, for regression. Adam optimizer was used along with Tanh activation function for classification and Leaky ReLu activation function for regression.

[0102] The SIUN was configured to introduce the following architectural ideas to preserve (1) shift invariance, and (2) spectral stability via utilization of: (i) windowing to preserve local information, and (ii) random seed-based sampling on each window with the data sampled at Nyquist frequency. Other embodiments may utilize another type of random sampling approach instead of a seed based sampling approach. Embodiments of the SIUN was configured to convert real-time sensing into learning problems like classification or regression, making it applicable to most sensor data analysis tasks. For the SIUN utilized in the experimental work discussed below, we utilized a random seed-based sampling at Nyquist rate to preserve signal amplitudes at specific instances of time, as shown in Fig. 3. This helped form the basis for separating one function from another and one signature from another.

[0103] The embodiment of the SIUN was configured to infer a periodic structure from an undersampled set of data as compared to contemporary approaches that require use of 100% of data. A neural network was utilized for the SIUN instead of other machine learning models for modelling as they (i) give a general differentiable learnable mapping that can be fine-tuned for different tasks (like classification and regression) by changing the type of loss function and hyperparameters, (ii) can keep improving with larger datasets, and (iii) the same architectural ideas can be extended to other sensing modalities like images and videos which have redundant data. However, other machine learning models can be utilized in other embodiments (e.g. SVM, kNN, logistic regression, etc.).

[0104] As noted above, we evaluated an exemplary embodiment of our apparatus 1 configured as the SIUN for evaluation of sensor data to evaluate how an embodiment could perform and the type of advantages it may be able to provide. Other embodiments can be provided that can also provide similar types of benefits. The experimental work for an embodiment configured as the SIUN is non-limiting and provides examples of the type of advantages and benefits different embodiments of our process and apparatus can provide.

[0105] EXAMPLES

[0106] In a first set of experiments, we performed classification studies on undersampled sensor measurements for multiple datasets spanning several different experimental conditions. The standard Case Western Reserve University (CWRU) bearing fault dataset is the first dataset we selected for use. We have done more than 144 experiments to get comprehensive experimental results by varying all the parameters in the Case Western Reserve University bearing fault dataset. Figures 4-9 and 29-31 provide examples of different individual instances of those 144 experiments that were obtained by varying (i) complexity of the network, and (ii) training dataset size, out of the many other variable parameters.

[0107] For some of our experiments included in a first set of experiments, we evaluated the exemplary embodiment on two different experimental conditions. These were (i) Driving end shaft speed = 1730 rpm and a fault diameter of 0.007 inches, and (ii) Driving end shaft speed = 1797 rpm and a fault diameter of 0.021 inches. This dataset had data pertinent to four classes of operation: Normal, Ball defect, Inner Race defect, and Outer Race defect. In a second set of experiments, we utilized a vibration-based dataset as a second dataset. The second dataset included sensor data for evaluation of shaft unbalance (11 grams) generated using SpectraQuest’s Machinery Fault (SMF) Simulator.

[0108] For the first dataset (the bearing fault dataset), the neural network of the embodiment was created to perform a multi-class classification evaluation with four classes: Normal, Ball defect, Inner Race defect, and Outer Race defect.

[0109] For the experimental results shown in Figure 29-31, the 12 KHz sampling frequency and Drive end dataset of the CWRU dataset was utilized with three different faults (ball defect, inner race defect, and outer race defect centered at 6:00) at 3 fault diameters (0.007 inches, 0.014 inches, 0.021 inches) are considered as 9 faulty classes. This resulted in a 10-class classification problem with 1 normal class and 9 faulty classes. The first 90,000 data points of each of the 10 classes were used for training and the next 30,000 data points were used for testing, resulting in a 75-25 train-test split. A window length of 2000 was used, with overlapping windows used to create lx, lOx, 20x, and 40x data augmentation in training set. The experiments were repeated four times. The confusion matrix of Figure 30 shows that there was no class imbalance in the results of the experimental results shown in Figures 29 and 31.

[0110] We also performed a set of experiments using the University of California Irvine Human Activity Recognition (UCI HAR) dataset. This dataset includes human activity recognition data collected from 30 subjects while performing daily activities of living including walking, walking upstairs, walking downstairs, sitting, standing, and laying. The UCI HAR dataset consists of different train and test folders of sizes 192 and 77 MB respectively. The sampling frequency of channels in this dataset is 50 Hz, and windows of length 2.4 seconds are used, creating windows of 120 data points each. Overlapping windows are used to create lx, 2x, 5x, and lOx data augmentation in the train set. The experiments are repeated ten times and results from this experimental work is shown in Figure 33.

[0111] Another set of experiments were conducted on the Physical Activity Monitoring (PAMAP2) dataset, which consists of 18 classes of activities (including 6 optional activities) performed by 9 subjects. The dataset has 54 channels of data, out of which 51 are used in this work to evaluate SIUN performance. The sampling frequency of channels in this dataset is 100 Hz, and windows of length 1 second are used, creating windows of 100 data points each. In this dataset, windowing is done prior to 70-30 split and no overlapping windows are used. This method is used in this dataset and not others, as it has activities from each individual captured sequentially. If wc split the dataset into 70-30 before windowing, we will only show certain classes to the train set and certain classes to the test set. Overlapping windows are not used even in the train set to avoid data leakage into the test set. The experiments were repeated three times. Results from this experimental work are shown in Figure 34.

[0112] Another set of experiments are performed on a machinery fault simulator dataset, wherein the experimentation was performed and the data was collected by the first author. A shaft unbalance was created by adding a weight of 11 grams on the machinery fault simulator to study the normal and unbalanced fault signatures. Each experiment was 20 seconds of data sampled at 4096 Hz for 2 classes - normal, and shaft unbalance, amounting to a total of 4 MB of data for training the network / experiment. A window length of 400 is used, with overlapping windows used to create lx, 5x, lOx, and 20x data augmentation in the train set. The experiments were repeated five times. A 75-25 train-test split is used in this study. Figure 32 shows the results from this experimental work.

[0113] For the classification experimental work discussed above having the results shown in Figures 29-35, we used 64x32x16, 64x16, 50x40, 30x30 architectures for CWRU, UCI HAR, PAMAP2, and MFS datasets respectively.

[0114] Another set of experiments that were conducted using the shaft unbalanced dataset generated via the SMF simulator. For this set of experiments, the neural network was defined to solve a binary classification problem with an objective to detect an unbalance in the rotating shaft.

[0115] The following Figures 4-6 illustrate some results of this experimentation.

[0116] As can be appreciated from Figures 4-6, we performed the evaluation in these experiments by use of neural networks that were defined as “simple”, “medium” and “complex” involving “more data” and “less data.” The “less data” referred to a neural network that was trained with less training data and the “more data” referred to a neural network that was trained with more data.

[0117] For these different tests that were performed, Figure 4 illustrates the identified test accuracy with percent undersampling for the bearing fault dataset having driving end shaft speed = 1730 rpm and a fault diameter of 0.007 inches. Figure 5 illustrates the identified test accuracy with percent undersampling for the bearing fault dataset having driving end shaft speed = 1797 rpm and a fault diameter of 0.021 inches. Figure 6 illustrates the identified test accuracy with % undcrsampling for the dataset having a shaft unbalance at a shaft speed of 17 Hz.

[0118] As can be seen from Figures 4-6, the test accuracy was found to be highly accurate after less than 60% of the sensor data was utilized for most embodiments regardless of the complexity of the neural network. In some situations, the accuracy was found to be quite high when less than 20% of the sensor data was used (e.g. Figures 4-6).

[0119] Further, Figure 29, and Figures 31-34 show that accuracy of 90% or higher can be provided by use of only 10%-20% of the sensor data.

[0120] The graphs of Figures 4-6, 29, and 31-34 are the maximum accuracy achieved over a number of different experimental runs. The reason for choosing maximum plots is that this permitted the testing to be conducted with defined neural network architecture that performs the best for a given percentage of undersampling. An empirical observation is that the network perfomis the best between 10-21 % of the total sensor data based on this particular set of tests that were performed.

[0121] Moreover, as seen from Figures 4-6, 29, and 31-34, the highest accuracy achieved depended on the size of training data, rather than percentage undersampling, unlike in an information theoretic approach.

[0122] We also conducted experimental work to evaluate how an embodiment of the SIUN could be perform regression tasks. For the regression experimental work, we used 100 100x2-5, 200x200x2-5, and 400x400x2-5 architectures to predict 2-5 frequencies in a broad range of 20 Hz - 20 kHz.

[0123] Figure 35 shows the results of 12 experiments done by varying number of frequencies (4 frequencies - 2, 3, 4, and 5) and SIUN architecture (100x100, 200x200, 400x400). These results showed that (i) the embodiment of the SIUN reached 80%+ accuracies for some experiments with just 20% undersampling, and (ii) there were minor performance improvements with the amount of raw data collected. The embodiment of the SIUN trained on simulated data predicts the frequency of actual undersampled sound and vibration calibration signals as shown in Figures 36 and 37 respectively, indicating that the requisite information was preserved even though there was significantly less data utilized via the undersampling.

[0124] For these experiments, the sound calibration signal had a frequency of 945 Hz and the vibration calibration signal had a frequency of 159 Hz. The results of Figures 35-37 show that the trained embodiment of the STUN was effective in preserving the necessary information from undcrsamplcd data for solving regression tasks.

[0125] Based on the conducted testing, we hypothesized that the latent representation learned in the defined neural network for classifying into four classes could also be able to perform binary classification. To test this, we collapsed the four classes of the bearing fault classification problem into two classes: (i) Normal, and (ii) Abnormal, and investigated if transfer learning can be used here in additional testing. As per the following Table 1, we found that binary classification performed after transfer learning provided a good separation between the two classes of interest, thus, showing that latent representations learned from undersampled measurements preserved information of interest.

[0126] Table 1. Transfer learning accuracy for binary classification

[0127] To study the integrity of embodiments of the neural network evaluation process for undersampled sensor data for multi-class classification, we plotted the confusion matrix for all the datasets used in the above noted testing. Those confusion plots are shown in Figures 7, 8, and 9. Figure 7 illustrates the confusion matrix for the CWRU bearing dataset for the driving end shaft speed = 1730 rpm and a fault diameter of 0.007 inches and Figure 8 illustrates the confusion matrix or the CWRU dataset for the driving end shaft speed = 1797 rpm and a fault diameter of 0.021 inches. These datasets were used as the first dataset in the above noted testing. Tn Figures 7 and 8, the classes are defined such that class 0 is normal data, class 1 is outer race defect, class 2 is a ball defect, and class 3 is an inner race defect.

[0128] Figure 9 illustrates the confusion matrix for the shaft unbalanced dataset of the second dataset. In Figure 9, the classes are defined such that class 0 is normal and class 1 is unbalanced shaft data.

[0129] We also performed regression testing to evaluate how embodiments can evaluate sensor data from different sensor modalities. For the regression testing, we evaluated sound and vibration as sensor modalities. For this regression testing, we synthetically generated signals having 2-5 frequencies each, in the range of 20-20,000 Hz, with a sampling frequency of 44,100 Hz. Our objective was to study if the embodiment of our apparatus 1 configured to utilize a STUN could preserve frequency information when trained on undersampled measurements. Figures 10-13 shows the results of undersampled sensing-based regression using SIUN.

[0130] Figure 10 illustrates the results from the fraction of test data having less than 15% prediction error for a signal having 2 frequencies + noise. Figure 11 illustrates the results from the fraction of test data having less than 15% prediction error for a signal having 3 frequencies + noise, Figure 12 illustrates the results from the fraction of test data having less than 15% prediction error for a signal having 4 frequencies + noise. Figure 13 illustrates the results from the fraction of test data having less than 15% prediction error for a signal having 5 frequencies + noise.

[0131] Again, we found that embodiments could perform very well with a significant undersampling of sensor data. In many cases, undersampling of only 8%-30% of the sensor data was needed for the embodiment to provide a highly accurate evaluation of the signal being evaluated.

[0132] We also conducted testing concerning the impact training of the defined neural network that may perform the evaluation of the undersampled portion of the sensor data may have on the performance of embodiments of our apparatus. In this evaluation, we utilized synthetically created sensor data for the training to also evaluate whether the synthetic (e.g. non-real world data) could be sufficient to provide adequate training evaluation of the data to permit an inference on actual sound and vibration calibration signals. We performed this testing because we believed it would show that embodiments can utilize neural networks that are defined to learn the temporal relationship between different data points and would be able to identify frequencies independent of the windows (shift-invariant). As can be appreciated from Figures 14-17, this was found to be correct.

[0133] Figure 14 shows the type of simulated training utilized to train a defined neural network.

[0134] The signal and model characteristics for the training was a signal with 1 frequency with noise that was simulated. As shown in Figure 15, the trained neural network provided a median determination of 144.5 Hz for this 159 Hz vibration calibration signal.

[0135] Figure 16 shows the type of simulated training utilized to train a defined neural network. The signal and model characteristics for the training was a signal with 1 frequency with noise that was simulated. As shown in Figure 16, the trained neural network provided a median determination of 947.7 Hz for this 945 Hz sound calibration signal.

[0136] Figure 20 quantifies the computational and engineering efficiency of the embodiments of our apparatus that were utilized in our testing. Undersampling as a phenomenon directly lead to shallow networks, which are computationally efficient as quantified below. The insight from Table 2 can apply to both classification and regression problems lending to lightweight compute mechanisms that can be utilized for use in conjunction with undersampled sensing-based learning (USBL) to provide sufficiently accurate sensor data evaluations at substantially lower computational costs and bandwidth costs to permit improved monitoring and evaluation of devices and processes. Figures 21 and 22 further illustrate the substantial reduction in computational complexity, time, and cost, as obtained from the conducted testing and analysis of embodiments of our system and process.

[0137] To further study the integrity of embodiments of the neural network evaluation process for undersampled sensor data for multi-class classification in situations where the data was nonperiodic (e.g. temperature data, pressure data, etc.), synthesized sensor data as shown in Figure 23 for temperature data having multiple different classifications, or labels - Normal, Fault 1, Fault 2, and Fault 3 - were used. These classes are defined based on their different slope characteristics, (or signatures). Each of the data instances generated a slightly different signal based on different amplitudes, SNR, slope, etc. The labeled non-periodic data included 1000 windows for each type for the conducted training. Each defined window included 100 points sampled at 1 Hz for the 1000 windows / class set of data, which resulted in a total of 4000 windows being evaluated. Figure 24 illustrates the first 20 windows of each class of data to further illustrate the windows. We performed undersampling, train-testing and model training to evaluate how the trained network using the undersampled data would then perform in classifying the data. Table 2 illustrates the results of the evaluation, which shows high accuracy was obtained with a use of undersampling (15% to 8% of all the overall data being used in the undersampling).

[0138] Table 2; Results from undersampling of classified data of Figures 23-24

[0139] To further study the integrity of embodiments of the neural network evaluation process for undersampled sensor data for multi-class classification in situations where the data was nonperiodic (e.g. temperature data, pressure data, etc.), synthesized sensor data as shown in Figure 25 for temperature data having multiple different classifications, or labels -Normal, Fault 1, and Fault 2 - were used. These classes are defined based on their different slope characteristics, (or signatures). Each of the data instances generated a slightly different signal based on different amplitudes, SNR, slope, etc. The labeled non-periodic data included 1000 windows for each type for the conducted training. Each defined window included 100 points sampled at 1 Hz for the 1000 windows / class set of data, which resulted in a total of 4000 windows being evaluated. Figure 24 illustrates the first 20 windows of each class of data to further illustrate the windows. We performed undersampling, train-testing and model training to evaluate how the trained network using the undersampled data would then perform in classifying the data. Table 3 illustrates the results of the evaluation, which shows high accuracy was obtained with a use of undersampling (15% to 8% of all the overall data being used in the undersampling). Table 3; Results from undersampling of classified data of Figures 25-26

[0140] We also conducted evaluations and testing on sensor data that may be unlabeled, or unclassified. The evaluated data was also non-periodic data. Examples of synthesized and unlabeled data that were used in the conducted testing is shown in Figure 27 and 28. Figure 27 is a graph illustrating a snapshot of 20 unlabeled time series windows having non-periodic signals. Figure 28 a graph illustrating a snapshot of 10 unlabeled time series windows having non-periodic signals of a different type than the data of Figure 27.

[0141] A lot of sensor data that may be collected in different situations or environments can be unlabeled data. For example, there may be 300 channels of data collected per flight via airplane sensors, with each channel having 50,000 time windows. There may be 10,000 such assets deployed at any given point in time, leading to billions of unlabeled time windows for analysis. We utilized the data of Figures 27 and 28 to evaluate how embodiments that may utilize our undersampling based approach could account for unlabeled sensor data that utilized a multiple step approach: (i) create clusters using unsupervised techniques, and (ii) implement an embodiment of our SIUN on this new labeled dataset which was created by using the clusters created from (i) as labels.

[0142] In the conducted testing, the creation of clusters for use in allocation of labels was performed by converting each window into a cluster using different unsupervised techniques: k- means, mean shift, spectral clustering, DBSCAN, and HDBSCAN. We evaluated use of each of these techniques and found that they all worked well, but that HDBSCAN provided the most accurate clusters for the data. For the data of Figure 27, there were four clusters, or labels defined for the data (c.g. labels of type 1, type 2, type 3, and type 4). For the data of Figure 28, there were three clusters, or labels of data defined for the data (e.g. labels of class 0, class 1, and class 2). After selection of the clusters for labeling of the data, the labeled data was used for training of the embodiment of the SIUN via use of undersampled data as discussed above (e.g. only 10% of data points in each window was used for training). This was performed for the data of Figure 27 and was also performed in a separate test using the data of Figure 28 based on the classifications, or labels, obtained via the clustering. The testing accuracy that was obtained after training for both conducted testing scenarios of Figures 27 and 28 was 100%, meaning the embodiment of the trained SIUN classified each of the test windows of data correctly.

[0143] The conducted testing showed that embodiments can operate on undersampled sensor measurement data to solve different downstream tasks like classification and regression. Further, the testing showed that embodiments can define neural networks that can be trained using simulated, or synthetic data that can permit the trained neural network to evaluate real- world sensor data accurately with improved computational efficiency (faster and lesser computational complexity) while also substantially reducing bandwidth, storage, and subsequent computing requirements. For instance, substantially less sensor data is needed for the evaluation of the sensor data to be performed to classify a condition or provide an evaluation of the condition being measured or monitored (e.g. only 5% to 20% of the collected data may be utilized in the undersampling in some embodiments). Further, testing showed use of undersampling can be used for all kinds of data, including periodic data, non-periodic data, labeled data, and non-labeled data.

[0144] We also performed experimental work to compare the perfomiance (test accuracy) and model complexity of an exemplary embodiment of the SIUN with a state-of-the-art Convolutional Neural Network (CNN). Within model complexity, we calculated the number of parameters and floating-point operations for each of the models. A detailed comparison is provided in Table 4 below. Test accuracy was used as a metric to quantify the number of correct class predictions for classification, and to measure the fraction of test data having <15% prediction error for regression, to account for noise in the signals.

[0145] Confusion matrices were used to check for any class imbalance in the model predictions. Different configurations of a feed-forward network are used for different classification and regression problems. For the CWRU dataset, the SIUN achieved 96.0% test accuracy on just 30% of the raw data. A LcNct5-bascd CNN achieved 99.77% accuracy on 100% of the raw data, as reported in literature. The SIUN achieved a 435.01x reduction in the number of FLOPS required while only being just 3.77% lower in accuracy.

[0146] For the UCI HAR dataset, the SIUN achieved 90.67% accuracy on just 20% of the raw data. The CNN achieved 92.71% accuracy on 100% of the raw data as reported in literature. The SIUN used 26.84x less FLOPS than the CNN, while only being 2.04% lower in accuracy.

[0147] For the PAMAP2 dataset, the SIUN achieved 91.1% test accuracy onjust 10% of the raw data. A CNN achieved 91.0% accuracy on 100% of raw data as reported in literature. The SIUN used 7.97x less FLOPS while being 0.1% more accurate than the CNN.

[0148] For the MFS dataset, the SIUN achieved 100.0% test accuracy onjust 20% of the raw data. A CNN benchmark was not available for the MFS dataset as this dataset was created by the inventors and there are no published results for a conventional CNN on such data.

[0149] Table 4; Results from undersampling of classified data

[0150] We also performed an evaluation of the SIUN by benchmarking the SIUN for edge computing use-cases on a Nvidia Jetson Nano, an Intel i7, and a Raspberry Pi 3. Model training and transfer learning on the edge (Jetson Nano) was only possible when we used SIUN (10-50% of raw data). Both capabilities are unobtainable for some use-cases if we used greater than 50% of the raw data, due to memory and computation restrictions. Our conducted benchmark work showed that training times increased non-linearly with the greater percentage of data used in the training. For example, the Nano runs out of resources when training for greater than 50% of the data was used. The benchmarking work we conducted showed that a reduction in bandwidth of at least lOx could be obtained for data transmission in training of a model when the SIUN was utilized. This benchmarking showed that embodiments can permit small processing capacity devices to train and / or utilize a machine learning model while the data sets for training could be significantly reduced in size.

[0151] Embodiments of the SIUN utilized in the experimental work were configured to preserve shift invariance and spectral stability; which can be two primary requirements for time and frequency domain analysis of temporal signals. Different windows with the same infomiation when passed through SIUN should give the same output, e.g. we wanted position independent sampling). To do this for embodiments of the SIUN utilized in the experimental work, we first decided the window length based on the sensor data characteristics in order to maintain the stationarity of data within the window and then undersampled each window using a random sampling scheme. Preserving spectral stability can help make sure that the machine learning model is neither selective to low nor high frequency signals. In training the SIUN, both low and high frequency signals (between 20 Hz and 20 kHz) were utilized to train the SIUN on individual and composite signals with up to 5 frequencies and noise. Differential data augmentation for training on varying numbers of frequencies was used to make sure that the SIUM was equally sensitive to low or high frequency signals. Spectral stability was demonstrated as shown in Figure 35, indicating that the embodiment of the SIUN utilized in the experimental work was not selective for lower or higher frequencies and can preserve spectral information in temporal or spatial signals.

[0152] Embodiments can be utilized in numerous different types of settings. Embodiments can be utilized in conjunction with drone operation, industrial processing, and various manufacturing processes or devices. Embodiments can also be utilized in conjunction with vehicle sensors and other types of sensor and detector arrangements.

[0153] Embodiments of our apparatus and method can permit substantial reduction in computational complexity and manifold computing speed-ups. Embodiments can also provide lightweight data collection, transmission, storage, cloud computing, and analytics. The cascading effects of all these advantages can be very prominent in terms of cost and complexity when deployed in conjunction with a network or system that may utilize many machines (100s or 1000s of machines, for example). For instance, embodiments can be employed in situations where sensors are used for continuous remote monitoring (e.g. high-Capex industrial machines: wind turbines, gas turbines, jet engines, utility machinery, power generation, deployments, ocean awareness monitoring, agri-tech, and consumer goods production, etc.). As another example, embodiments can be utilized in conjunction with the outer space technologies industries (e.g. low-cost custom satellites (small satellite market), logistics, market research, etc.), specifically Radio Frequency-based satellite data analysis and the Low Earth Orbit (LEO) economy. Embodiments can be utilized to provide near real-time capabilities while also providing immense savings to satellite-based edge compute, transmission, and other data transmission pipeline- related advantages (e.g. bandwidth advantages). In yet other embodiments, the monitoring industrial emissions or wildfires can be provided with improved operational efficiency.

[0154] Embodiments can provide both (or at least one of): (i) computational efficiencies and (ii) pipeline-based advantages. In some embodiments, the advantages can be substantial (e.g. a 10X or more than 10X reduction in bandwidth, storage, and computational processing requirements).

[0155] Embodiments can be configured to utilize a selective learning approach to sensing, where the amount of data collected is problem dependent to analyze large amounts of sensor data. The amount of sensor data to be collected for a class of problems can be a learnable property of an exemplary embodiment of the apparatus (e.g. the SIUN utilized in the experimental work, etc.). Embodiments can provide a general trainable model that can be configured to preserve shift invariance and spectral stability for temporal sensor data. This architecture can successfully solve problems formulated as classification and regression. Reducing the amount of data collected significantly reduces the number of nodes needed for data representation and computation, which can permit lightweighting of the device (e.g. utilization of significantly less processing power and electricity).

[0156] Also, our experimental results show that performance improvements are observed with data augmentation (larger training datasets) rather than by collecting more than a certain fraction of raw data. Embodiments can be configured for utilization in conjunction with more quickly processing and addressing multiple sensing problems in various domains, such as satellite data analysis, drones and Unmanned Aerial Vehicles (UAVs), manufacturing condition monitoring, underwater monitoring, and other embedded Al applications. For example, embodiments of the SIUN significantly reduced the required computation (flops), power, storage, memory (RAM), and bandwidth required for different sensor applications. Some of the considerations in actual deployment can include implementation on bare-metal devices to help provide micro to milli second level prediction latencies, reducing the number of flops to reduce power consumption.

[0157] Embodiments of our process and apparatus can provide a selective learning-based machine learning model architecture that can be implemented to provide significant improvements in power, compute, storage, transmission, and latency for data-driven decision making utilized in a wide variety of different machine applications. The experimental results highlight the significance of problem-dependent sampling for efficient sensing can be provided for accurate interpretation of the data that can avoid collecting all the raw data. Embodiments can be implemented in a hardware agnostic manner so that embodiments can be provided for electronic, photonic, or other computer hardware modalities.

[0158] It should be appreciated that modifications can be made to the above discussed embodiments to meet a particular set of design criteria. For instance, the specific type of undersampling portion used, the type of randomization used to form the undersampling portion, and the number of nodes used for the neural network processing, and the amount of training and type of training (use of just simulated signals, use of real world and simulated data, use of only real world data, etc.), can be adapated to meet a particular set of design criteria.

[0159] It should therefore be understood that while certain present preferred embodiments of our apparatus and method for the evaluation of sensor data and methods for making and using the same have been shown and described above, it is to be distinctly understood that the invention is not limited thereto but may be otherwise variously embodied and practiced within the scope of the following claims.

Claims

What is claimed is:

1. An apparatus for evaluation of sensor data, the apparatus comprising: a computer device integrated with and / or communicatively connectable to at least one sensor to receive sensor data from the at least one sensor, the computer device having a processor connected to a non-transitory computer readable medium and at least one transceiver unit; the computer device configured to undersample the sensor data received from the sensor for evaluation of the undersampled sensor data.

2. The apparatus of claim 1, wherein the computer device is also configured to evaluate the undersampled sensor data via a neural network or machine learning model defined in code of the non-transitory computer readable medium.

3. The apparatus of claim 1 , wherein the computer device is also configured to communicatively connect to a remote computer device for sending the undersampled sensor data to the remote computer device, the remote computer device configured to evaluate the undersampled sensor data via a neural network or machine learning model defined in code of a non-transitory computer readable medium of the remote computer device.

4. The apparatus of claim 1, wherein the at last one sensor is at least one of: a temperature sensor, a humidity sensor, a pressure sensor, a proximity sensor, a level sensor, an accelerometer, a gyroscope sensor, a gas sensor, an infrared sensor, a microphone, a sound sensor, a vibrational sensor, a power sensor, radio frequency sensor, distributed fiber optics sensing (DFOS) device, Human Activity Recognition (HAR) device, thermal radiation sensor, ultrasound sensor, or a light sensor.

5. The apparatus of claim 1, wherein the computer device is an edge device.

6. The apparatus of claim 1, wherein the computer device is integrated with the at least one sensor.

7. The apparatus of claim 1, wherein the computer device is a remote computer device.

8. The apparatus of claim 1, wherein the computer device configured to undersample the sensor data by forming a randomization portion of the sensor data that includes less than or equal to 60% of the sensor data, less than or equal to 40% of the sensor data, or between 5% and 20% of the sensor data.

9. The apparatus of claim 1, wherein the computer device configured to undersample the sensor data via a randomization process, Gaussian randomization process or a unifomi randomization process defined in code stored in the non-transitory computer readable medium.

10. A method for evaluation of sensor data comprising: collecting sensor data from at least one sensor; selecting a portion of the sensor data for evaluation via undersampling of the collected sensor data; and at least one of: transmitting the selected portion of the sensor data for the evaluation of the sensor data; and / or evaluating the selected portion of the sensor data.

11. The method of claim 10, wherein the evaluating of the selected portion of the sensor data comprises: feeding the selected portion of the sensor data to an input layer of a neural network defined in code of a non-transitory computer readable medium of a computer device for evaluation of the sensor data.

12. The method of claim 11, wherein the computer device is a computer device communicatively connected to the at least one sensor.

13. The method of claim 11 , wherein the computer device is integrated with the at least one sensor.

14. The method of claim 11, wherein the computer device is remote from an edge device that receives the sensor data from the at least one sensor and performs the selecting of the portion of the sensor data, the edge device also performing the transmitting of the selected portion of the sensor data for the evaluation of the sensor data, the transmitting of the selected portion of the sensor data for the evaluation of the sensor data comprising the edge device transmitting the selected portion of the sensor device to a remote computer device, the remote computer device performing the evaluating of the selected portion of the sensor data.

15. The method of claim 11, wherein the evaluating of the selected portion of the sensor data comprises: classification of a condition based on the selected portion of the sensor data; or performing a regression analysis of the selected portion of the sensor data to determine a condition or parameter.

16. The method of claim 11, wherein the portion of the sensor data that is selected for evaluation via undersampling of the collected sensor data is less than or equal to 60% of the collected sensor data, less than or equal to 40% of the collected sensor data, or is between 5% and 20% of the collected sensor data; and wherein the method also comprises performing a randomization for the selected portion of the sensor data for evaluation of the selected portion of the sensor data.

17. A lightweight sensing and data analysis apparatus, comprising:(i) an edge device for collecting undersampled measurements, and one of:(ii) a lightweight computer device on the edge device using shift-invariant undersampled networks, or(iii) the edge device configured for transmitting the collected undersampled measurements based on a positional encoding scheme to at least one remote processing element configured to: receive randomly sampled data from the edge device;store the randomly sampled data in a data store; run a shift-invariant undcrsamplcd network to perform a classification or regression analysis.

18. The lightweight sensing and data analysis apparatus of claim 17, wherein: when only a label is needed for evaluation of the collected undersampled measurements, the lightweight computer device is utilized; and otherwise, the at least one remote processing element is utilized.

19. The lightweight sensing and data analysis apparatus of claim 18, wherein the utilization of the at least one remote processing element is configured to preserve more information via latent representations learned in the network.

20. The lightweight sensing and data analysis apparatus of claim 17, wherein the positional encoding scheme is a fixed seed process.

Images