Method, apparatus and system for a monitoring device

Abstract

The present disclosure relates to methods, apparatuses, systems, and articles, including a monitoring system that includes an image sensor for obtaining image data of a device and a regulator for causing the image sensor to obtain the image data of the device, forming an impression from the image data, and using the impression and the image data to determine a ruling.

Description

[0001] Related Applications

[0002] This application is a continuation of U.S. Patent Application No. 15 / 718,874, filed Sep. 28, 2017, which is incorporated herein by reference in its entirety. Technical Field

[0003] This disclosure generally relates to monitoring, and more particularly to methods, apparatuses, and systems for monitoring devices. Background Art

[0004] Conventional monitoring systems typically rely on the analysis of alerts and event streams from sensors to infer abnormal conditions that have manifested. However, mechanical failures may present an indirect and diffuse correlation to behavioral parameters over a period of time. As a result, mechanical failures may escape detection until a moment such as severe degradation occurs. Brief Description of the Drawings

[0005] Figure 1 is a schematic diagram of an example device for monitoring systems and structures implemented in an example vehicle constructed in accordance with some teachings of the present disclosure.

[0006] Figure 2 is Figure 1 a block diagram of an example implementation of a regulator 1150 of

[0007] Figures 3A - 3C illustrates an example implementation of an example regulator to which an example image sensor provides example image data in accordance with some teachings of the present disclosure Figure 1 of

[0008] Figures 4A - 4B illustrates example image data from an example image sensor of Figures 3A - 3C at a first moment and at a second moment, respectively, in accordance with some teachings of the present disclosure.

[0009] Figures 5A - 5B presents a flowchart representation of computer-executable instructions that may be executed to implement Figures 1 - 2 an example regulator of

[0010] Figure 6 illustrates a representation of example image data from an example image sensor of Figures 1 - 2 an example regulator of Figures 5A - 5B in accordance with example instructions of Figure 1 and / or Figures 3A - 3C of

[0011] Figure 7 illustrates for an example regulator of Figures 1 - 2 to be used to from from Figure 1and / or Figures 3A - 3C An example training model that forms an example impression from example image data of an example image sensor.

[0012] Figure 8 Illustrated by Figures 1 - 2 An example implementation by an example regulator of Figures 5A - 5B example instructions on example image data from an example image sensor of Figure 1 and / or Figures 3A - 3C An example implementation of example image data of an example image sensor.

[0013] Figure 9 Is a block diagram of an example processor platform that can execute Figures 5A - 5B instructions of Figures 1 - 2 to implement an example regulator.

[0014] These figures are not drawn to scale. As used in this patent, reciting any component (e.g., layer, film, region, or plate) as being positioned in any way on (e.g., positioned on, located on, disposed on, or formed on, etc.) another component indicates that the recited component is in contact with the other component, or that the recited component is above the other component and one or more intermediate components are between the recited component and the other component. Reciting any component as being in contact with another component means that there are no intermediate components between the two components. Detailed Description

[0015] Conventional monitoring systems may not be able to detect the degradation of mechanical systems. In a manually operated system (such as a vehicle with a driver), vehicle sensing data is supplemented by humans who can notice subtle changes in vehicle performance, unusual noises, unusual odors, etc., and can assess the need for further investigation. Such conventional monitoring systems use data streams from system sensors (e.g., engine speed, vibration, tire pressure, etc.) as indicators, but these data streams sometimes prove insufficient to detect problems or indicate the severity of problems. Autonomous systems similarly receive data streams from internal sensors and analyze the data streams to draw inferences therefrom. This monitoring paradigm and analysis apply to simple degradation mechanisms. For example, a low tire pressure indicator is sufficient to alert for low tire pressure. However, various mechanical fault mechanisms may prove difficult to diagnose via on-vehicle sensors. For example, a blown cylinder head gasket may manifest in multiple ways (e.g., misfires, reduced compression, overheating, swelling of the radiator cap, corrosion of fluids, oil leaks, coolant leaks, white exhaust, etc.).

[0016] In accordance with some teachings of the present disclosure, autonomous devices (e.g., autonomous land vehicles, autonomous aerial vehicles, drones, robots, spacecraft, industrial equipment or machinery, etc.) and / or manually operated devices (e.g., ground vehicles, aircraft, ships, etc.) implement example regulators to monitor one or more systems, subsystems, or components to evaluate damage and / or degradation of the device / equipment (including any (multiple) systems, (multiple) subsystems, or (multiple) components). The example regulators help eliminate human in-the-loop manual screening of autonomous devices and help increase the autonomy of the devices.

[0017] Figure 1 FIG. 4 is a schematic diagram of an example system 100 for monitoring systems and structures implemented in an example device 110, which can be autonomous or manually operated. Although Figure 1 the example of FIG. 4 depicts the device 110, the teachings herein equally apply to other types of autonomous devices and / or manually operated devices, such as those described above.

[0018] As Figure 1 shown, the device 110 includes one or more example image sensors 120 (hereinafter referred to as "image sensors 120") in one or more regions of the vehicle (e.g., one or more regions of the engine compartment, (multiple) electric motors, chassis, braking system, etc.). In some examples, the image sensors 120 include thermal image sensors, spatial image sensors, and / or optical image sensors. Figure 1 It is also shown that the device 110 includes one or more example sensors 125 (e.g., one or more pressure sensors, one or more vibration sensors, one or more speed sensors, and / or one or more acceleration sensors, etc.) in one or more regions of the vehicle to provide telemetry data for one or more systems or subsystems of the device 110.

[0019] The image sensors 120 are used to obtain example image data of the (multiple) regions of the device 110 in which the image sensors 120 are disposed, such as thermal image data, spatial image data, and / or optical image data. In some examples, the image sensors 120 may include a non-contact MLX90620 temperature measurement device from Melexis, Belgium, which includes a 16×4 element far-infrared (FIR) thermopile sensor array configured to generate a real-time map of heat values. In some examples, the image sensors 120 include, as RealSense TM (RealSense TM ) spatial image sensor of the Depth Module D400.

[0020] The image sensor 120 outputs image data to an example regulator 150 via an example communication path 130 (such as a hardwired communication path or a wireless communication path). In some embodiments, the regulator 150 is disposed within the device 110. For example, the regulator 150 may be disposed within the dashboard, under the seat, or in the trunk of the device 110. In some examples, the regulator 150 is disposed at a remote location (e.g., outside the device 110, in a different area from the device 110, etc.). As described below, the regulator 150 processes the image data from the image sensor 120 and outputs the image data and / or its derivatives to an example RF broadcast tower 160 and / or an example network 165 via a communication device 155.

[0021] In some examples, the communication device 155 includes devices such as a transmitter, transceiver, modem, and / or network interface card to facilitate the exchange of image data with one or more external machines 170 (e.g., any type of computing device, computer, server, etc.) via the RF broadcast tower 160 and / or the network 165. In some examples, the communication device 155 may communicate with the network 165 directly or indirectly (e.g., via one or more intermediate devices) via an Ethernet connection, digital subscriber line (DSL), telephone line, coaxial cable, cellular phone system, 10Base-T connection, FireWire connector, or universal serial bus (USB) connector. Thus, although an example RF broadcast tower 160 and an example communication device 155 are indicated in Figure 1 the example, in some examples, the example regulator 150 is connected to one or more external machines 170 via a hardwired connection (e.g., a USB connection).

[0022] Figure 2 is Figure 1 a block diagram of an example implementation of the regulator 150. In Figure 2 the example implementation, the regulator 150 includes an example image manager 210, an example impression manager 220, and an exemplary impression comparator 230.

[0023] Typically, the example regulator 150 is used to cause the image sensor 120 to obtain example image data of the structure of the device 110 at a first moment, and an example impression can be formed from the image data. The impression facilitates a comparison between the actual data from the sensor data and the data calculated as a result of applying a trained model or impression to previous data. For example, in a first example, where the previous data samples at times t-1, t-2, … t-N are known and the example regulator 150 receives sensor data at time t, the regulator 150 applies the impression to the data samples at times t-1, t-2, … t-N and uses the impression to calculate the estimated data for time t. The regulator 150 then compares the estimated data for time t with the actual data at time t and determines a ruling as to whether the comparison is favorable (e.g., “good” state) or unfavorable (e.g., “bad” state). In a second example, where the previous data samples at times t-1, t-2, … t-N are known and the example regulator 150 receives sensor data at time t, the regulator 150 applies the impression to the data samples at times t, t-1, t-2, … t-N and determines a ruling as to whether the comparison is favorable (e.g., “good” state), unfavorable (e.g., “bad” state), or “unknown”. The impression can be continuously updated after each data sampling, periodically or aperiodically. For example, in some examples, the regulator 150 updates the impression periodically using batch data. In some examples, the regulator 150 updates the impression when new classifications become available in response to telemetry data or an external expert system. The impression allows the regulator 150 to determine whether the measured data is close to the predicted data, where the regulator 150 calculates a ruling (e.g., “good” state or “bad” state) by comparing the predicted data with the measured data (e.g., neural network “feed-forward” evaluation) and modifies or updates the impression to incorporate input data that the regulator 150 was previously not familiar with (e.g., neural network “backpropagation” or training).

[0024] The example image manager 210 is used to receive example image data from the image sensor 120 and process the example image data, and is used to pass the image data to the example impression manager 220 for processing. In some examples, the example image manager 210 obtains image data of the structure of the device 110 in response to a request from the regulator 150.

[0025] The example impression manager 220 is used to use the image data to form an impression of the structure of the device 110 imaged by the image sensor 120 or a trained model. In some examples, values corresponding to the image data obtained by the image sensor 120 are assigned to each volume and / or surface area in the field of view or the viewpoint of the image sensor 120. In some examples, the impression can be a set of weights in a matrix representing a neural network (e.g., an artificial neural network (ANN), a recurrent neural network (RNN), a convolutional neural network (CNN), a deep neural network (DNN), etc.) or some other representation of behavior (e.g., a "decision tree", a support vector machine (SVM), a logistic regression (LR), etc.). The impression is trained (e.g., updated matrix coefficients, etc.) using the data from the image sensor 120. For example, a thermal image sensor will output image data including temperature readings at each point or pixel in the field of view (e.g., a 640x480 thermal image sensor can include 307,200 pixels), and a spatial image sensor will output the distance at each point or pixel in the field of view (e.g., the point cloud of a 3D image sensor, etc.) (e.g., via time-of-flight). The point cloud data for temperature and / or distance is brought into a common reference frame (e.g., a polar coordinate system, a Cartesian coordinate system, etc.). The result of the training is a modified impression that accommodates or integrates the valid variations in the image data.

[0026] In some examples, the impression can include an arrangement of raw data (such as the temperature in a volume or surface area defined by a selected coordinate system) or a derived quantity of the raw data (such as a spatial thermal gradient map within a given volume and / or a mapping of the spatial thermal gradient onto a coefficient vector (e.g., a Radon transform)). The example impression manager 220 can use any expression of the image data to uniquely identify the imaging operation state. For example, the impression manager 220 can convert the image data into an alternative representation via a mathematical transform, a linear transform, a matrix representation, a linear mapping, an eigenvalue decomposition, a wavelet decomposition, a geometric multiscale analysis, a polygonal 3D model, a surface model, a non-uniform rational basis spline (NURBS) surface model, a polygonal mesh, and store and / or export the representation in an appropriate format (e.g., as a standard tessellation language (STL) file, a standard ACIS text (SAT) file, and / or an OBJ geometry file or any other 3D modeling file format).

[0027] In some examples, an initial impression is provided by the manufacturer of the device 110 (e.g., a vehicle, etc.), and the initial impression is updated using the image data from the image sensor 120.

[0028] In some examples, the example impression comparator 230 is used to apply an impression to samples of image data at times t-1, t-2, …… t-N, and is used to calculate the estimated image data for time t. Then, the impression comparator 230 compares the estimated image data for time t with the actual image data from the imager 120 for time t to determine the level of correspondence between the estimated image data for time t and the actual image data at time t. Then, the impression comparator 230 makes a determination as to whether the actual data corresponds to a "known good" state or a "known bad" state. In some examples, the example impression comparator 230 is used to apply an impression to samples of image data at times t, t-1, t-2, …… t-N, and is used to calculate the estimated image data for the determination at time t (e.g., "known good" state, "known bad" state, "unknown" state, etc.).

[0029] In some instances, the impression manager 220 (e.g., via a Radon transform, a Hough transform, a Funk transform, a combination of transforms, etc.) maps a spatial gradient and / or a thermal gradient onto a coefficient vector to form and / or update an impression, and the impression comparator 230 is used to compare basis vectors in a vector space between sets of image data at different times to determine the correspondence between the detected image data and a known state (e.g., a good state, a bad state, etc.) and / or an unknown state.

[0030] In some examples, the conditioner 150 outputs an example impression and / or image data related to the impression, or a derivative thereof, to the example memory 250. In the example operation state manager 252 corresponding to the operating state of the device, the example memory 250 includes an example impression 254, and the example impression 254 includes image data related to the impression 254 and / or a derivative thereof. In some examples, the operation state manager 252 differentiates between multiple operation states (e.g., one or more "known good" states, one or more "known bad" states, etc.). The operation state manager 252 also includes an example first image data set 256 and image data sets subsequent to the example first image data set 256 up to the example Nth image data set 258, where N is any integer. In some examples, the first image data set 256 and / or another image data set up to the Nth image data set 258 includes more than one image data set (e.g., multiple image data sets from multiple different times). In some examples, the first image data set 256 is pre-loaded into the memory 250 by the vendor of the device 110.

[0031] In some examples, the memory 250 is local to the device 110. In some examples, the memory 250 is remote from the device 110, and communication between the conditioner 150 and the memory 250 is via the communication device 155 and / or via the communication device 155 and any intermediate devices (such as, the RF broadcast tower 160 and / or the network 165).

[0032] Although Figure 2 illustrates an example manner of implementing Figure 1 the conditioner 150, Figure 2 one or more of the elements, processes, and / or devices shown in summary may be combined, split, rearranged, omitted, eliminated, and / or implemented in any way. Further, the example image manager 210, the example impression manager 220, and / or the example impression comparator 230, and / or more generally, Figure 2 the example conditioner 150 may be implemented by hardware, software, firmware, and / or any combination of hardware, software, and / or firmware. Thus, for example, any one of the example image manager 210, the example impression manager 220, and / or the example impression comparator 230, and / or more generally, the example conditioner 150 may be implemented by one or more analog or digital circuits, logic circuits, programmable processors, application specific integrated circuits (ASICs), programmable logic devices (PLDs), and / or field programmable logic devices (FPLDs). When any one of the apparatus or system claims of this patent covers a pure software and / or firmware implementation, at least one of the example image manager 210, the example impression manager 220, and / or the example impression comparator 230, and / or more generally, the example conditioner 150 is hereby expressly defined to include a non-transitory computer-readable storage device or storage disk containing software and / or firmware, such as a memory, a digital versatile disc (DVD), a compact disc (CD), a Blu-ray disc, and so on. Further still, Figure 1 the example conditioner 150 may include one or more elements, processes, and / or devices additional to or in place of Figure 2 those shown, and / or may include more than one of any or all of the elements, processes, and devices shown.

[0033] In Figures 5A - 5B is shown a flowchart representing example machine-readable instructions for implementing Figure 2 the conditioner 150. In this example, the machine-readable instructions include instructions for being executed by a processor (such as in conjunction with Figure 9The program may be embodied in software stored on a non-transitory computer-readable storage medium such as a CD-ROM, floppy disk, hard drive, digital versatile disk (DVD), Blu-ray disk, or memory associated with the processor 912, but the entire program and / or portions thereof may alternatively be executed by a device other than the processor 912, and / or embodied in firmware or dedicated hardware. Further, although reference is made to Figures 5A - 5B The flowchart illustrated in describes an example program, but many other methods of implementing the example regulator 150 may be used instead. For example, the order of execution of the blocks may be changed, and / or some of the blocks described may be changed, eliminated, or combined. Additionally or alternatively, any or all of the blocks may be implemented by one or more hardware circuits (e.g., discrete and / or integrated analog and / or digital circuits, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), comparators, operational amplifiers (op-amps), logic circuits, etc.) that are structured to perform corresponding operations without executing software or firmware.

[0034] As mentioned above, the present invention may be implemented using encoded instructions (eg, computer and / or machine readable instructions) stored on a non-transitory computer and / or machine readable medium. Figures 5A - 5B Example process, the non-transient computer and / or machine readable medium such as hard disk drive, flash memory, read-only memory, compact disk, digital versatile disk, cache, random access memory and / or any other storage device or storage disk in which information is stored for any duration (e.g., in an extended time period, permanently, for a brief instance, for temporary buffering and / or for caching of information). As used herein, the term non-transient computer readable medium is explicitly defined as including any type of computer readable storage device and / or storage disk, and excludes propagation signals and excludes transmission media. "Include" and "include" (and all forms and tenses thereof) are used as open terms in this article. Therefore, whenever a claim lists any content following any form of "include" or "include" (e.g., including, including, etc.), it is understood that additional elements, items, etc. may exist without exceeding the scope of the corresponding claim. As used herein, when the phrase "at least" is used as a transitional term in the preamble of a claim, it is open in the same manner as the terms "include" and "include".

[0035] Figure 3AAn example image sensor 302 disposed in an example engine compartment structure 304 is shown. The example image sensor 302 includes an example laser 310 powered by a power source (not shown), and the example laser 310 is arranged to emit an example light beam of collimated light 315 at an example photoelectric sensor 320. In some examples, the photoelectric sensor 320 includes a photodiode. The laser 310 may include any laser of any wavelength (e.g., any wavelength between 650 - 1550 nm), and may include, for example, a diode laser or a semiconductor laser. In some examples, the image sensor 302 includes an Adafruit VL53L0X time-of-flight distance sensor. In some examples, the power source for the laser 310 includes a battery of a device (e.g., an autonomous and / or manually operated device, device 110, etc.) that implements the image sensor 302. The image sensor 302 and the photoelectric sensor 320 form part of an optical circuit, where photons of the incident light beam of collimated light 315 are converted into a current representing the alignment between the laser 310 and the photoelectric sensor 320. A change in the intensity of the incident light beam of collimated light 315 on the photoelectric sensor 320 may indicate a shift of a part of the example engine compartment structure 304 relative to another part of the engine compartment structure 304.

[0036] In some examples, the image sensor 302 includes a plurality of image sensors 302 that produce a vector of measured values, where each image sensor among the image sensors 302 produces a scalar value combinable over time as a tuple on an n-vector, where n is any integer.

[0037] Figure 3BAn example image sensor 302 disposed in an example engine compartment structure 325 is shown. In some examples, the example image sensor 325 is a time-of-flight image sensor, such as, but not limited to, a range-gated image sensor, a direct time-of-flight image sensor, or a sonic ranger. The example image sensor 325 includes an example laser 310 powered by a power source (not shown), the example laser 310 being arranged to emit an example beam of collimated light 315 at an object 330. The example image sensor 325 further includes a photosensor 320 for receiving reflected light from the object 330 in response to an incident beam of collimated light 315. In some examples, the light sensor 320 includes a collection lens disposed proximate to the laser 310, the collection lens being arranged to focus incident light traveling thereto onto a solid-state photodiode (e.g., a linear array camera, a CMOS array, etc.). The laser 310 may include any laser of any wavelength (e.g., any wavelength between 650 - 1550 nm), and may include, for example, a diode laser or a semiconductor laser. In some examples, the power source for the laser 310 includes a battery of a device (e.g., an autonomous and / or manually operated device, device 110, etc.) implementing the image sensor 302. The image sensor 302 and the photosensor 320 form part of an optical circuit where photons of the incident light 332 are converted into an electric current representing the distance between the image sensor 325 and the object 330, which in turn represents the alignment between the image sensor 325 and the object 330. A change in the intensity of the incident light 332 on the photosensor 320 and / or a phase shift of the incident light 332 may indicate a displacement of a portion of the example engine compartment structure 304 relative to another portion of the engine compartment structure 304. A combination of distance data between multiple points enables the development of a computational geometry of the components.

[0038] Figure 3C An example image sensor 302 disposed in an example engine compartment structure 340 is shown. In some examples, the example image sensor 340 is a light detection and ranging (LIDAR) device or a 3D camera including an example laser scanner 345 and / or a light source. In some examples, the laser scanner 345 is arranged to emit a scanning beam 351 of collimated light across a selected volume of the structure of the device (e.g., the volume within the engine compartment of the device 110, etc.) to generate a point cloud or a depth map of distance information using a phase shift in the return signal to the laser scanner 345. For example, Figure 3CIllustrated is a first example cone 360 of reflected light incident from a first example object 370 of the structure of an autonomous and / or manually operated device, a second example cone 362 of reflected light incident from a second example object 372 of the structure of the autonomous and / or manually operated device, and a third example cone 364 of reflected light incident from a third example object 374 of the structure of the autonomous and / or manually operated device, in response to illumination of an image sensor 340.

[0039] The reflected light from the first example object 370, the second example object 372, and the third example object 374 interacts with one or more lenses and photoelectric sensors of the image sensor 340, which generates distance data for each point in the point cloud via time-of-flight. The image sensor 340 is used to create a map (e.g., a distance map and / or a thermal map) of a selected volume of the structure of the autonomous and / or manually operated device (e.g., a volume within an engine compartment, etc.), from which the regulator 150 can detect misalignments and temperature deviations. In some examples, the light source 350 includes one or more light emitters for illuminating a selected volume of the structure of the autonomous and / or manually operated device (e.g., a volume within an engine compartment, etc.). In some examples, the light source 350 includes solid-state diodes, lamps, and / or bulbs for outputting light within one or more wavelength ranges (e.g., the visible spectrum, the infrared spectrum, etc.). In some examples, the image sensor 340 includes RealSense TM (RealSense TM ) technology, such as, RealSense TM Depth Module D400 series image sensors.

[0040] In some examples, the impression manager 220 integrates or correlates image data from an image sensor 120 (e.g., image sensors 302, 325, 340) with telemetry data from one or more additional sensors 125 of the autonomous and / or manually operated device. For example, the image data used by the impression manager 220 to form and / or update an impression 254 can be correlated with data from one or more other sensors 125 (e.g., pressure sensors, vibration sensors, speed sensors, acceleration sensors, etc.), which are operatively associated with one or more systems or subsystems of the device (e.g., the autonomous and / or manually operated device, device 110). Thus, the impression comparator 230 is notified of changes in operating conditions, and the impression comparator 230 is able to contextually determine whether the decisions made indicate a changed operating condition or a potential fault.

[0041] Figures 4A - 4BIllustrated are example image data of one or more example image sensors 120 at one or more locations within a selected volume 405 from structures of autonomous and / or manually operated devices. In Figures 4A - 4B the example, the selected volume 405 includes an engine compartment of a device 110 (e.g., an engine-based vehicle), and the selected structure 410 includes an example engine. In Figures 4A - 4B for clarity, the engine cover of the device 110 has been removed. In some instances, one or more image sensors 120 are disposed on the underside of the engine cover to face the selected volume 405 and the selected structure 410.

[0042] Figure 4A represents example image data at a first moment, and Figure 4B represents example image data at a second moment. In Figure 4A a first set of components is shown as having a first temperature gradient 420, a second set of components is shown as having a second temperature gradient 430, and a third set of components (including example component 445) is shown as having a third temperature gradient 440, these temperature gradients being represented by different fill degrees. In the illustrated example, the second temperature gradient 430 is greater than the first temperature gradient 420, and the third temperature gradient 440 is greater than the second temperature gradient 430.

[0043] In Figure 4B a first set of components is shown as having a first temperature gradient 420, a second set of components is shown as having a second temperature gradient 430, and a third set of components is shown as having a third temperature 440.

[0044] However, compared with Figure 4A component 445 in the third set of components in Figure 4B is shown as having a fourth temperature gradient 450 that is higher than the third temperature gradient 440.

[0045] In Figures 5A - 5B and Figures 6 - 8 an example comparison of using example data of Figures 4A - 4B is described by way of example.

[0046] Figures 5A - 5B A program or instruction of Figure 5A begins with program 500 at example box 505 in Figure 5AAt example block 510, image manager 210 and / or regulator 150 receive N sets of telemetry data from sensor(s) 125. In some examples, at block 515, image manager 210 and / or regulator 150 may perform preprocessing on the image data from image sensor 120 and / or the telemetry data from sensor 125. The preprocessing can be used, for example, to suppress distortion in the data, eliminate noise in the data, enhance the data, and / or normalize the data. For example, image manager 210 may transform the image data into an appropriate coordinate system and convert it into a format suitable for further processing by image manager 210 and / or impression manager 220 (e.g., convert laser scanner data point by point into a raster model or other format acceptable for downstream processing, correct the gray values of the image data, perform edge detection and segmentation methods for identifying uniform regions, employ classification methods for classifying regions of the structure(s) represented by the image data, etc.).

[0047] Then, at Figure 5A example block 520, impression manager 220 is used to form impression 254 or update impression 254, and at example block 522 is used to store impression 254 in a physical, non-transitory storage medium. In some examples, impression manager 220 receives image data from Figure 1 image sensor 120 and / or Figures 3A - 3C image sensors 302, 325, and / or 340, and at block 520 converts the image data into impression 254, which includes an array of data vectors representing the image data for the selected volume 405 and / or the selected structure 410 (e.g., the volume of the space in the engine compartment of device 110, etc.). In some examples, when impression 254 is formed or updated at Figure 5A block 520, control passes back to block 505 to receive additional image data.

[0048] Figure 5B Example instruction 524 starting at example block 525 is shown, where image manager 210 of regulator 150 receives image data (e.g., a set of image data, N sets of image data, etc.) from image sensor 120 (e.g., 302, 325, 340). At block 530, image manager 210 and / or regulator 150 receive telemetry data (e.g., a set of image data, N sets of image data, etc.) from sensor(s) 125.

[0049] In some examples, at example block 535 and / or example block 540, the image manager 210 and / or the regulator 150 may perform preprocessing on the image data from the image sensor 120 and / or the telemetry data from the sensor 125, respectively. The preprocessing may be used, for example, to suppress distortion in the data, eliminate noise in the data, enhance the data, normalize the data, and / or transform the image data into an appropriate coordinate system and convert it into a format suitable for further processing (such as by the impression manager 220 and / or the impression comparator 230).

[0050] At example block 545, the impression comparator 230 applies the impression 254 to the image data. For example, the regulator 150 may access the impression 254 from the operation state manager 252 using the impression manager 220 and apply the impression 254 to the previous samples of the image data at times t-1, t-2,... t-N. As described above, the impression 254 may include, for example, a mapping of spatial gradients and / or thermal gradients onto a coefficient vector or a set of weights in a matrix representing a neural network (e.g., ANN / RNN / CNN / DNN, "decision tree", support vector machine, LR, etc.). At example block 545, the impression comparator 230 may also apply the impression 254 to calculate the estimated data for time t based on the impression applied to the previous samples of the image data at times t-1, t-2,... t-N. At block 550, the impression comparator 230 uses the impression and the image data, such as by comparing the impression 254 of the estimated data at time t with the actual data at time t, to determine a verdict. At block 555, the impression comparator 230 makes a determination as to whether the comparison of the impression 254 of the estimated data at time t with the actual data at t is favorable and reflects a good verdict (e.g., reflects a known "good" state of the device 110). In some examples, in the case where the previous samples of the data at times t-1, t-2,... t-N are known, at block 550, the impression comparator 230 applies the impression 254 to the data samples at times t, t-1, t-2,... t-N to determine a verdict, and at block 555, determines whether the verdict corresponds to a good verdict or a bad verdict.

[0051] If the result at block 555 is "yes", control passes to example block 560 where the regulator 150 and / or the impression comparator 230 determine whether the telemetry data from the sensor(s) 125 of the device 110 is within acceptable limits. If the result at block 555 is "no", control passes to example block 565 where the impression comparator 230 makes a determination as to whether the comparison of the estimated data at time t, impression 254, with the actual data at time t is adverse and reflects a poor verdict (e.g., reflects a known "poor" state of the device 110). In some examples, where previous samples of the data at times t-1, t-2, t-N are known, at block 555 the impression comparator 230 applies the impression 254 to the data samples at times t, t-1, t-2, t-N and at block 555 makes a determination as to whether the comparison is adverse and reflects a poor verdict.

[0052] If the result at block 565 is "no", control passes to example block 570 where the impression comparator 230 and / or the regulator 150 store the image data in the memory 250 for later evaluation. Then, control passes to example block 560 where, as discussed above, the regulator 150 and / or the impression comparator 230 determine whether the telemetry data from the sensor(s) 125 of the device 110 is within acceptable limits.

[0053] At block 560, if the telemetry data from the sensor(s) 125 of device 110 is within acceptable limits (i.e., the result is "yes"), then control passes to example block 575, where the impression comparator 230 and / or the regulator 150 updates the impression 254 to incorporate the image data as "good" (e.g., reflecting normal operating conditions, reflecting normal structural conditions, etc.). Then, control passes to block 525. If the telemetry data from the sensor(s) 125 of device 110 is not within acceptable limits and the result at block 560 is "no", then control passes to example block 580. At block 580, the impression comparator 230 and / or the regulator 150 updates the impression 254 to incorporate the image data as "bad" (e.g., reflecting abnormal operating conditions, reflecting abnormal structural conditions, etc.). Then, control passes back to block 585. At block 585, the impression comparator 230 and / or the regulator 150 stores the image data in the memory 250 for later evaluation. Then, control passes to example block 590. At block 590, the regulator 150 and / or the impression comparator 230 outputs a deviation report related to the impression 254 and / or the image data (e.g., reporting locally to the controller, reporting remotely to a central facility or server, etc.). If a particular deviation report (e.g., temperature at a particular location of the structure or system of the autonomously or manually operated device, displacement of the structure or system of the autonomously or manually operated device, etc.) is later positively correlated with a particular performance issue and / or maintenance issue, then the impression and / or the image data can be marked as a known problem impression. Particularly for multiple similarly configured devices 110 (e.g., a fleet of drones, a fleet of vehicles, etc.), a deviation report issued by one device 110 notifying of a potential and / or actual performance and / or maintenance issue can enable trend tracking of the problem, targeted preventive maintenance, and timely corrective actions not only for one device 110 but also for other similarly configured devices 110.

[0054] Figure 6 Illustrated is a representation of example image data 600 from Figures 1 - 2 image sensor 120 by regulator 150 according to Figures 5A - 5B the example flowchart of Figure 1 and / or Figures 3A - 3C . In Figure 6 , a series of input data vectors 610A - 610F, C L (t0) - C L (t5) represent image data output by image sensor 120 at five different times (i.e., t0 - t5) over a selected volume 405 (e.g., a 4x4 volume in the space within the engine compartment of device 110, etc.). In some examples, the input data vectors 610A - 610N, C L (t0) - CL (t N )(where N is any integer) is expanded into an example array 614 of example columns 616A - 616N and example rows 618A - 618N. Each column (e.g., 616A) and row (e.g., 618A) in the array 614 is represented by an example block 620 corresponding to the image data (e.g., temperature, etc.) of a part of the device 110, such as a selected volume, at a specific moment.

[0055] The first input data vector 610A represents the image data of the selected volume 405 at the first moment (t0). Each block 620 in the selected volume 405 has a uniform first temperature, which is expressed as a uniform filling in the first input data vector 610A of Figure 6 The second input data vector 610B represents the image data of the selected volume 405 at the second moment (t1). Each block 620 (except block 622) in the selected volume 405 has a first temperature, which is expressed as a uniform filling in the second input data vector 610B of Figure 6 Block 622 is shown to have a second temperature higher than the first temperature, which is expressed as a filling different from the filling of the first temperature. The third input data vector 610C represents the image data of the selected volume 405 at the third moment (t2). Each block 620 (except blocks 624 and 626) in the selected volume 405 has a first temperature, which is expressed as a uniform filling in the third input data vector 610C of Figure 6 Block 624 is shown to have a third temperature higher than the second temperature, which is expressed as a filling different from the filling of the second temperature. Block 626 is shown to have a second temperature higher than the first temperature, which is expressed as a filling different from the filling of the first temperature.

[0056] The fourth input data vector 610D represents the image data of the selected volume 405 at the fourth moment (t3). Some of the blocks 620 of the selected volume 405 have a first temperature, block 628 is shown as having a fourth temperature higher than the third temperature, and block 630 is shown as having the third temperature, where different fillings are used to represent each of the different temperatures. The fifth input data vector 610E represents the image data of the selected volume 405 at the fifth moment (t4). Some of the blocks 620 of the selected volume 405 are shown as having a first temperature, block 632 is shown as having the fourth temperature, and block 634 is shown as having a higher third temperature, where different fillings are used to represent each of the different temperatures. The box 636 of the fifth input data vector 610E is shown as having a second temperature. The sixth input data vector 610F represents the image data of the selected volume 405 at the sixth moment (t5). Some of the boxes 620 of the selected volume 405 are shown as having a first temperature, box 638 is shown as having the fourth temperature, and boxes 640 and 642 are shown as having the third temperature.

[0057] Figure 7 Illustrated by Figure 1 the regulator 150 for forming and / or updating the impression 254 for the device 110 from the image data of Figure 1 and / or Figures 3A - 3C the image sensor 120 of the example training model.

[0058] Figure 7 Shows an example input data vector (C L (t X )) of the device 110 over time, where the example first training sample 702 occurs at the first moment, the example second training sample 704 occurs at the second moment, and the example third training sample 706 occurs at the third moment. In some examples, multiple training samples are used to develop the impression 254 to take into account normal variations (such as variations in sensor measurements (e.g., sensor accuracy, etc.) and / or non-substantive variations in operating system performance).

[0059] In each of the first training sample 702, the second training sample 704, and the third training sample 704, rows 618F, 618G, and 618J include blocks 620 that exhibit a higher temperature than the remaining blocks 620, and the remaining blocks 620 are all at the first temperature 708. In the first training sample 702, row 618F shows that the block 620 at column 616A (C L (t - 3)) has a second temperature 710 higher than the first temperature 708, and the block 620 at column 616B (C L (t - 2)) has a third temperature 712 higher than the second temperature 710, and column 616C (CL (at time t-1)) block 620 has a fourth temperature 714 that is higher than a third temperature 712, and column 616N(C L (at time t)) block 620 is shown to have the fourth temperature 714. Row 618G of the first training sample 702 shows column 616A(C L (at time t-3)) block 620 has a first temperature 708, column 616B(C L (at time t-2)) block 620 has a second temperature 710, column 616C(C L (at time t-1)) block 620 has the third temperature 712, and column 616N(C L (at time t)) box 620 is shown to have the third temperature 712. Row 618J of the first training sample 702 shows that at column 616N(C L (at time t)) box 620 has the second temperature 710.

[0060] In the second training sample 704 and the third training sample 706, row 618F shows that column 616A(C L (at time t-3)) block 620 has the third temperature 712, column 616B(C L (at time t-2)) block 620 has a fourth temperature, column 616C(C L (at time t-1)) block 620 has a fourth temperature, and column 616N(C L (at time t)) block 620 has the fourth temperature 714. Row 618G of the first training sample 702 shows that column 616A(C L (at time t-3)) block 620 has the second temperature, column 616B(C L (at time t-2)) block 620 has the third temperature, column 616C(C L (at time t-1)) block 620 has the third temperature, and column 616N(C L (at time t)) block 620 has the third temperature. Row 618J of the second training sample 704 shows that at column 616C(C L (at time t-1)) block 620 has the second temperature, and at column 616N(C L (at time t)) block 620 has the third temperature.

[0061] In some examples, such as shown in the example of Figure 7 , image data from the image sensor 120 or a derived quantity thereof is transmitted from the device 110 via the regulator 150 and the communication device 155 to an external machine 170 (e.g., any kind of computing device, computer, server, etc.) via the RF broadcast tower 160 and / or the network 165.

[0062] As described above, in some examples, the training of the impression 254 includes the implementation of neural networks, decision trees, support vector machines (SVMs), and / or other machine learning applications. The training of the impression 254 can include, for example, updating matrices or coefficients in response to image data from the image sensor 120 to modify the impression 254 to accommodate valid variations in the image data. In Figure 7 an example, a training model (e.g., a backpropagation model, a decision tree-based model, etc.) is applied to historical image data (e.g., the first training sample 702 and the second training sample 704) and current image data (e.g., the third training sample 704) such that the impression or the trained model will predict image data close to the current image data based on the historical image data. In other words, if it is desired to train the impression 254 to predict future values, the training process is advantageously informed via multiple examples of results (x1, y1), (x2, y2),... (xN, yN), where x will be a vector of input data and y will be the desired output. During the training process of the impression 254, the coefficients of the impression 254 are adjusted to incorporate the "knowledge" of the input image data pattern until the impression begins to approximate or actually approximate the desired output. At this point, in the case of normal, known behavior of the device 110, further input of image data into the impression 254 should result in an output that is close to the actual image data, and an output that deviates from the actual image data can be identified (e.g., an incorrect prediction that differs from the actual image data by more than a predetermined percentage difference, an incorrect prediction that differs from the actual image data by more than a predetermined threshold, etc.).

[0063] In some examples, the impression 254 is trained during normal operation of the device 110 for a sufficient amount of time so that the impression 254 learns the actual behavior of the device 110 before any potential possibility of abnormal behavior of the device 110. In some examples, the device 110 can include in the memory 250 an impression 254 supplied by the vendor (e.g., values of coefficients of a pre-trained model, etc.).

[0064] Figure 8 is illustrated by Figures 1 - 2 the regulator 150 of Figure 5A and / or Figure 5B implementing a representation of the image data from Figure 1 and / or Figures 3A - 3C the image sensor 120 according to Figure 8 example instructions. Specifically, Figure 8 represents the monitoring phase of the impression 254 during operation of the device 110. On the Figure 8 left is a representation of an example history 805 of the device 110 (e.g., an example autonomous or manually operated device, etc.). In Figure 7corresponds to the described training sample 704. In some examples, the input data vector (C L (t X )) (e.g., Figure 8 of C L (t - 3), C L (t - 2) and C L (t - 1)) is analyzed via a function F (e.g., a prediction model derived from the training sample, a prediction function in the form of a neural network, an adjustment via a decision tree learning model, an association rule learning model, etc.) to produce an impression 254 or a trained model, including an example array 830 of expected values 831A - 831P (e.g., a data vector N * L (t) etc. containing data distributed based on predictions of historical values). N * L (t) and (C L (t X )) are normalized to a common basis.

[0065] Figure 8 In the example array 850, the image data corresponds to the actual temperature, and more specifically, example rows 851A - 851E indicate the first temperature 708, example row 851F indicates the fourth temperature 714, example row 851G indicates the fourth temperature 714, example rows 851H - 851I indicate the first temperature 708, example row 851J indicates the fourth temperature 714, and example rows 851K - 851P indicate the first temperature 708.

[0066] Then, the expected values 831A - 831P are compared by the impression comparator 230 with the example rows 851A - 851P of the data vector (C L (t)) in the example array 850 corresponding to the actual image data.

[0067] In some examples, the data vectors (C L (t)) (e.g., the actual values of the image data) of rows 851A - 851P of the array 850 are compared with the data vectors of the rows of the expected values 831A - 831P of the array 830 (N * L (t)) (e.g., the expected values of the impression 254) to determine whether any comparison between the corresponding data vectors (e.g., the comparison of the data vector of row 851A with the data vector of row 831A, etc.) is greater than a threshold difference. In some examples, when the data vectors represent temperature (e.g., absolute temperature, temperature gradient, etc.), the threshold difference can be expressed in the form of a temperature difference (e.g., a difference of 1°F, 2°F, 3°F…10°F, 20°F, 30°F, etc.). By way of illustration, the data vector C 10(t) is shown as corresponding to a fourth temperature 714, and a data vector N of row 831J * 10 (t) (e.g., an impression) is shown as corresponding to a third temperature 712, indicating the difference therebetween. In some examples, when the data vector represents a distance or a dimension, the threshold difference can be expressed in the form of a dimension difference (e.g., a difference of 0.1 mm, 0.2 mm, 0.3 mm, etc.). In some examples, the threshold difference is determined via the sum of absolute differences, the sum of squares of differences, etc.

[0068] Figure 9 is capable of executing Figures 5A - 5B instructions to implement Figure 2 is a block diagram of an example processor platform 900 of an example regulator 150. The processor platform 900 can be, for example, a server, a personal computer, a mobile device (e.g., a cellular phone, a smart phone, a tablet device such as an iPad TM or the like), a vehicle controller, a drone controller, a robotic device controller, a personal digital assistant (PDA), an Internet device, or any other type of computing device.

[0069] The illustrated example processor platform 900 includes a processor 912. The illustrated example processor 912 is hardware. For example, the processor 912 can be implemented by one or more integrated circuits, logic circuits, microprocessors, or controllers from any desired family or manufacturer. The hardware processor can be a semiconductor-based (e.g., silicon-based) device. In this example, the processor 912 implements the regulator 150, the image manager 210, the impression manager 220, and the impression comparator 230.

[0070] The illustrated example processor 912 includes local memory 913 (e.g., a cache). The illustrated example processor 912 communicates with a main memory including volatile memory 914 and non-volatile memory 916 via a bus 918. The volatile memory 914 can be implemented by synchronous dynamic random access memory (SDRAM), dynamic random access memory (DRAM), RAMBUS dynamic random access memory (RDRAM), and / or any other type of random access memory device. The non-volatile memory 916 can be implemented by flash memory and / or any other desired type of memory device. Access to the volatile memory 914, the non-volatile memory 916, the local memory, and / or the main memory is controlled by a memory controller.

[0071] The illustrated example processor platform 900 further includes interface circuitry 920. The interface circuitry 920 can be implemented by any type of interface standard, such as an Ethernet interface, a universal serial bus (USB), and / or a PCI Express interface.

[0072] In the illustrated example, one or more input devices 922 are connected to interface circuit 920. The input device(s) 922 permit a user to input data and / or commands into processor 912. The input device(s) may be implemented by, for example, audio sensors, microphones, cameras (still or video), keyboards, buttons, mice, touchscreens, trackpads, trackballs, point mice, and / or voice recognition systems.

[0073] One or more output devices 924 are also connected to interface circuit 920 of the illustrated example. Output device 924 may be implemented by, for example, display devices (such as light emitting diodes (LEDs), organic light emitting diodes (OLEDs), liquid crystal displays, touchscreens, haptic output devices, printers, and / or speakers). Accordingly, interface circuit 920 of the illustrated example typically includes a graphics driver card, a graphics driver chip, and / or a graphics driver processor.

[0074] Interface circuit 920 of the illustrated example also includes communication devices such as transmitters, receivers, transceivers, modems, and / or network interface cards to facilitate the exchange of data with external machines (such as any type of computing device) via network 165 (such as, for example, an Ethernet connection, digital subscriber line (DSL), telephone line, coaxial cable, cellular telephone system, etc.).

[0075] Processor platform 900 of the illustrated example also includes one or more mass storage devices 928 for storing software and / or data. Examples of such mass storage devices 928 include floppy disk drives, hard disk drives, compact disk drives, Blu-ray disk drives, RAID systems, and digital versatile disk (DVD) drives.

[0076] Figures 5A - 5B The encoded instructions 932 may be stored in mass storage device 928, stored in volatile memory 914, stored in non-volatile memory 916, and / or stored on a removable tangible computer-readable storage medium such as a CD or DVD.

[0077] In view of the foregoing, it will be appreciated that example methods, apparatuses, and articles have been disclosed that are capable of early detection of various known and / or unforeseen faults and capable of early intervention. This is particularly advantageous for autonomous devices, which can help detect problems before the autonomous device becomes irrecoverable, and for detecting devices with a high degree of similarity, because knowledge obtained on one autonomous and / or manually operated device can be applied to other similar autonomous and / or manually operated devices.

[0078] Example 1 is a monitoring system, the monitoring system including: an image sensor for obtaining image data of a device; and a regulator for causing the image sensor to: obtain image data of the device; form an impression from the image data; use the impression and the image data to generate an expected value for the state of the device at a predetermined moment; and compare the image data of the structure at the predetermined moment with the generated expected value.

[0079] Example 2 includes the monitoring system of Example 1, wherein the image sensor includes at least one of a thermal image sensor, a spatial image sensor, and an optical image sensor, and wherein the image data includes at least one of thermal image data, spatial image data, and optical image data.

[0080] Example 3 includes the monitoring system of Example 1 or Example 2, wherein the image sensor includes a thermal image sensor and a spatial image sensor, and wherein the image data includes thermal image data and spatial image data.

[0081] Example 4 includes the monitoring system of any one of Examples 1-3, wherein the image sensor is configured to: during operation of the device, obtain image data of the device via the image sensor.

[0082] Example 5 includes the monitoring system of any one of Examples 1-4, wherein the image sensor is configured to: during multiple operations of the device, obtain image data of the device via the image sensor.

[0083] Example 6 includes the monitoring system of any one of Examples 1-5, wherein the impression is formed from multiple sets of image data.

[0084] Example 7 includes the monitoring system of any one of Examples 1-6, further including a non-transitory machine-readable medium for storing at least one of the impression or the image data.

[0085] Example 8 includes the monitoring system of any one of Examples 1-7, wherein the impression includes a mapping of at least one of a spatial gradient or a thermal gradient within a selected volume of the structure of the device onto a coefficient vector to allow comparison with actual values from the image data on a corresponding basis space.

[0086] Example 9 includes the monitoring system of any one of Examples 1-8, wherein the regulator is configured to determine whether telemetry data of an instrument of one or more systems or subsystems from the monitoring device is within acceptable limits after comparing the image data of the device at a predetermined moment with the expected value generated from the impression.

[0087] Example 10 includes the monitoring system of any one of Examples 1-9, and further includes a communication device configured to transfer at least one of an impression, image data, or a deviation report related to the impression to a remote device.

[0088] Example 11 includes the monitoring system of any one of Examples 1-10, wherein a regulator is configured to generate a deviation report in response to a difference between an expected value generated by an impression and an actual value of image data.

[0089] Example 12 includes the monitoring system of any one of Examples 1-11, wherein the remote device includes another device configured similarly to the device.

[0090] Example 13 includes the monitoring system of any one of Examples 1-12, wherein the remote device includes a central server or service that communicates with a plurality of similarly configured devices.

[0091] Example 14 is a method for automated monitoring of a device, including: during an operating state of the device, imaging the device to obtain first image data; forming an impression from the first image data; during a subsequent operating state of the device, imaging the device to obtain second image data; estimating a value of the second image data using the impression; and comparing the estimated value of the second image data with the actual value of the second image data.

[0092] Example 15 includes the method for automated monitoring of Example 14, and further includes: determining whether a difference between the estimated value of the second image data and the actual value of the second image data is less than a threshold difference.

[0093] Example 16 includes the method for automated monitoring of Example 14 or Example 15, and further includes: determining whether telemetry data from a sensor of the device is within acceptable limits.

[0094] Example 17 includes the method for automated monitoring of any one of Examples 14-16, and further includes: when the telemetry data is within an acceptable range, updating the impression to incorporate the second image data as good data.

[0095] Example 18 includes the method for automated monitoring of any one of Examples 14-17, and further includes: when the telemetry data from a sensor of the device is not within acceptable limits, updating the impression to incorporate the second image data as bad data.

[0096] Example 19 includes the method for automated monitoring of any one of Examples 14-18, and further includes: outputting a deviation report.

[0097] Example 20 includes the method for automated monitoring of any one of Examples 14-19, and further includes: after determining that the difference between the estimated value and the actual value of the second image data is less than a threshold difference, comparing the estimated value of the second image data with a known value corresponding to an adverse outcome to determine whether the estimated value of the second image data corresponds to an adverse outcome.

[0098] Example 21 includes the method for automated monitoring of any one of Examples 14-20, and further includes: when determining that the estimated value of the second image data corresponds to an adverse outcome, outputting a deviation report.

[0099] Example 22 includes the method for automated monitoring of any one of Examples 14-21, and further includes: if the estimated value of the second image data is determined not to correspond to an adverse outcome, determining whether telemetry data from a sensor of the device is within acceptable limits.

[0100] Example 23 includes the method for automated monitoring of any one of Examples 14-22, and further includes: when the telemetry data is within acceptable limits, updating the impression to incorporate the second image data as good data.

[0101] Example 24 is a system that includes: an imaging device configured to obtain image data of a device; and an adjustment device configured to cause an image sensor to: obtain image data of the device; form an impression from the image data; use the impression and the image data to generate an expected value for the state of the device at a predetermined time; and compare the image data of the device at the predetermined time with the generated expected value.

[0102] Example 25 includes the system of Example 24, wherein the imaging device includes at least one of a thermal imaging device, a spatial imaging device, and an optical imaging device, and wherein the image data includes at least one of thermal image data, spatial image data, and optical image data.

[0103] Example 26 includes the system of Example 24 or Example 25, wherein the imaging device includes a thermal imaging device and a spatial imaging device, and wherein the image data includes thermal image data and spatial image data.

[0104] Example 27 includes the system of Examples 24-26, wherein the imaging device is configured to: during operation of the device, obtain image data of the device via the imaging device.

[0105] Example 28 includes the system of Examples 24-27, wherein the imaging device is configured to: between multiple operating cycles of the device, obtain image data of the device via the imaging device.

[0106] Example 29 includes the system of any one of Examples 24-28, wherein an impression is formed from multiple sets of image data.

[0107] Example 30 includes the system of Examples 24-29, and further includes a communication device configured to transmit at least one of the following to a remote device: an impression, image data, or a deviation report related to the impression in response to a difference between an expected value generated from the impression and an actual value of the image data exceeding a threshold difference.

[0108] Example 31 is a non-transitory machine-readable medium including executable instructions that, when executed, cause at least one processor to: image a device during an operating state of the device to obtain first image data; form an impression from the first image data; image the device during the operating state of the device to obtain second image data; estimate a value of the second image data using the impression; and compare the estimated value of the second image data with the actual value of the second image data.

[0109] Example 32 includes the non-transitory machine-readable medium of Example 31, and further includes executable instructions that, when executed, cause at least one processor to determine whether a difference between the estimated value of the second image and the actual value of the second image data is less than a threshold difference.

[0110] Example 33 includes the non-transitory machine-readable medium of Example 31 or Example 32, and further includes executable instructions that, when executed, cause at least one processor to determine whether telemetry data from a sensor of the device is within acceptable limits.

[0111] Example 34 includes the non-transitory machine-readable medium of Examples 31-33, and further includes executable instructions that, when executed, cause at least one processor to update the impression to incorporate the image data as good data when the telemetry data is within acceptable limits.

[0112] Example 35 includes the non-transitory machine-readable medium of Examples 31-34, and further includes executable instructions that, when executed, cause at least one processor to update the impression to incorporate the image data as bad data when the telemetry data from a sensor of the device is not within acceptable limits.

[0113] Example 36 includes the non-transitory machine-readable medium of Examples 31-35, and further includes executable instructions that, when executed, cause at least one processor to output a deviation report when a difference between the estimated value of the second image and the actual value of the second image data is greater than a threshold difference.

[0114] Example 37 is a monitoring system that includes: an image sensor for obtaining image data of a device; and an adjuster for causing the image sensor to: obtain image data of the device; form an impression from the image data; and use the impression and the image data to determine a ruling.

[0115] Example 38 includes the monitoring system of Example 37, wherein the ruling is determined by: using the impression and the image data to generate an expected value for the state of the device at a predetermined time, and comparing the image data for the device at the predetermined time with the generated expected value.

[0116] Example 39 includes the monitoring system of Example 37 or Example 38, wherein the image is used to directly determine the ruling using the image data.

[0117] Although certain example methods, devices, and articles have been disclosed herein, the scope covered by this patent is not limited thereto. Instead, this patent covers all methods, devices, and articles that fall within the scope of the claims of this patent. For example, although the examples herein have disclosed an implementation of the image sensor 120 to obtain image data representing the temperature of one or more objects in a specified volume 405, in some examples, a non-contact thermometer (pyrometer) and / or an array of contact thermometers (multiple thermometers) may be placed on the surface of one or more parts to provide temperature data instead of image data.

Claims

1. A monitoring system, comprising: A thermal image sensor configured to obtain a thermal image of a device; And A regulator configured to: Form an impression from the thermal image, the impression being configured to map a thermal gradient within a selected volume of the structure of the device onto a coefficient vector, Generate an expected value for the state of the device at a moment of interest based on the impression, and Compare the thermal image data of the device at the moment of interest with the generated expected value.

2. The monitoring system according to claim 1, further comprising an optical image sensor configured to obtain optical image data.

3. The monitoring system according to claim 1 or claim 2, further comprising a spatial image sensor configured to obtain spatial image data.

4. The monitoring system according to claim 1 or claim 2, characterized in that, The thermal image sensor is configured to: obtain the thermal image of the device during operation of the device.

5. The monitoring system according to claim 1 or claim 2, characterized in that The thermal image sensor is configured to: obtain the thermal image of the device between multiple operating cycles of the device.

6. The monitoring system according to any one of claims 1 or 2, characterized in that, The regulator is configured to form the impression from multiple sets of the thermal image data.

7. The monitoring system according to claim 3, characterized in that The impression is configured to: map a spatial gradient within the selected volume of the structure of the device onto the coefficient vector to allow comparison with actual values from the spatial image data on a corresponding basis space.

8. The monitoring system according to claim 1 or claim 2, further comprising a communication device configured to transmit at least one of the impression, the thermal image data, or a deviation report related to the impression to a remote device, the regulator being configured to generate the deviation report in response to a difference between an expected value generated from the impression and an actual value of the thermal image data.

9. The monitoring system according to claim 1 or claim 2, characterized in that, The regulator is configured to: update the coefficient vector based on a change in the thermal image data to modify the impression.

10. The monitoring system according to claim 1 or claim 2, characterized in that, The regulator is configured to: Form the impression based on multiple training samples occurring over time; and Adjust the coefficient vector for the corresponding training sample.

11. The monitoring system according to claim 1, characterized in that, The regulator is further configured to: determine a ruling based on the comparison.

12. The monitoring system according to claim 11, characterized in that, The impression is configured to directly use the thermal image data to determine the ruling.

13. A method for automated monitoring of a device, comprising: Performing thermal imaging on the device during an operating state of the device to obtain first thermal image data; Forming an impression from the first thermal image, the impression being configured to map a thermal gradient within a selected volume of the structure of the device onto a coefficient vector; Imaging the device during a subsequent operating state of the device to obtain second thermal image data; Estimating a value of the second thermal image data using the impression; And Comparing the estimated value of the second thermal image data with the actual value of the second thermal image data.

14. The method according to claim 13, further comprising: Determining whether a difference between the estimated value of the second thermal image data and the actual value of the second thermal image data is less than a threshold.

15. The method according to claim 14, further comprising: Determining whether telemetry data from a sensor of the device is within acceptable limits.

16. The method according to claim 15, further comprising: Updating the impression to incorporate the second thermal image data when the telemetry data is within acceptable limits.

17. The method according to claim 15, further comprising: Identify the second thermal image data as bad data when the telemetry data from the sensors of the device is not within acceptable limits.

18. The method according to any one of claims 13 to 17, further comprising: Output a deviation report in response to the difference between the expected value generated from the impression and the actual value of the second thermal image data.

19. The method according to claim 14, further comprising: After determining that the difference between the estimated value and the actual value of the second thermal image data is less than the threshold, compare the estimated value of the second thermal image data with a known value corresponding to a bad result to determine whether the estimated value of the second thermal image data corresponds to the bad result.

20. A computer device for monitoring, comprising: Means for thermally imaging the device to obtain a thermal image of the device; Means for forming an impression from the thermal image, the impression for mapping a thermal gradient within a selected volume of the structure of the device onto a coefficient vector; Means for generating an expected value for the state of the device at a moment of interest based on the impression; And Means for comparing the thermal image data of the device at the moment of interest with the generated expected value.

21. The computer device according to claim 20, further comprising means for optically imaging the device to obtain an optical image of the device.

22. The computer device according to claim 20 or claim 21, further comprising means for spatially imaging the device to obtain a spatial image.

23. The computer device according to claim 20 or claim 21, characterized in that, The means for thermally imaging the device is configured to: obtain the thermal image of the device during operation of the device.

24. The computer device according to claim 20 or claim 21, characterized in that, The means for thermally imaging the device is configured to: obtain the thermal image of the device between multiple operating cycles of the device.

25. The computer device according to claim 20 or claim 21, characterized in that, The impression is formed from multiple thermal images.

26. The computer device according to claim 22, characterized in that, The impression is configured to: map a spatial gradient within the selected volume of the structure of the device onto the coefficient vector to allow comparison with the actual value from the spatial image on a corresponding basis space.

27. The computer device according to claim 20 or claim 21, further comprising means for transmitting to a remote device: the impression, the thermal image data, or a deviation report related to the impression in response to the difference between the expected value generated from the impression and the actual value of the thermal image data exceeding a threshold difference.

28. The computer device according to claim 20 or claim 21, characterized in that, The means for forming the impression is configured to: update the coefficient vector based on changes in the thermal image data to modify the impression.

29. The computer device according to claim 20 or claim 21, characterized in that, The means for forming the impression is configured to: Form the impression based on multiple training samples occurring over time; and Adjust the coefficient vector for the corresponding training samples.

Images