Predictive asset maintenance using thermal and conventional imaging

Abstract

A method for predicting time to failure condition of an asset receives an image pair of a normal state asset and a historical image pair of the asset at the asset's fault condition from an inspection. A first model is trained to reconstruct the input images to resemble the normal state asset. Using the reconstruction of the images by the first model as a similarity base, a similarity coefficient is generated for the historical image pair of normal and infrared images, the historical image pair including a timestamp of image capture. A second model is trained to predict time to failure condition of the asset based on the timestamp of image capture, the timestamp of the asset's fault condition, and the similarity coefficient of the historical images. The method calculates a predicted time to failure condition of the asset in response to receiving the real-time image pair.

Description

[Technical Field]

[0001] The present invention relates to asset maintenance and failure prediction, and more particularly to the use of machine learning recognition of asset condition based on visual and infrared images of the asset to predict maintenance intervention. [Background technology]

[0002] Enterprise asset management (EAM) is a specific type of enterprise resource planning (ERP) that involves the integrated management of key business processes, often conducted in real time through software and technology. EAM can be characterized as having functionality that supports management activities for large assets such as truck fleets, construction equipment, turbines, oil platforms, and energy generation facilities. EAM systems and software facilitate the design, configuration, operation, maintenance, and replacement of large assets and asset components.

[0003] Asset failure prevention techniques involve the use of data to track the mean-time-between-failure (MTBF) of asset components. Businesses and organizations benefit from accurately determining MTBF and taking steps to perform maintenance actions to prevent failures while avoiding performing maintenance before it is needed.

[0004] Infrared (IR) light is a type of radiant electromagnetic energy that is invisible to human vision but can be felt or measured as heat. Capturing IR images with special devices, such as thermal imaging cameras, provides additional details about anomalies that cannot be detected with visible light images or human vision. In many cases, IR images provide a so-called thermal signature that provides information about normal or expected operating conditions. Summary of the Invention

[0005]

[0009] Embodiments of the present invention disclose a computer-implemented method, computer program product, and system for predicting a time to failure state of an asset. The computer-implemented method includes a computer processor receiving a plurality of historical image pairs of an asset, including a sampling of image pairs of the asset having a normal state status and a sampling of image pairs of the asset having a fault state, each image pair including an infrared image and a corresponding conventional image. The processor trains a first model to reconstruct each image of the image pairs based on training data from the sampling of image pairs of the asset having a normal state status. For each of the plurality of historical image pairs, the processor generates a similarity coefficient between the historical image pair and a reconstruction of the historical image pair by the first model, the historical image pair including a timestamp of capture of the historical image pair. The computer processor trains a second model to predict a time to failure state of the asset based on the similarity coefficients of the plurality of historical image pairs of the asset and the timestamps associated with the capture of the plurality of historical image pairs, respectively. The processor, in response to receiving a real-time image pair of the asset, generates a real-time similarity coefficient between the real-time image pair and the real-time image pair reconstructed by the first model, the real-time image pair including a timestamp of capture of the real-time image pair, and the processor calculates a predicted time to a failure condition of the asset based on a correlation between the similarity coefficient of the real-time image pair and the timestamp of capture of the real-time image pair, a historical image pair among a plurality of historical image pairs having a corresponding similarity coefficient, and a time delta determined from the timestamp of capture of the real-time image pair and a timestamp of a failure condition of the asset following the historical image pair having the corresponding similarity coefficient. [Brief explanation of the drawings]

[0006] [Figure 1]FIG. 1 is a functional block diagram illustrating a distributed computing environment, according to one embodiment of the present invention. [Figure 2] 2 is a flowchart illustrating the operational steps of a failure prediction program including an image rebuild module and an image similarity module as components operating in the distributed computing environment of FIG. 1 in accordance with an embodiment of the present invention. [Figure 3] FIG. 3 is a block diagram of components of a computing system including a computing device configured to operatively execute a fault prediction program including the image reconstruction module and image similarity module of FIG. 2 in accordance with one embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0007] Embodiments of the present invention recognize that accurately determining when to perform preventive maintenance on expensive assets can avoid costly or damaging consequences of asset failure and prevent increased operating costs by replacing components or performing maintenance activities less frequently than necessary. In many cases, the occurrence of asset failure can have significant consequences, encouraging increased frequency of maintenance activities. Embodiments of the present invention recognize that asset owners often choose a "lesser of two evils" practice philosophy and accept the inconvenience and additional costs associated with more frequent scheduled asset maintenance.

[0008] Embodiments of the present invention also recognize that asset inspections often require manual activities to determine an indication of whether an asset requires maintenance, and that manual activities may involve operational or personnel risks or may present challenges to accurate inspection results due to limited access to the asset or asset components. Embodiments further recognize that asset owners make decisions that balance the risk of greater consequences of asset failure by performing more frequent asset inspections, which involves higher costs and potential risks to inspectors.

[0009] Embodiments of the present invention provide computer-implemented methods, computer program products, and computer systems for predicting a fault condition associated with an asset based on generating similarity coefficients from normal visible light images and infrared images that provide a quantified difference between the current asset condition and the asset's normal state. Embodiments also provide machine learning models for predicting when a given asset is likely to fail or require maintenance-related intervention by correlating the similarity coefficients between the normal and infrared images of a current inspection with corresponding similarity coefficients of image pairs from multiple historical normal and infrared image pairs associated with previous fault conditions and previous inspections of the asset condition.

[0010] In one embodiment of the present invention, the term "image" refers to a digitized representation of an object, rendered as a mosaic of pixels in a format that allows for computer-based analysis. Embodiments include conventional images, which are images that capture electromagnetic wavelengths related to visible light, and infrared images, which capture electromagnetic wavelengths beyond (i.e., above) the electromagnetic wavelengths of visible red light.

[0011] In some embodiments, the capture of the conventional and infrared images occurs substantially simultaneously and includes computer-recognizable features associated with groupings of pixels within each image. In some embodiments, the term “similarity coefficient” or “similarity coefficient” refers to determining a value, often referred to as a distance, between corresponding regions of a pair of current light source images (i.e., a pair of a current visible light source image and a previous light source image), more specifically, a magnitude value assigned to a vector in a multidimensional space representing the distance between one or more features located within each region of the image pair. A larger distance indicates less similarity between the features of the image pair, while a smaller distance value indicates more similarity between the features of the image pair. For example, computer image analysis identifies a set of features in each of two infrared images of a component of an asset. The regions of pixels within each region of the infrared images include the identified features, and the analysis includes generating a similarity coefficient between the respective regions of the infrared images and generating attributes of the pixel regions that can be assigned as different dimensions of the vector representation.

[0012] In some embodiments of the present invention, an asset refers to a physical object that performs an operation that may be inspected by image capture and that, over time, prevents the asset from performing its expected service. In some embodiments, an asset includes a large item such as a vehicle, construction equipment, turbine, generator, aircraft part, etc., although embodiments of the present invention are not limited by the size, type, or role of the asset. In some embodiments, a primary asset may include secondary components, which may also be considered secondary assets of the primary asset. Embodiments of the present invention consider measures to prevent or avoid a failure of the operational performance of an asset as asset intervention to perform preventative activities, such as maintenance. In some embodiments, reference to an asset failure state includes an asset degradation state. In some embodiments, asset inspection is performed before and after asset intervention and includes capturing conventional and infrared images of the asset. For clarity and simplicity, references herein to an asset failure, asset failure event, or asset failure state include a near-failure or imminent failure asset state that supports preventative intent.

[0013] Embodiments provide a practical solution to the problem of determining accurate predictions of when to perform an intervention on an asset for maintenance or other preventative measures prior to asset failure. Embodiments receive a sampling of image pairs for an asset, where the image pairs reflect the asset's normal condition status, and receive multiple historical image pairs for the asset, including the asset's fault condition and asset inspection. In some embodiments, a set of historical image pairs of similar or identical assets or asset components (with respective timestamp information) may be combined into a historical collection of image pairs for that asset type. For example, image pairs of brake assemblies for a fleet of similar trucks may be combined to create a larger set of historical image pairs. The historical image pairs for the asset include timestamps of image capture and information related to specific intervention actions. In some embodiments, a response to an asset's fault condition includes an intervention action to repair, remediate, or replace the asset. In other embodiments, the intervention provides for an inspection of the asset, and includes a timestamp of the inspection along with the capture of the image pair for the asset.

[0014] For example, an enterprise organization may associate asset interventions with work orders or other formal documented information (such as hiring or scheduling of the intervention activity). The work order may include documentation of the intervention activity and a timestamp of the activity; if the intervention includes repair or replacement of the asset, the work order information includes the type of failure, the duration of the repair or maintenance, the parts replaced, the adjustments made, and the costs associated with the intervention. Embodiments of the present invention include intervention activity information involving inspections, maintenance activities, and failure event repairs in multiple historical image pairs. Timestamp information associated with historical images of inspections and asset failure events serves as data indicative of the duration until a subsequent asset failure event. Timestamp information associated with image pairs captured during the intervention provides data for machine learning prediction of the time to asset failure.

[0015] Embodiments perform segmentation of the conventional and infrared images of an image pair to provide a more fine-grained representation of the asset's components. In some embodiments, image segmentation involves segmenting a square matrix of regions (e.g., n * 2 regions).

[0016] An embodiment trains a first model to receive image regions of an asset as input and reconstruct the input into a normal-state image of the asset. Training the first model includes exclusively providing image pairs of the asset in a normal state from a sampling of image pairs of the asset as training data input. In this way, the first model establishes a reference condition for the asset's normal state taking into account various image inputs from the real-time inspection. The real-time inspection image pairs may be captured under various conditions that may differ from the original sampling of the image pairs of the asset in a normal state. Using the original sampling of the image pairs as a reference for the normal state of the real-time image pairs may inaccurately affect the similarity coefficient results in areas of the image pairs that would not otherwise contribute. For example, angle, lighting, or non-impacting accumulated debris may cause inaccurate calculation of the similarity coefficient.

[0017] In various embodiments, training the first model includes identifying and recognizing features of the asset within each area segment of the sampling of received image pairs. In some embodiments, the normal state status of an asset includes new assets, replaced components of the asset, and reconditioned assets or components of the asset that place the asset in an operating state where wear, stress, fatigue, vibration, excessive heat, or other attributes that indicate the asset's operating condition is moving away from new or near-new and approaching a fault state are absent or reduced. Training the model to identify features within the area of ​​the sampling of image pairs of assets with normal state status provides a reference point for generating a similarity coefficient between the historical image and the reference image. To generate the reference image, the first model reconstructs copies of the normal real-time image and the infrared real-time image received as input to the asset's normal state (i.e., to a state as close to "normal" as possible), which provides a reference for changes to the asset and improves the accuracy of the calculated similarity coefficient. Reconstructing the real-time image avoids problems associated with non-contributing components of the image affecting the analysis and determination of the similarity coefficient.

[0018] Embodiments of the present invention generate a similarity coefficient between a reference image pair depicting a normal state of an asset and each image pair from a plurality of historical image pairs of the asset. Embodiments generate a similarity coefficient for each segmented region of each image pair of the plurality of historical image pairs. In various embodiments, providing each historical image pair to a first model enables reconstruction of the historical image pair. The reconstruction forms a reference image pair that includes a region of the normal state asset. Embodiments generate a similarity coefficient between the segmented region of the reconstructed normal image and infrared image of the historical image pair and the segmented region of each normal image and infrared image of the historical image pair. In some embodiments, a similarity coefficient is generated for each image pair of the plurality of historical image pairs.

[0019] Embodiments perform the generation of a similarity coefficient for each historical image pair of a plurality of historical image pairs of an asset. The plurality of historical image pairs represent inspections and instances of asset failure events captured from previous intervention activities of repair, maintenance, and inspection of the asset, and include timestamps and additional information associated with the intervention activities. The resulting set of similarity coefficients for the plurality of historical image pairs provides a range of similarity values ​​associated with the asset from normal to faulty status. In some embodiments, the similarity coefficient is calculated as the Euclidean distance between pixels in a region of the reference image of the asset and corresponding pixels of the region in the real-time image of the image pair of the inspection of the asset. In other embodiments, the similarity coefficient is calculated based on a cosine similarity method or other similarity method.

[0020] Embodiments of the present invention train a second model to predict a time to a fault condition of an asset. The predicted time to fault condition enables planning and execution of a maintenance intervention on the asset to prevent the fault condition. Determining the predicted time to failure includes correlating a similarity coefficient of a real-time image pair from an asset inspection with a corresponding similarity coefficient from an image pair of a plurality of historical image pairs. Furthermore, the duration between a timestamp associated with a corresponding historical image pair and a subsequent historical asset fault condition timestamp is a contributing factor to determining the predicted time to failure of the asset. In some embodiments, adjusting the predicted time of intervention to a timeframe prior to the predicted fault condition of the asset prevents the failure from occurring. In some embodiments, information associated with the plurality of historical image pairs provides data for training a second model for predicting a fault condition of the asset. The historical image pair information includes a timestamp of the historical intervention, pairs of conventional and infrared images captured during the historical intervention, and details about the intervention, such as details of the fault condition, maintenance / repair performed, time to perform the intervention, and cost of the intervention.

[0021] In response to receiving a real-time image pair of an asset, an embodiment of the present invention generates a real-time similarity coefficient for each region of an infrared image and a conventional image. In some embodiments, a real-time inspection of an asset, in which the image pair is captured, includes a timestamp associated with the image pair. An embodiment of the present invention generates a similarity coefficient from the real-time image pair of the inspection by sending copies of the infrared image and the conventional image as input to a first model, which outputs a reconstruction of the sent images toward an image depicting a reference normal state asset. The real-time image pair of the inspection and the reconstructed image pair reference provide a basis for generating a similarity coefficient for the real-time inspection. An embodiment generates a similarity coefficient for each segmented region of the corresponding infrared image and each segmented region of the corresponding conventional image.

[0022] An embodiment of the present invention calculates a predicted time to failure event of an asset based on a correlation between (i) a similarity coefficient of a region of a real-time image pair and a timestamp of the capture of the real-time image pair, and (ii) a time delta determined from a region of a historical image pair among a plurality of historical image pairs having a corresponding similarity coefficient and a timestamp of the capture of an image pair of a fault state of the asset following the timestamp of the capture of the real-time image pair and the historical image pair having the corresponding similarity coefficient.

[0023] In some embodiments, a work order or other documented asset intervention information source includes a pair of images of the asset condition (i.e., an infrared image and a conventional visible light image) and a timestamp of the image capture, and may include one or a combination of the duration of the intervention, activities performed during the intervention, whether the intervention was in response to a fault condition of the asset, and the cost of the intervention. In some embodiments, a work order each associated with multiple historical images of the asset includes an asset inspection intervention after a normal condition status of the asset and before the fault condition of the asset.

[0024] In some embodiments, the first model includes an autoencoder neural network and training using normal and infrared images of the asset with a normal state status. In some embodiments, a separately developed and trained version of the first model performs infrared image reconstruction toward corresponding infrared or normal images, respectively, that exhibit similarity to the asset's normal state status (i.e., a very small similarity distance value of the similarity coefficient), and a different version of the first model developed and trained using the normal images performs reconstruction of the asset's normal image. In some embodiments, the development and training of the second model includes separate versions that use normal image similarity coefficients and infrared image similarity coefficients, respectively, from real-time inspections of the asset. In embodiments of the present invention, the second model is recognized to perform a regression function to learn patterns of time deltas to the asset's fault state from inspection images and timestamps associated with their capture, as well as similarity coefficients of image representations of historical fault states, historical inspection image representations, and timestamps associated with interventions.

[0025] The description of various embodiments of the present invention is presented for illustrative purposes, but is not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope of the described embodiments. The terms used herein are selected to best explain the principles of the embodiments, practical applications, or technical improvements beyond those found in the market, or to enable those skilled in the art to understand the embodiments disclosed herein.

[0026] The present invention will now be described in detail with reference to the drawings. Figure 1 presents a functional block diagram illustrating a distributed computing environment, generally designated 100, in accordance with one embodiment of the present invention. Figure 1 is intended only to provide an illustration of one implementation and is not intended to suggest any limitations with regard to the environments in which different embodiments may be implemented. Many modifications to the depicted environment may be made by one skilled in the art without departing from the scope of the present invention as defined in the claims.

[0027] The distributed computing environment 100 includes a computing device 110, a normal asset image pair 120, a historical intervention record 130, and a current asset inspection 160, all interconnected via a network 150. The distributed computing environment further includes an asset 140 from which the current asset inspection 160 is obtained.

[0028] Computing device 110 includes user interface 115 and failure prediction program 200, which is further illustrated as including the functionality of image reconstruction module 117 and image similarity module 119. In some embodiments, computing device 110 may be a blade server, a web server, a laptop computer, a desktop computer, a standalone mobile computing device, a smartphone, a tablet computer, or another electronic device or computing system capable of receiving, transmitting, and processing data. In other embodiments, computing device 110 may be a computing device that interacts with applications and services hosted and operating in a cloud computing environment. In other embodiments, computing device 110 may be a netbook computer, a personal digital assistant (PDA), or other programmable electronic device capable of receiving data from and communicating with other devices (shown and not shown) in distributed computing environment 100 via network 150 and performing the operations of failure prediction program 200. Alternatively, in some embodiments, computing device 110 may be communicatively connected to a remotely operating failure prediction program 200. Computing device 110 may include internal and external hardware components illustrated in further detail in FIG.

[0029] User interface 115 provides an interface for accessing features and functionality of computing device 110. In some embodiments of the present invention, user interface 115 provides access for operating and selecting options of risk prevention program 200 and may also support initiating support for training of the machine learning model component of failure prediction program 200. User interface 115 may also provide access and operational control of other applications, features, and functionality (not shown) of computing device 110. In some embodiments, user interface 115 provides display output and input capabilities to computing device 110. In other embodiments, user interface 115 provides display output and allows selection of options and functionality associated with failure prediction program 200 running on computing device 110.

[0030] The user interface 115 supports access to alerts, notifications, and provides access to communication forms. In one embodiment, the user interface 115 may be a graphical user interface (GUI) or web user interface (WUI) that can receive user input and display text, documents, web browser windows, user options, application interfaces, and instructions for operation. It may also include information (e.g., graphics, text, and sound) that a program presents to a user and control sequences that a user uses to control the program. In another embodiment, the user interface 115 may also include mobile application software that provides a respective interface to the features and functionality of the computing device 110. The user interface 115 enables each user of the computing device 110 to receive, see, hear, and respond to input, access applications, view the content of online conversations, and perform available functions.

[0031] The failure prediction program 200 includes an image reconstruction module 117 and an image similarity module 119. The failure prediction program 200 predicts the duration from an inspection of an asset, which includes capturing an image pair of the asset or a component of the asset, to a failure condition of the asset. The failure prediction program 200 determines the time to failure condition of the asset by correlating a similarity coefficient of a segmented region of the image pair of the asset captured from the inspection with a similarity coefficient of multiple historical image pairs of the asset captured over time, which include historical image pairs of the inspection and image pairs of the asset's failure condition.

[0032] The failure prediction program 200 segments each image of an image pair into an n-by-n region of pixels to obtain a more fine-grained representation of the asset and its components. The image pair includes a normal image in light visible to the human eye and an infrared image in light invisible to the human eye that indicates the heat level of the asset and asset components. The failure prediction program 200 determines a similarity coefficient by applying a similarity function to each segmented region of the image pair and a corresponding image of the image pair depicting the asset in a normal condition status. The failure prediction program 200 includes an image reconstruction module 117 in which a model receives image pair inputs from historical or real-time inspections and is trained to reconstruct or reconstruct the received inputs into a state that reflects an asset having a normal condition status. The failure prediction program 200 determines a similarity coefficient between the segmented regions of the received image pair and the reconstructed image pair, and performs the similarity coefficient determination for each of a plurality of historical image pairs and real-time inspections.

[0033] The failure prediction program 200 includes a second model that calculates the time from a real-time inspection, where receipt of an image pair includes a timestamp of the image pair capture, to a fault condition of the asset. A plurality of historical image pairs consisting of an infrared image and a conventional image, and the timestamp and information associated with each of the historical image pairs, provide training data for the second model. Training the second model enables it to learn correlations between the timing and conditions of the inspection and similarity coefficients associated with the image pair regions of the inspection, and the plurality of historical image pairs and their associated timing and conditions.

[0034] The image reconstruction module 117 receives the normal-state asset image pairs 120 as training data and performs reconstruction or reconstructing of the received images to result in image data that depicts or approximately depicts the asset as having a normal-state status. In some embodiments, the image reconstruction module 117 includes an autoencoder convolutional neural network model that is trained with input of selected images representing the asset 140 in a normal-state status. In some embodiments, the training of the image reconstruction module 117 enables it to perform image reconstruction for both the normal images and the infrared images. In other embodiments, the image reconstruction module 117 includes a separate trained neural network for the normal images and a separate trained neural network for the infrared images. Once properly trained, the image reconstruction module 117 receives multiple historical image pairs of the asset 140 from the historical intervention record 130 and reconstructs each image pair (or a copy of each historical image pair) to generate normal and infrared reference images for each historical image pair. Generating a similarity coefficient for each historical image pair includes comparing similarity distances between image segment features of the reconstructed reference image and the historical image pair reflecting the condition of the asset 140. The image reconstruction module 117 receives images of the current asset inspection 160 to generate a similarity coefficient baseline for similarity coefficient determination of the ongoing inspection, enabling prediction of the time to failure condition of the asset performed by the failure prediction program 200.

[0035] The image similarity module 119 calculates a similarity coefficient between a segmented region of one image of an image pair and a corresponding segmented region of the same image type in another image pair. In some embodiments, the similarity coefficient corresponds to a Euclidean distance calculated from pixels of a region of a first image and a corresponding region of a second image. For example, segmenting a first infrared image of a historical image pair divides each image of the image pair into 16 regions, and a digital copy of the first infrared image region is input to the image reconstruction module 117, which executes as a first model of the failure prediction program 200. The image reconstruction module 117 reconstructs the input image region to approximately depict the normal condition status of the asset. The image similarity module 119 determines the Euclidean distance between pixels of the corresponding region of the first infrared image of the historical image pair and pixels of the reconstructed image region depicting the normal condition asset. The greater the calculated distance between the regions, the lower the similarity and the higher the likelihood that the asset is in a less normal condition.

[0036] The normal state asset image pairs 120 provide a source of selected thermal and conventional images of assets that include images that are free of signs of deterioration, wear, wear, damage, or other distinguishing characteristics from images of near-new, operational assets. The image reconstruction module 117 receives image pairs from the normal state asset image pairs 120 as training data, which enables the image reconstruction module 117 to reconstruct the received images into thermal and conventional images of assets having normal state status.

[0037] Historical intervention record 130 includes a collection of inspection, maintenance, repair, and replacement activities performed on asset 140. In some embodiments, historical intervention record 130 includes image pairs, timestamps for when the images were taken, also referred to as when the images were captured, and data related to the intervention. In some embodiments, the intervention is associated with digitized thermal images, conventional images of the asset, and digitally recorded work orders or other documentation that include the intervention activity, duration of the intervention, and costs associated with the intervention. In some embodiments, a fault condition of the asset leads to repair, replacement, maintenance, or other corrective action intervention. In other embodiments, inspections of the asset after a repair intervention and prior to the asset's fault condition create intermediate image pairs and timestamp information.

[0038] The current asset inspection 160 represents a real-time intervention activity that includes creating an image pair including an infrared image and a conventional image of the asset, and a timestamp of the image pair capture and other status information related to the inspection are included in the record of the current asset inspection. In some embodiments, the failure prediction program 200 processes the image pairs created by the current asset inspection 160 to predict the time to a failure condition of the asset from the time frame of the timestamp of the current inspection image pair. In some embodiments, a remotely guided drone performs the real-time asset inspection of the current asset inspection 160, thereby avoiding human exposure to a risk condition. In other embodiments, a camera or set of cameras positioned to capture infrared and conventional images of the asset at specified intervals provides the real-time inspection image pair for the current asset inspection 160.

[0039] Asset 140 performs an operational function upon which a positive outcome depends. In an embodiment of the present invention, operation of asset 140 results in degradation or wear and tear of asset 140, requiring intervention for continued operation of asset 140. In an exemplary embodiment, asset 140 may be, but is not limited to, a motor, a turbine, a vehicle component, a power generation system, or a component of baggage handling equipment.

[0040] 2 is a flowchart illustrating operational steps of a failure prediction program 200, including components of the image reconstruction module 117 and the image similarity module 119, operating in the distributed computing environment of FIG. 1 in accordance with an embodiment of the present invention. The failure prediction program 200 receives a selection of normal status image pairs for an asset. The normal status image pairs include a conventional image and an infrared image of the asset.

[0041] The failure prediction program 200 segments the received normal state image pair and multiple historical image pairs into regions (step 210). The received normal state image pair reflects a near-new type of asset condition. The capture of multiple historical image pairs associated with historical interventions, including inspections performed between instances of the asset's fault state and images taken during interventions to repair or replace the asset, occurs approximately simultaneously. The image pairs include normal visible light and infrared images of the asset. The historical image pairs of the asset include timestamps of the image captures and information related to the specific intervention activity.

[0042] The failure prediction program 200 segments the image pair into a square matrix of regions specified by an "n x n" format, where "n" is a non-zero positive integer. Image segmentation allows for a finer-grained representation of assets and asset components. Each region of the image contains a certain amount of pixels and serves as the basis for calculating a similarity coefficient by performing a similarity function between a region of the image and a corresponding region of a reconstructed version of the first model's image (i.e., the normal-state asset image).

[0043] The failure prediction program 200 includes training a first model to reconstruct image inputs to resemble images of an asset having a normal state status (step 220). The failure prediction program 200 includes an image reconstruction module 117 that receives inputs of images of the asset and reconstructs the images to resemble the normal state images of the asset based on the training, which includes receiving a selection of normal state images of the asset.

[0044] Thus, training the image reconstruction module 117 involves only using images depicting the asset's normal state status as training data. In some embodiments, the reconstruction function of the image reconstruction module 117 of the failure prediction program 200 creates a reference base of the asset in normal states, which provides a reference point for determining similarity coefficients between historical image pairs and real-time images from inspections of the asset. Using image reconstructions of specific image pairs of the asset instead of always reusing one instance of a normal state asset image improves the consistency and accuracy of the similarity coefficients and prevents additional pixel noise due to differences in lighting, angle, and other conditions present when the images are captured.

[0045] The failure prediction program 200 generates a similarity coefficient between each historical image pair and the corresponding reconstructed historical image pair (step 230). The failure prediction program 200 applies a similarity function to a segmented region of each image of an image pair among the plurality of historical image pairs. The similarity function determines a similarity metric between the segmented region of each image of each historical image pair and the segmented region of each image of the historical image pair sent to the first model for reconstructing the image to resemble an image of the asset having a normal condition status close to that performed by the trained model. In some embodiments, determining the Euclidean distance between corresponding regions of the images provides the similarity coefficient metric. In other embodiments, other similarity functions, such as a cosine similarity function, provide the similarity coefficient metric. Embodiments of the present invention are not limited by the type or manner of similarity function applied when determining the similarity coefficient between regions of the images. In embodiments of the present invention, the failure prediction program 200 provides the similarity coefficients of the infrared image and the conventional image through the image similarity module 119.

[0046] The failure prediction program 200 trains a second model to predict the duration to a failure state of the asset (step 240). The failure prediction program 200 utilizes timestamps and other information associated with each image pair of historical interventions of the asset. The similarity coefficients of multiple historical image pairs and the timestamp and intervention information associated with each historical image pair provide a basis for training the second model to correlate similarity coefficients and durations between the timestamps of the historical image pair captures and the timestamps of subsequent failure states of the asset. By training the second model across multiple historical image pairs, including the asset's failure state and inspection of the asset between the failures, the second model forms a correlation between the state represented by the similarity coefficients of the infrared image and the conventional image and the expected time to failure state of the asset.

[0047] In some embodiments, additional information associated with historical interventions of an asset, such as recorded in conjunction with work orders for the interventions, provides a prediction of the duration and cost associated with interventions performed prior to a predicted failure condition of the asset.

[0048] The failure prediction program 200 receives real-time image pairs of an inspection of an asset and segments the received real-time image pairs (step 250). In various embodiments, an asset owner schedules an inspection of the asset while the asset is in operation. The inspection includes capturing infrared and conventional images of the asset, and a real-time image pair associated with the inspection is formed that includes a timestamp of the image pair capture. The failure prediction program 200 receives the real-time image pairs and subsequently processes the image pairs to enable a prediction of the time from the time of image pair capture from the real-time inspection of the asset to a failure condition of the asset. The failure prediction program 200 segments the image pairs into regions.

[0049] The failure prediction program 200 generates real-time similarity coefficients between the real-time image pair and a reconstruction of the image pair from the first model (step 260). The failure prediction program 200 applies a similarity function to the images of the real-time image pair to generate similarity coefficients for the segmented regions. The failure prediction program 200 sends the infrared and conventional images as digitized files to the first model, image reconstruction module 117, which reconstructs the infrared and conventional images from the real-time inspection into images corresponding to the normal condition status of the asset. The failure prediction program 200 generates similarity coefficients between the reconstructed image regions that resemble an asset having a normal condition status and regions of the real-time infrared and conventional images of the inspection image pair.

[0050] The failure prediction program 200 determines a predicted time to a failure state of the asset and presents the predicted time in a notification (step 270). The failure prediction program 200 provides the generated similarity coefficients to a trained second model, which correlates the similarity coefficients and timestamps of the regions of the real-time image pair with the similarity coefficients and timestamps of the regions of the historical image pair used to train the second model. The failure prediction program 200 determines a prediction of the duration from the time of capture of the real-time inspection image pair to the predicted failure state of the asset. In some embodiments of the present invention, the failure prediction program 200 includes the predicted time to a failure state of the asset in a notification presented (e.g., on a display). In some embodiments, the failure prediction program 200 presents the notification to one or more designated recipients by a predetermined delivery mode, such as an SMS message, email, a pop-up display on a monitor associated with the asset, or other notification delivery medium.

[0051] Having determined the predicted time to failure condition of the asset and provided that predicted time in the notification, the failure prediction program 200 terminates. To further clarify aspects of the present invention, a use example is provided to illustrate an embodiment of the present invention.

[0052] In one embodiment, inspections are performed on autonomous elevators to create image pairs including visible light and infrared images. Embodiments of the present invention eliminate the need for repeated human inspections, thus preventing excessive outage time. Eliminating human inspections within elevator shafts also reduces personnel risks and accidents associated with in-person inspections, as imaging can be performed from mounted camera systems or camera capture of asset images from secure or remote locations. Predicting asset time-to-failure conditions reduces or eliminates inspections, maximizing operational time through tighter scheduling of maintenance interventions.

[0053] Embodiments of the present invention provide direct benefits to the inspection of autonomous escalators and moving walkways. Installing image capture sensors / cameras inside the asset allows for accurate prediction of the asset's time to failure condition, enabling efficient scheduling of maintenance intervention activities. This results in operational time and cost savings and promotes safer conditions for human interaction.

[0054] Trains, subways, buses, and trucks can use scheduled stops as an opportunity to capture images of critical asset components, allowing maintenance activities to be precisely scheduled before predicted failure conditions occur. Scheduled maintenance is based not on regular intervals or distance traveled, but on evidence-based data and machine learning, which prevents asset and / or asset component failure conditions from occurring while enabling improved operational efficiency within the vehicle.

[0055] FIG. 3 illustrates a block diagram of components of a computing system including a computing device configured to operatively execute a fault prediction program including the image reconstruction module and image similarity module of FIG. 2, according to one embodiment of the present invention.

[0056] Computing device 305 includes components and functional capabilities similar to those of computing device 110 (FIG. 1) in accordance with the illustrative embodiment of the invention. It should be understood that FIG. 3 is provided as an illustration of one implementation only and is not intended to suggest any limitations with regard to the environments in which different embodiments may be implemented. Many modifications to the depicted environment may be made.

[0057] Computing device 305 includes communications fabric 302, which provides communication between computer processor 304, memory 306, persistent storage 308, communications unit 310, and input / output (I / O) interface 312. Communications fabric 302 may be implemented using any architecture designed to pass data and / or control information between a processor (such as a microprocessor, communications and network processor), system memory, peripheral devices, and any other hardware components in the system. For example, communications fabric 302 may be implemented using one or more buses.

[0058] Memory 306, cache memory 316, and persistent storage 308 are computer-readable storage media. In this embodiment, memory 306 includes random access memory (RAM) 314. In general, memory 306 may include any suitable volatile or non-volatile computer-readable storage media.

[0059] In one embodiment, the failure prediction program 200 is stored in persistent storage 308 for execution by one or more of the respective computer processors 304 via one or more memories of memory 306. In this embodiment, persistent storage 308 includes a magnetic hard disk drive. As an alternative to, or in addition to, a magnetic hard disk drive, persistent storage 308 may include a solid-state hard drive, a semiconductor storage device, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, or any other computer-readable storage medium capable of storing program instructions or digital information.

[0060] The media used by persistent storage 308 may be removable. For example, a removable hard drive may be used for persistent storage 308. Other examples include optical and magnetic disks, thumb drives, and smart cards that are inserted into a drive for transfer onto another computer-readable storage medium that is also part of persistent storage 308.

[0061] The communications unit 310, in these examples, provides for communication with other data processing systems or devices, including resources of the distributed computing environment 100. In these examples, the communications unit 310 includes one or more network interface cards. The communications unit 310 may provide communication by using either or both physical and wireless communications links. The failure prediction program 200 may be downloaded to the persistent storage 308 through the communications unit 310.

[0062] The I / O interface 312 allows for the input and output of data with other devices that may be connected to the computing system 300. For example, the I / O interface 312 may provide a connection to an external device 318, such as a keyboard, keypad, touch screen, or any other suitable input device or combination thereof. The external device 318 may also include a portable computer-readable storage medium, such as a thumb drive, a portable optical or magnetic disk, and a memory card. Software and data used to practice embodiments of the present invention, such as the failure prediction program 200, may be stored on such a portable computer-readable storage medium and loaded into the persistent storage 308 via the I / O interface 312. The I / O interface 312 also connects to a display 320.

[0063] Display 320 provides a mechanism for displaying data to a user and may be, for example, a computer monitor.

[0064] The foregoing description is an example of an embodiment of the present invention, and variations and substitutions can be made in implementations without departing from the novel aspects of the embodiments.

[0065] The programs described herein are identified based on the application in which they are implemented in particular embodiments of the invention, but it should be understood that any particular program nomenclature herein is used merely as a matter of convenience, and thus the invention should not be limited to use with any particular application identified and / or suggested by such nomenclature.

[0066] The present invention may be a system, method, and / or computer program product integrated at any possible level of technical detail, and may include a computer-readable storage medium (or multiple computer-readable storage media) having computer-readable program instructions for causing a processor to implement aspects of the present invention.

[0067] A computer-readable storage medium may be a tangible device capable of retaining and storing instructions for use by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the above. A non-exhaustive list of more specific examples of computer-readable storage media includes the following: portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital versatile disk (DVD), memory sticks, floppy disks, mechanically encoded devices such as punch cards or raised structures in grooves with instructions recorded on them, and any suitable combination of the above. As used herein, computer-readable storage media should not be construed as being ephemeral signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g., light pulses through fiber optic cable), or electrical signals transmitted over electrical wires.

[0068] The computer-readable program instructions described herein may be downloaded from a computer-readable storage medium to each computing / processing device or to an external computer or storage device via a network, such as the Internet, a local area network, a wide area network, or a wireless network, or a combination thereof. The network may include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to a computer-readable storage medium within the respective computing / processing device for storage.

[0069] Computer-readable program instructions for carrying out the operations of the present invention may be either assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, configuration data for integrated circuits, or source or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk®, C++, and procedural programming languages ​​such as the "C" programming language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer as a standalone software package, partially on the user's computer, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, to carry out aspects of the present invention, electronic circuitry including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may execute computer-readable program instructions by utilizing state information of the computer-readable program instructions to personalize the electronic circuitry.

[0070] Aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0071] These computer-readable program instructions may be provided to a processor of a computer or other programmable data processing apparatus to produce a machine, such that the instructions, when executed by the processor of the computer or other programmable data processing apparatus, produce means for performing the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams. These computer-readable program instructions may also be stored on a computer-readable medium, such that the computer-readable storage medium on which the instructions are stored comprises an article of manufacture containing instructions implementing aspects of the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams, and can direct a computer, programmable data processing apparatus, or other device, or combination thereof, to function in a particular manner.

[0072] The computer-readable program instructions may also be loaded into a computer, other programmable data processing apparatus, or other device to create a computer-implemented process that causes the computer, other programmable apparatus, or other device to perform a series of operational steps, such that the instructions, which execute on the computer, other programmable apparatus, or other device, perform the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams.

[0073] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of instructions, including one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be accomplished as a single step, or may be executed concurrently, substantially concurrently, partially, or entirely in a time-overlapping manner, or in some cases, the blocks may be executed in the reverse order, depending on the functionality involved. It should also be noted that each block in the block diagrams and / or flowchart diagrams, and combinations of blocks in the block diagrams and / or flowchart diagrams, may be implemented by a dedicated hardware-based system that performs the specified functions or actions or a combination of dedicated hardware and computer instructions.

Claims

1. 1. A computerized method for predicting a time to failure condition of an asset, comprising: receiving, by one or more processors, a plurality of historical image pairs of the asset, including image pairs of the asset in a fault condition, each image pair including an infrared image and a corresponding conventional image; receiving, by the one or more processors, a first model trained to reconstruct each image in an image pair of the asset into a normal condition status of the asset; generating, by the one or more processors, for each of the plurality of historical image pairs, a similarity coefficient between the historical image pair and a reconstruction of the historical image pair using the first model, the historical image pair including a timestamp of capture of the historical image pair; receiving, by the one or more processors, a second model trained to predict a duration until a fault condition of the asset occurs based on the similarity coefficient and timestamps of capture of the plurality of historical image pairs, respectively; generating, by the one or more processors, in response to receiving a real-time image pair of the asset, a real-time similarity coefficient between the real-time image pair and the real-time image pair reconstructed by the first model, the real-time image pair including a timestamp of capture of the real-time image pair; calculating, by the one or more processors, a predicted time to the fault condition of the asset based on a correlation between the similarity coefficient of the real-time image pair and the timestamp of the capture of the real-time image pair, a historical image pair among the plurality of historical image pairs having a corresponding similarity coefficient, and a time delta determined from the timestamp of the capture of the real-time image pair and a timestamp of the fault condition of the asset following the historical image pair having the corresponding similarity coefficient; A method comprising:

2. 2. The method of claim 1, wherein the first model is a set of self-encoding convolutional neural networks that receive as inputs infrared and conventional images of the asset.

3. Segmenting, by the one or more processors, image pairs of the asset, the plurality of historical image pairs, and the real-time image pairs into regions; generating, by the one or more processors, similarity coefficients for the sampling of image pairs of the asset, the plurality of historical image pairs, and the regions of the real-time image pairs; The method of claim 1 further comprising:

4. The method of claim 3 , wherein the regions are segmented into square matrices of regions.

5. The method of claim 1 , wherein the similarity coefficient comprises a multidimensional vector of Euclidean distances between segmented regions of each image of the image pair.

6. the first model and the second model are receiving, by one or more processors, a sampling of image pairs of the asset from the plurality of historical image pairs having normal condition statuses of the asset, the normal condition status including a range of conditions of the asset from a newly replaced condition to a faulty condition of the asset; training, by the one or more processors, the first model to reconstruct each image of an image pair based on training data from the sampling of image pairs of the asset having the normal condition status; generating, by the one or more processors, a second model for predicting a duration until the failure condition of the asset; training, by the one or more processors, the second model based on the similarity coefficients of the plurality of historical image pairs of the asset and timestamps of capture of the plurality of historical image pairs, respectively; The method of claim 1 further comprising:

7. The method of claim 1 , wherein the plurality of received historical image pairs of the asset includes a range of states between the normal state status and the fault state of the asset.

8. 1. A computer program for predicting a time to failure condition of an asset, the computer comprising: receiving a plurality of historical image pairs of the asset, each image pair including an image pair of the asset in a fault condition, the image pair including an infrared image and a corresponding conventional image; receiving a first model trained to reconstruct each image of the image pair of the asset into a normal condition status of the asset; generating, for each of the plurality of historical image pairs, a similarity coefficient between the historical image pair and a reconstruction of the historical image pair according to the first model, the historical image pair including a timestamp of capture of the historical image pair; receiving a second model trained to predict a duration until a fault condition of the asset occurs based on the similarity coefficient and timestamps of capture of the plurality of historical image pairs, respectively; in response to receiving a real-time image pair of the asset, generating a real-time similarity coefficient between the real-time image pair and the real-time image pair reconstructed by the first model, the real-time image pair including a timestamp of capture of the real-time image pair; calculating a predicted time to the fault condition of the asset based on a correlation between the similarity coefficient of the real-time image pair and the timestamp of the capture of the real-time image pair, a historical image pair of the plurality of historical image pairs having a corresponding similarity coefficient, and a time delta determined from the timestamp of the capture of the real-time image pair and a timestamp of the fault condition of the asset following the historical image pair having the corresponding similarity coefficient; A computer program for executing the above.

9. The first model comprises: receiving a sampling of image pairs of the asset having normal condition statuses from the plurality of historical image pairs of the asset, the normal condition status including a range of conditions of the asset from a newly replaced condition to a faulty condition of the asset; training the first model to reconstruct each image of an image pair based on training data from the sampling of image pairs of the asset having the normal condition status; generating a second model for predicting the duration to the failure condition of the asset; training the second model based on the similarity coefficients of the plurality of historical image pairs of the asset and timestamps of capture of the plurality of historical image pairs, respectively; 9. The computer program product of claim 8, which causes the computer program product to execute the following:

10. 1. A computer system for predicting a time to failure condition of an asset, comprising: one or more computer processors; one or more computer-readable storage media and program instructions collectively stored on the one or more computer-readable storage media, the program instructions comprising: program instructions for receiving a plurality of historical image pairs of the asset, including an image pair of the asset in a fault condition, each image pair including an infrared image and a corresponding conventional image; receiving a first model trained to reconstruct each image of the image pair of the asset into a normal condition status of the asset; program instructions for generating, for each of the plurality of historical image pairs of the asset, a similarity coefficient between the historical image pair and a reconstruction of the historical image pair according to the first model, the historical image pair including a timestamp of capture of the historical image pair; receiving a second model trained to predict a duration until a fault condition of the asset occurs based on the similarity coefficient and timestamps of capture of the plurality of historical image pairs, respectively; program instructions for generating, in response to receiving a real-time image pair of the asset, a real-time similarity coefficient between the real-time image pair and the real-time image pair reconstructed by the first model, the real-time image pair including a timestamp of the capture of the real-time image pair; program instructions for calculating a predicted time to the fault condition of the asset based on a correlation between the similarity coefficient of the real-time image pair and the timestamp of the capture of the real-time image pair, a historical image pair of the plurality of historical image pairs of the asset having a corresponding similarity coefficient, and a time delta determined from the timestamp of the capture of the real-time image pair and a timestamp of the fault condition of the asset following the historical image pair having the corresponding similarity coefficient; 1. A computer system comprising:

11. program instructions for sampling image pairs of the asset, segmenting the plurality of historical image pairs and the real-time image pairs into regions, wherein the regions are segmented into square matrices of regions; program instructions for generating similarity coefficients for the regions of the sampling of image pairs of the asset, the plurality of historical image pairs, and the real-time image pairs; 11. The computer system of claim 10, further comprising:

12. The first model comprises: program instructions for receiving, from the plurality of historical image pairs of the asset, a sampling of image pairs of the asset having a normal condition status, including a range of conditions of the asset from a newly replaced condition to a failed condition of the asset; program instructions for training the first model to reconstruct each image of an image pair based on training data from the sampling of image pairs of the asset having the normal condition status; program instructions for generating a second model for predicting a duration to the failure condition of the asset; program instructions for training the second model based on the similarity coefficients of the plurality of historical image pairs of the asset and timestamps of capture of the plurality of historical image pairs, respectively; 12. The computer system of claim 11, further comprising:

Images