Ai-based joint prediction of repair time and actions from textual fault reports

Abstract

A system applies an AI-based model pipeline for joint prediction of repair time and repair actions based on a received fault report related to manufacturing automation machines and equipment. Model pipeline includes a first encoder configured as a transformer neural network trained to transform fault report text to a latent fault report representation that groups semantically similar fault texts. Another transformer encoder is trained to transform the latent fault report representation to a latent repair report representation. Repair time prediction module with a feed forward neural network is trained to generate repair time predictions from the latent repair report representation against actual repair times. Repair action prediction module configured as a transformer decoder neural network is trained to generate repair action predictions from the latent repair report representation against actual repair actions. End-to-end training of the pipeline is achieved by minimizing a combined loss function of the joint prediction.

Description

AI-BASED JOINT PREDICTION OF REPAIR TIME AND ACTIONS FROM TEXTUAL FAULT REPORTSTECHNICAL FIELD

[0001] This application relates to artificial intelligence (Al). More particularly, this application relates to applying Al-based joint prediction of repair time and repair actions to machinery and equipment in an automation system from textual fault reports.BACKGROUND

[0002] Machines and other equipment used for manufacturing automation need to be maintained regularly and, even with regular maintenance, are bound to fail at certain times, due to wear and tear or other causes. On the level of a production plant, effective repair planning and scheduling is essential to ensure that downtimes remain short and that a high level of overall equipment efficiency (OEE) can be maintained during and after the event of a failure.

[0003] The ability to perform effective repair planning depends on how well it can be anticipated at failure time, how long the repair will take and what kind of repair will be necessary. An accurate repair time estimate will allow the planning team to decide whether production should be shifted to an alternative machine or line, as well as other remedial actions. A recommendation for repair actions likely to be needed to be performed could assist the maintenance team in preparing for the repair and performing it more efficiently (e.g., by ordering parts in advance or bringing the correct repair equipment). Especially for mechanical repairs, knowing in advance which equipment might or might not be necessary, could save time and resources. To this end, a technical problem to be solved is predicting both the repair time (in hours) and required repair actions, based on a textual fault report.

[0004] While some AT-based methods have been proposed to solve either one of these problems, existing methods do not consider correlations between the two desired predictions and thus fail to provide accurate and consistent predictions in many cases. For example, solving a prediction of each problem in parallel can produce inconsistent results, such as predicting a quick repair time for a repair action that is complicated. Such errors can be caused by each prediction pipeline relying on different data domains.

[0005] Predictive maintenance is an increasingly common Al-based approach to predicting machine faults early, before they happen, based on the history of the machine, including information such as current uptime and the previous fault history. Predicting failures before they happen is a much more challenging problem and prone to yield much less uncertain results.

[0006] In most cases, repair time and action prediction are currently solved manually by experts evaluating the fault report and making estimations. This approach, however, requires large manual effort and constitutes a bottleneck in production environments with a large number of equipment and failure cases. Further, to be able to make accurate estimations, experts need have a certain amount of experience and be familiar with a larger number of very different types of equipment.

[0007] Beyond manual prediction of repair time and repair action, several Al-based methods have been proposed in the literature. However, these methods solve either problem (repair time prediction or repair action prediction) individually, leading to potentially inconsistent and thus inaccurate predictions.SUMMARY

[0008] In contrast to the conventional approach described above, this disclosure proposes a joint prediction method that predicts repair time and actions together in a single Al-based pipeline. By doing so, the proposed framework can leverage relationships between repair time and repair actions. For instance, a certain predicted repair time might rule out several possible repair actions that would require much more time. On the other hand, knowing what repair actions would likely need to be performed, a much more accurate prediction for the repair time can be made.

[0009] In an aspect, a system applies an Al-based model pipeline for joint prediction of repair time and repair actions based on a received fault report related to manufacturing automation machines and equipment. Model pipeline includes a first encoder configured as a transformer neural network trained to transform fault report text to a latent fault report representation that groups semantically similar fault texts. Another transformer encoder is trained to transform the latent fault report representation to a latent repair report representation. Repair time prediction module with a feed forward neural network is trained to generate repair time predictions from the latent repair report representation against actual repair times. Repair action prediction module configured as a transformer decoder neural network is trained to generate repair action predictions from the latent repair report representation against actual repair actions. End-to-end training of the pipeline is achieved by minimizing a combined loss function of the joint prediction.BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Non-limiting and non-exhaustive embodiments of the present disclosure are described with reference to the following FIGURES, wherein like reference numerals refer to like elements throughout the drawings unless otherwise specified.

[0011] FTG. 1 illustrates an example of an AT-based pipeline during pre-training for joint prediction of repair time and repair actions to machinery in an automation system in accordance with embodiments of this disclosure.

[0012] FIG. 2 shows a flowchart example of a pre-training and training process for an AI- based pipeline for joint prediction of repair time and repair actions to machinery in an automation system in accordance with embodiments of this disclosure.

[0013] FIG. 3 illustrates an example of an Al-based pipeline during end-to-end training for joint prediction of repair time and repair actions to machinery in an automation system in accordance with embodiments of this disclosure.

[0014] FIG. 4 illustrates an example of a trained Al-based pipeline for joint prediction of repair time and repair actions to machinery in an automation system in accordance with embodiments of this disclosure.

[0015] FIG. 5 is an example of a computing environment within which embodiments of the disclosure may be implemented.DETAILED DESCRIPTION

[0016] FIG. 1 illustrates a framework example for an Al-based model pipeline for joint prediction of repair time and repair actions based on a fault report related to machinery in an automation system in accordance with embodiments of this disclosure. The innovation provided by the disclosed embodiments includes construction of model capable of learning to make accurate predictions for repair time and repair actions, both predictions based on the same latent representation of fault report text. As shown in FIG. 1, model pipeline 100 is configured as an AI- based pipeline having Encoder 101, Encoder 102, Encoder 103, repair time prediction (RTP)module 11 1 , and repair action prediction (RAP) module 1 12, each of which are trained as part of a multi-step process. Each of Encoders 101, 102 and 103 is configured as a transformer neural network model that can automatically transform one type of input into another type of output. These encoders encode input words of text as a vector using embedding layers and transformer layers. RTP module 111 is configured as a simple feed forward neural network. RAP module 112 is configured as a transformer neural network decoder.

[0017] The source information for the trained model pipeline 100 is historical fault report text x in natural language (i.e., a sequence of language tokens). The typical fault report information includes a description of the fault (e.g., a symptom), description of the affected machine or component. Basic preprocessing of this text of fault report text x may be performed (e.g., stop word removal and / or stemming).

[0018] Dependencies for pre-training of the transformer neural networks during initialization of model pipeline 100 are shown by arrows. During initialization and pre-training phase, other given input includes a set of historical failure cases. For each failure case, the accompanying data includes a fault report and a corresponding repair report that was generated in response. Hence, in addition to the fault text x, the actual repair time yf G IR and a natural language description of the performed repair actionsis provided, taken from the corresponding repair report. Similar to the fault text x, some basic text pre-processing is applied to the repair action description.

[0019] FIG. 2 shows a flowchart example of pre-training and training process for an Al-based pipeline for joint prediction of repair time and repair actions to machinery in an automation system in accordance with embodiments of this disclosure. The following steps of FIG. 2 are described with reference to the pipeline 100 components shown in FIGs. 1 and 3.

[0020] Initialization of model pipeline 100 begins with pre-training of the transformer Encoders 101, 102, 103 and transformer decoder RAP 112. At 201, Encoder 101 is initialized with a publicly available pre-trained transformer model (e.g., BERT, RoBERT a) that is already able to learn semantic representations of general texts. It is then fine-tuned in a self- supervised way on the set of historical fault texts x where Encoder 101 maps fault texts x to a semantic embedding vector denoted as6 Bdl. which is a latent numeric vector representation of fault report texts. Here, a “numeric” vector simply means a real-valued vector. The transformer Encoder 101 takes a representation of an input text (basically a sequence of tokens) and maps that to a real-valued embedding vector. The similarities between these vectors (e.g., based on Euclidean distance) correspond to the sematic similarities of the texts that correspond to these vectors. Hence, Encoder 101 groups semantically similar fault texts within the learned latent space using the numeric vectors. Accordingly, Encoder 101 performs a transformation of input text into the latent vector representation hi.

[0021] At 202, Encoder 103 is pre-trained as a transformer model in a similar manner as Encoder 101, except the source of training data for self-supervised fine-tuning is a set of historical repair action descriptionsobtained from repair reports instead of the fault reports. The historical repair action descriptions correspond to the fault report texts, in that for each historical fault text, a corresponding repair report is available. The output of Encoder 103 during pre-training is latent textual vector representation of repair reports, denoted as h2£ IRd2. Accordingly, Encoder 103 performs a transformation of input textinto the latent vector representation hi.

[0022] In an embodiment, both Encoder 101 and Encoder 103 are pre-trained independently and in parallel.

[0023] At 203a, using vectors h generated by Encoder 101 and vectors h2generated byEncoder 103, Encoder 102 is pre-trained to map embedding vectorof a particular historical fault text x (of a particular fault report), to the embedding vector h2of the corresponding historical repair report. The learned transformation for Encoder 102 is a mapping from one latent vector space to another latent vector space. In the transformed latent vector space, fault texts with similar repair times and actions are ultimately grouped together.

[0024] When pre-training Encoder 102 at step 203a, it is important to maintain correspondence between fault text and related repair report information belonging to the same particular fault case. For a certain set (or batch) of historical cases, Encoder 101 embeds the fault texts and Encoder 102 embeds the repair reports. This process can be performed in parallel. Alternatively, the embeddings could be pre-computed after pre- training of Encoder 101 and Encoder 103 has completed and before pre-training of Encoder 102 has started. Afterwards, Encoder 102 is trained to map the embedded fault texts to their corresponding embedded fault reports.

[0025] At step 203b, the decoder of RAP module 112 is pre-trained based on the learned repair actions embedded in latent repair report vectors h from pre-trained Encoder 103. The objective of pre-training RAP module 112 is to reliably map latent repair report representations / 12 back to natural text descriptions of repair actions. Empirically, a least complex model that can still reliably perform this mapping is determined for RAP module 112. In an embodiment, with reference to FIG. 1, the pre- training of RAP module 112 begins with feeding actual repair action textto pre-trained Encoder 103 yielding latent representation h% then fed to decoder RAP module 112 for learning the decoding of the embedded repair action text to predicted repair action text j2.

[0026] Once pre-training for initialization is concluded, at 204, the model pipeline 100 is trained end-to-end towards the joint prediction objective. The flow for the training process isshown in FIG. 3. The training process updates all model parameters except for the parameters of Encoder 101 and Encoder 103. The parameters of Encoder 101 remain frozen after pre-training and Encoder 103 is no longer required after model initialization. Training is performed in a supervised way based on actual repair times and repair actions from the historical case data. A combined loss function for the joint prediction is minimized using a backpropagation algorithm. During the forward pass, a fault text x as training input is mapped to its latent fault report representationand latent repair report representation h2using Encoder 101 and Encoder 102, respectively. Based on latent repair report representation h2, feed forward neural network of RTP module 111 is trained to predict repair time y1as a scalar output yxG R . The predicted repair time yxis compared to the actual repair time jq for first loss Li. Additionally, from the same latent repair report representation h2, a predicted repair action description y2is generated using transformer decoder of RAP module 112 and compared to the actual repair action description y^ for second loss Li. Training of RAP module 112 is controlled to avoid over-parametrizing the transformer neural network decoder. In an embodiment, the separate loss functions are combined using combined loss function L. Balancing the separate losses Li andL2 for each prediction can be balanced by a hyperparameter. For example, a combined loss function L can be expressed by the following:yj), where a cross-entropy loss I is computed on a bag of words encoded representations of the predicted repair action description y2and actual repair action descriptions y and hyperparameter a balances the two loss functions for repair time prediction y and repair action prediction y2, respectively. By minimizing the combined loss function L, the model learns latentrepresentations of the fault text that allow for consistent and highly accurate prediction of both repair time and repair actions.

[0027] FTG. 4 illustrates an example of a trained AT-based pipeline for joint prediction of repair time and repair actions to machinery in an automation system in accordance with embodiments of this disclosure. With model pipeline 100 now trained, joint prediction for a new case is performed by mapping the fault text x to its latent fault report representation hi and latent repair report representation / ?2 using Encoder 101 and Encoder 102 as shown in FIG. 4, and then feeding hi through RTP module 111 and RAP module 112 to jointly obtain predictions for the repair time y±and action description y2, respectively.

[0028] FIG. 5 shows an example of a computer environment within which embodiments of the disclosure may be implemented. A computing device 510 includes one or more processors 515 and memory 511 (e.g., a non-transitory computer readable media) on which is stored various computer applications, modules or executable programs. The processors 515 may include one or more central processing units (CPUs), graphical processing units (GPUs), or any other processor known in the art. In an embodiment, Encoders 101, 102 103 and RAP decoder 112 are stored and trained / executed on a GPU for accelerated processing.

[0029] More generally, a processor as described herein is a device for executing machine- readable instructions stored on a computer readable medium, for performing tasks and may comprise any one or combination of, hardware and firmware. A processor may also comprise memory storing machine-readable instructions executable for performing tasks. A processor acts upon information by manipulating, analyzing, modifying, converting or transmitting information for use by an executable procedure or an information device, and / or by routing the information to an output device. A processor may use or comprise the capabilities of a computer, controller ormicroprocessor, for example, and be conditioned using executable instructions to perform special purpose functions not performed by a general purpose computer. A processor may include any type of suitable processing unit including, but not limited to, a central processing unit, a microprocessor, a Reduced Instruction Set Computer (RISC) microprocessor, a Complex Instruction Set Computer (CISC) microprocessor, a microcontroller, an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), a System-on-a-Chip (SoC), a digital signal processor (DSP), and so forth. Further, the processor(s) 520 may have any suitable microarchitecture design that includes any number of constituent components such as, for example, registers, multiplexers, arithmetic logic units, cache controllers for controlling read / write operations to cache memory, branch predictors, or the like. The microarchitecture design of the processor may be capable of supporting any of a variety of instruction sets. A processor may be coupled (electrically and / or as comprising executable components) with any other processor enabling interaction and / or communication there-between. A user interface processor or generator is a known element comprising electronic circuitry or software or a combination of both for generating display images or portions thereof. A user interface comprises one or more display images enabling user interaction with a processor or other device.

[0030] In an embodiment, memory 511 includes one or more of the following modules: transformer Encoders 101, 102, 103, RTP module 111 , and transformer decoder RAP module 112. As shown in FIG. 5, as an alternative computer implementation of Encoders 101, 102, 103, RTP 111, RAP 112, one or more of such modules may be deployed as cloud-based or web-based operations in one or more remote modules 541, or as a divided operation shared by local modules101 , 102, 103, 1 1 1 , 1 12, and web-based modules 541.

[0031] A network 560, such as a local area network (LAN), wide area network (WAN), or an internet based network, connects training data 551 to modules 101, 102, 103, 111, 112 of computing device 510 and to modules 541, 542. Training data 551 corresponds to historical fault reports and repair reports, as well as latent representations hi, hi generated during initialization of model pipeline 100.

[0032] User interface module 514 provides an interface between modules 101, 102, 103, 111, 112 and user interface 530 devices, such as display device 531 and user input device 532. GUI engine 513 drives the display of an interactive user interface on display device 531, allowing a user to receive visualizations of analysis results and assisting user entry of learning objectives and domain constraints for modules 101, 102, 103, 111, 112, 541.

[0033] Computer readable medium instructions for carrying out operations of the present disclosure may be assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, or either source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C++ or the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The computer readable program instructions may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example,programmable logic circuitry, field-programmable gate arrays (FPGA), or programmable logic arrays (PLA) may execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present disclosure.

[0034] Aspects of the present disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, may be implemented by computer readable medium instructions.

[0035] The program modules, applications, computer-executable instructions, code, or the like depicted in FIG. 5 as being stored in the system memory 511 are merely illustrative and not exhaustive and that processing described as being supported by any particular module may alternatively be distributed across multiple modules or performed by a different module. In addition, various program module(s), script(s), plug-in(s), Application Programming Interface(s) (API(s)), or any other suitable computer-executable code hosted locally on the computer system 510, remote network devices storing modules 541, 542 and / or hosted on other computing device(s) accessible via one or more of the network(s) 560, may be provided to support functionality provided by the program modules, applications, or computer-executable code and / or additional or alternate functionality. Further, functionality may be modularized differently such that processing described as being supported collectively by the collection of program modules depicted in FIG. 5 may be performed by a fewer or greater number of modules, or functionality described as being supported by any particular module may be supported, at least in part, by another module. Inaddition, program modules that support the functionality described herein may form part of one or more applications executable across any number of systems or devices in accordance with any suitable computing model such as, for example, a client-server model, a peer-to-peer model, and so forth. In addition, any of the functionality described as being supported by any of the program modules depicted in FIG. 5 may be implemented, at least partially, in hardware and / or firmware across any number of devices.

[0036] It should further be appreciated that the computer system 510 may include alternate and / or additional hardware, software, or firmware components beyond those described or depicted without departing from the scope of the disclosure. More particularly, it should be appreciated that software, firmware, or hardware components depicted as forming part of the computer system 510 are merely illustrative and that some components may not be present or additional components may be provided in various embodiments. While various illustrative program modules have been depicted and described as software modules stored in system memory 511, it should be appreciated that functionality described as being supported by the program modules may be enabled by any combination of hardware, software, and / or firmware. It should further be appreciated that each of the above-mentioned modules may, in various embodiments, represent a logical partitioning of supported functionality. This logical partitioning is depicted for ease of explanation of the functionality and may not be representative of the structure of software, hardware, and / or firmware for implementing the functionality. Accordingly, it should be appreciated that functionality described as being provided by a particular module may, in various embodiments, be provided at least in part by one or more other modules. Further, one or more depicted modules may not be present in certain embodiments, while in other embodiments, additional modules not depicted may be present and may support at least a portion of the described functionality and / or additionalfunctionality. Moreover, while certain modules may be depicted and described as sub-modules of another module, in certain embodiments, such modules may be provided as independent modules or as sub-modules of other modules.

[0037] Although specific embodiments of the disclosure have been described, one of ordinary skill in the art will recognize that numerous other modifications and alternative embodiments are within the scope of the disclosure. For example, any of the functionality and / or processing capabilities described with respect to a particular device or component may be performed by any other device or component. Further, while various illustrative implementations and architectures have been described in accordance with embodiments of the disclosure, one of ordinary skill in the art will appreciate that numerous other modifications to the illustrative implementations and architectures described herein are also within the scope of this disclosure. In addition, it should be appreciated that any operation, element, component, data, or the like described herein as being based on another operation, element, component, data, or the like can be additionally based on one or more other operations, elements, components, data, or the like. Accordingly, the phrase “based on,” or variants thereof, should be interpreted as “based at least in part on.”

[0038] The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions noted in the block may occur out of the order noted in the Figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in thereverse order, depending upon the functionality involved. Tt will also be noted that each block of the block diagrams and / or flowchart illustration, and combinations of blocks in the block diagrams and / or flowchart illustration, can be implemented by special purpose hardware-based systems that perform the specified functions or acts or carry out combinations of special purpose hardware and computer instructions.

Claims

CLAIMSWhat is claimed is:

1. A computer based system for joint prediction of repair time and repair actions for machines and equipment used for manufacturing automation based on a received fault report, comprising: a processor; and a non-transitory memory having stored thereon modules executed by the processor, the modules comprising: an Al-based model pipeline comprising: a first encoder configured as a transformer neural network trained to transform fault report text to a latent fault report representation that groups semantically similar fault texts; a second encoder configured as a transformer neural network trained to transform the latent fault report representation to a latent repair report representation; a repair time prediction module configured as a feed forward neural network trained to generate repair time predictions from the latent repair report representation using a first loss function based on actual repair times; and a repair action prediction module configured as a transformer decoder neural network trained to generate repair action predictions from the latent repair report representation using a second loss function based on actual repair actions; wherein the model pipeline is trained end-to-end by minimizing a combined loss function of the first loss function and the second loss function.

2. The system of claim 1, further comprising: a third encoder configured as a transformer neural network pre-trained to transform a set of historical repair action descriptions to a latent textual vector representation of repair reports;wherein the second encoder is pre-trained by mapping the latent fault report representation to the latent repair report representation.

3. The system of claim 2, wherein the third encoder is excluded from the end-to-end training of the model pipeline.

4. The system of claim 2, wherein the repair action prediction module is pre-trained using the latent textual vector representation of repair reports from the pre-trained third encoder.

5. The system of claim 1, wherein the first encoder is pre-trained to learn semantic representations of general texts and fine-tuned in a self- supervised way on the set of historical fault texts.

6. The system of claim 1, wherein parameters of the first encoder remain frozen during the end-to-end training of the model pipeline.

7. The system of claim 1, wherein the combined loss function balances the first loss function and the second loss function using a hyper-parameter and a cross-entropy loss computed on a bag of words encoded representations of the predicted repair action description and the actual repair action descriptions.

Images