System and method for predicting gear damage in a transmission using n-dimensional multi modal ai approach

Abstract

An automated system and method for assessing the longevity of a transmission gear are provided. In Stage 1 of the method, the system provides a prediction of what the condition of a gear will be at 100% completion of a test cycle based on data obtained when the test cycle is only partially complete. In Stage 2 of the method, the system predicts what the remaining useful life (RUL) of the gear will be after the test cycle reaches 100% completion, based on either the forecasted prediction made during Stage 1 regarding what the condition of the gear will be at 100% completion of the test cycle or on data available after the test cycle actually reaches 100% completion. The RUL prediction made during Stage 2 specifies both the type of distress that the gear is expected to incur and when the distress is expected to occur.

Description

SYSTEM AND METHOD FOR PREDICTING GEAR DAMAGE IN A TRANSMISSION USING N-DIMENSIONAL MULTI MODAL Al APPROACHCROSS REFERENCE TO RELATED APPLICATIONS

[0001] This patent application claims the priority benefit of Indian Provisional Patent Application Serial No. 202411074368, filed October 1, 2024 entitled, “System And Method For Predicting Gear Damage In A Transmission Using N-Dimensional Multi Modal Al Approach”, the contents of which are incorporated by reference.FIELD OF THE INVENTION

[0002] The disclosed concept relates generally to vehicle transmissions, and in particular, to systems and methods for assessing the condition of gears in vehicle transmissions.BACKGROUND OF THE INVENTION

[0003] When gears in a transmission fail, it is most often due to fatigue and overloading. The most common form of distress and failure is actual breakage, but other modes of distress include surface fatigue known as pitting, normal and abnormal wear, and plastic flow. Common types of damage sustained by gears during testing include: bumps or swells on the teeth, cracks, and pitting of the gear after X hours. The aforementioned factors, in combination with the material degradation of a gear over its service lifespan, make a gearbox prone to fatigue, especially in harsh working environments. The interaction between gear fatigue and gear dynamics often results in high complexity measurements being required to predict the remaining useful life (RUL) of each gear.

[0004] Prior to being made commercially available, commercial and industrial (heavy duty) gears undergo a test cycle to test their performance. The duration of a test cycle is the minimum period of time for which a gear in a transmission is expected to function optimally. A test cycle for a commercial gear typically lasts in the range of nine to twelve months while a test cycle for a heavy duty gear can last as long as two years. After completion of the test cycle, the RUL of the gear is determined by way of a manual visual inspection. Existing solutions for determining gear condition after test cycle completion often rely either only on image data or only on vibration data. A typical process for assessing longevity of a commercial or industrialgear includes executing a complete test cycle and subsequently predicting RUL, which involves capturing images of the gear during the test cycle, manually inspecting the images, and removing and disassembling the gear from the transmission to determine the extent of damage identified from the images. This process is both time-consuming and prone to errors.

[0005] There is thus room for improvement in methods and systems for assessing the longevity of gears in a transmission.SUMMARY OF THE INVENTION

[0006] These needs, and others, are met by a fully automated system and method for assessing the longevity of a gear in a transmission. In a first stage of the method, Stage 1, the system provides a prediction of what the condition of a gear will be at 100% completion of a test cycle based on data obtained when the test cycle is only partially complete. In Stage 2 of the method, the system predicts what the remaining useful life (RUL) of the gear will be after the test cycle reaches 100% completion, based on either the forecasted prediction made during Stage 1 regarding what the condition of the gear will be at 100% completion of the test cycle or on data that becomes available once the test cycle actually reaches 100% completion. The RUL prediction made during Stage 2 specifies both the type of distress that the gear is expected to incur during the RUL and when the distress is expected to occur.

[0007] In accordance with one aspect of the disclosed concept, a method for determining the longevity of a gear used in a transmission is controller- implemented and has a first stage and a second stage. The first stage includes forecasting, at a first time that is earlier than 100% completion of a test cycle for the gear, predicted image data corresponding to 100% completion of the test cycle. The forecasting is performed using image data and sensor data collected for the gear up through the first time, the sensor data being time-variant. The second stage includes: gathering completed test cycle multi modal data including completed test cycle image data and completed test cycle sensor data, with the completed multi modal data being obtained for 100% completion of the test cycle for the gear and the completed test cycle sensor data being timevariant; encoding the completed test cycle image data and the completed test cycle sensor data to create encoded test cycle image data and encoded test cycle sensor data; running a fusion model that fuses the encoded test cycle image data and encoded test cycle sensor data into a single fusion representation; performing multi class classification with object detection on the singlefusion representation; and based on a result of the multi class classification with object detection, classifying the gear as falling within one of a plurality of gear failure modes and predicting a remaining useful life of the gear. When image data corresponding to actual 100% completion of the test cycle is available, the first stage is optional such that the second stage can be performed without first performing the first stage.

[0008] In accordance with another aspect of the disclosed concept, a system for determining the longevity of a gear used in a transmission includes: a number of cameras positioned to collect image data for the gear, a number of sensors positioned to collect timevariant sensor data for the gear, and a controller. The controller is configured to execute a method having a first stage and a second stage. The first stage includes forecasting, at a first time that is earlier than 100% completion of a test cycle for the gear, predicted image data corresponding to 100% completion of the test cycle. The forecasting is performed using the image data and the sensor data collected for the gear up through the first time. The second stage includes: gathering completed test cycle multi modal data including completed test cycle image data and completed test cycle sensor data, with the completed multi modal data being obtained for 100% completion of the test cycle for the gear and the completed test cycle sensor data being time-variant; encoding the completed test cycle image data and the completed test cycle sensor data to create encoded test cycle image data and encoded test cycle sensor data; running a fusion model that fuses the encoded test cycle image data and encoded test cycle sensor data into a single fusion representation; performing multi class classification with object detection on the single fusion representation; and based on a result of the multi class classification with object detection, classifying the gear as falling within one of a plurality of gear failure modes and predicting a remaining useful life of the gear. When image data corresponding to actual 100% completion of the test cycle is available, the first stage is optional such that the second stage can be performed without first performing the first stage.BRIEF DESCRIPTION OF THE DRAWINGS

[0009] A full understanding of the invention can be gained from the following description of the preferred embodiments when read in conjunction with the accompanying drawings in which:

[0010] FIG. 1 is a schematic representation of a condition assessment system for atransmission gear, in accordance with an example embodiment of the disclosed concept;

[0011] FIG. 2 is a flow chart of a two-stage method for training an automated system to assess the longevity of a transmission gear, in accordance with an example embodiment of the disclosed concept;

[0012] FIGS. 3A, 3B, and 3C respectively show images obtained for 12.5% completion of a test cycle for a transmission endurance test, 50% completion of the test cycle, and forecasted image data predicted for 100% completion of the test cycle by applying the method depicted in FIG. 2 to the image data shown in FIG. 3 A and 3B;

[0013] FIG. 4 is an alternative symbolic depiction of certain steps of the method depicted in FIG. 2, in accordance with an example embodiment of the disclosed concept;

[0014] FIG. 5 is a symbolic depiction of a convolution based fusion block that can be used for a certain step of the method depicted in FIG. 2, in accordance with an example embodiment of the disclosed concept; and

[0015] FIG. 6 is a symbolic depiction of classification and object detection steps that can be performed as part of the method depicted in FIG. 2, in accordance with an example embodiment of the disclosed concept.DETAILED DESCRIPTION OF THE INVENTION

[0016] Directional phrases used herein, such as, for example, left, right, front, back, top, bottom and derivatives thereof, relate to the orientation of the elements shown in the drawings and are not limiting upon the claims unless expressly recited therein.

[0017] As used herein, the singular form of “a”, “an”, and “the” include plural references unless the context clearly dictates otherwise.

[0018] As employed herein, employed herein, when ordinal terms such as “first” and “second” are used to modify a noun, such use is simply intended to distinguish one item from another, and is not intended to require a sequential order unless specifically stated.

[0019] As employed herein, the term “controller” shall mean a programmable analog and / or digital device that can store, retrieve and process data; a processor; a control circuit; a computer; a workstation; a personal computer; a microprocessor; a microcontroller; a microcomputer; a central processing unit; a mainframe computer; a mini-computer; a server; a networked processor; or any suitable processing device or apparatus.

[0020] As employed herein, the statement that two or more parts or components are “coupled” shall mean that the parts are joined or operate together either directly or indirectly, i.e., through one or more intermediate parts or components, so long as a link occurs. As used herein, “directly coupled” means that two elements are directly in contact with each other. As used herein, “fixedly coupled” or “fixed” means that two components are coupled so as to move as one while maintaining a constant orientation relative to each other.

[0021] As employed herein, the term “number” shall mean one or an integer greater than one (i.e., a plurality).

[0022] As previously stated, existing methods for testing and assessing the longevity of a gear in a commercial or industrial / heavy duty transmission primarily rely on monitoring the gear using only image data or only vibration data during a test cycle, and then manually inspecting the gear after completion of the test cycle to determine the remaining useful life (RUL), which involves removing and disassembling the gear from the transmission. Given that a test cycle for a commercial or industrial transmission gear can typically last in the range of nine months to two years, it will be appreciated that a significant amount of time is lost when a transmission gear has a design flaw that cannot be discovered until the test cycle is completed. In addition, it will be appreciated that manually performing a visual inspection of the gear after completion of the test cycle in order to determine RUL introduces a degree of subjectivity to the RUL assessment.

[0023] Disclosed herein are an innovative system 10 (FIG. 1) and method 100 (FIG. 2) for assessing the longevity of a transmission gear, in accordance with an exemplary embodiment of the disclosed concept. The system 10 and method 100 are fully automated and thus able to both reduce the time it takes to make a longevity assessment for a gear and remove the subjectivity involved in an RUL assessment. While the disclosed system 10 and method 100 were developed in order to assess the condition of commercial and industrial / heavy duty transmission gears, adapting the disclosed system 10 and method 100 to assess the condition of lighter duty transmission gears is also within the scope of the disclosed concept. The method 100 of FIG. 2 is designed to train an automated system to perform longevity assessments for transmission gears and may be employed, for example, to train the system 10 shown in FIG. 1 and is described in conjunction with the system 10 shown in FIG. 1.

[0024] The disclosed system 10 comprises a number of cameras 11, a number of sensors 12, and a controller 13 configured to receive the data captured by the camera(s) 11 and sensor(s)12 and to execute the method 100. The cameras 11 and sensors 12 can be coupled to or installed in close proximity to a gear 1 that needs to be monitored. As an initial matter, it is noted that the disclosed system 10 harnesses the power of artificial intelligence, and the system 10 is depicted as including a plurality of modules 14, 16, 18, 21, 23 that execute various neural network functions within the controller 13. It will be appreciated that the depiction of these modules as discrete entities is done solely for the sake of simplicity in order to facilitate a clear understanding of the disclosed innovative concepts, and that there can be overlap between functions of each of the depicted modules 14, 16, 18, 21, 23 without departing from the scope of the disclosed concept.

[0025] When using known methods and systems to determine the longevity of a transmission gear based solely on manual visual inspection, the condition of a gear at 100% completion of a test cycle can only be accurately determined after the test cycle is actually 100% complete. In contrast, the disclosed innovative system 10 and a first stage of the method 100 enable reliable predictions to be made about what the condition of a gear will be at 100% completion of a test cycle significantly earlier in time and prior to the test cycle actually being 100% complete, for example and without limitation at 12.5%, 25%, or 50% completion of a test cycle. In addition, the disclosed system 10 and a second stage of the method 100 eliminate the need to manually perform a visual inspection to determine RUL after data for 100% of the test cycle is available.

[0026] The disclosed method 100 and associated system 10 use an N-dimensional multi modal approach, with “N” referring to the number of types of data being collected. For example and without limitation, if image, vibration, sound, accelerometer, and gyroscope data were being used, then the multi modal approach would be 5-dimensional, i.e. N = 5. The disclosed method 100 is a two stage controller-implemented method of analyzing the multi modal data collected for the gear, including the image data captured by the number of cameras 11 and sensor data captured by the number of sensors 12, in order to detect and categorize different types of defects in the gear. The method 100 is designed to be executed by a controller that has access to training data (i.e. image data and sensor data from other gears previously monitored) for neural network learning. The controller 10 is one such controller having training data for neural network learning. It is noted that the exemplary embodiment of the disclosed method 100 always uses image data and at least one type of sensor data, and different types of sensor data can be used invarying combinations in addition to the image data. The disclosed method 100 aims to resolve three major challenges encountered by the presently used method of manual inspection: reliance on labor to perform physical examination, inconsistent and subjective assessments, and late detection of gear damage.

[0027] The first stage (labeled Stage 1 in FIG. 2) of the method 100 is directed toward executing a test cycle for the gear and forecasting, at different phases of partial completion of the test cycle, what the gear fatigue will be for the gear 1 at 100% completion of the test cycle. That is, at various phases of partial completion of the test cycle, such as 12.5% completion, 25% completion, and 50% completion, the image and sensor data collected to that point in the test cycle is analyzed in order to forecast what gear fatigue will be at 100% of completion of the test cycle.

[0028] In known methods and systems for assessing the longevity of a gear, after a test cycle has been completed, a manual inspection is performed to determine the RUL of the gear 1. The second stage (labeled Stage 2 in FIG. 2) of the method 100 is directed toward performing an automated inspection of the gear 1 and providing an RUL assessment based on data available for 100% completion of the test cycle, with the automated inspection replacing the manual inspection that is typically performed to make an RUL assessment after the test cycle is complete. During this second stage of the method 100, the disclosed system 10 predicts both the type(s) of failure that will occur in the gear 1 when the gear 1 reaches the point of failure and the timing of the point of failure.

[0029] Reference is now made to FIG. 2 to discuss the method 100 in detail. The method 100 comprises the two stages previously mentioned, Stage 1 and Stage 2. Stage 1 is performed to collect image and sensor data at various phases of partial completion of a test cycle (e.g. 10% completion, 20% completion, 50% completion), and Stage 2 is performed when image and sensor data is available for 100% of a test cycle. As will be detailed further hereinafter, the data available for 100% of a test cycle used in Stage 2 can either be: (1) data obtained from actual 100% completion of the test cycle, or (2) data that has been forecasted for 100% of the test cycle based on data gathered from a phase of partial completion of the test cycle, such as 12.5%, 25%, 50%, etc. If image and sensor data is available for actual 100% completion of a test cycle, Stage 1 can be skipped and Stage 2 can be performed without first performing Stage 1.

[0030] Continuing to refer to FIG. 2, step 101 of the method is determining whether ornot data exists for 100% completion of a test cycle. If data for 100% completion of a test cycle does not exist, then the method proceeds from step 101 to step 110 in order to perform Stage 1 of the method 100, and if data for 100% completion of a test cycle does exist, then the method proceeds from step 101 to step 120 in order to perform Stage 2. Stage 1 is executed by the Test Cycle Prediction Module 14 (FIG. 1) and Stage 2 is executed by the RUL Module 16 (FIG. 1). Hereafter, Stage 1 will be detailed first, and Stage 2 will be detailed second. In Stage 1 , data is collected throughout the duration of the test cycle. For each round of data collection performed during Stage 1 for a phase that is less than 100% completion of the test cycle, the data obtained to that point is used as input to the remainder of the Stage 1 steps. The data available for 100% completion of the test cycle is treated as Ground Truth in Stage 2.

[0031] At step 110, data is gathered for the current phase of test cycle completion (e.g., 10% completion, 20% completion, 50% completion, etc.). The data gathered includes image data and 1-D (one-dimensional) sensor data. In an exemplary embodiment, the image data is captured by RGB channels at a resolution of 1000 x 1000 pixels. The 1-D sensor data is timevariant and can include data from one or more types of sensors, such as accelerometers, temperature sensors, and gyroscopes, for example and without limitation. It will be appreciated that these types of sensor data can be used to determine, for example and without limitation: (1) the extent to which a gear is vibrating; (2) if abnormal noises such as grinding, whining, clunking, or knocking sounds are being produced by the gearbox; and (3) if overheating of the gearbox is occurring, which can be caused by a variety of factors such as friction, inadequate lubrication (e.g. due to an oil leak or oil quality degradation), or internal damage.

[0032] After data gathering at step 110, the method 100 proceeds to step 111. At step 111, data cleaning and preprocessing is performed and can include, for example and without limitation, scaling, interpolation of missing values, etc. After step 111 is complete, the method concurrently proceeds to step 112 and to step 113, with step 112 pertaining to sensor data (i.e. non-image data) and with step 113 pertaining to assessing the quality of the captured visual / image data. At step 112, the sensor data available to this point for the partial completion of the test cycle (i.e. less than 100% completion) are used to forecast what the sensor data will be at 100% completion of the test cycle. As part of forecasting at step 112, nonlinearities in the data such as trend, seasonality, etc. are removed. In addition, a Variational Autoencoder is used to approximate each type of data signal without any anomalies, and a deep learning-based AutoRegressor Model with Moving Average is used to forecast data at the n* time interval. The method then uses the forecasted sensor data produced at step 112 during step 115 (discussed later herein). At step 113, the captured image data quality is assessed. If the image data is determined to be of good quality at step 113, then the method proceeds to step 115 (discussed later herein).

[0033] If the image data quality is determined to need improvement at step 113, then optional data quality enhancement can be performed at steps 114A-114C. Steps 114A-114C are sometimes referred to hereafter collectively or generally as step 114. Step 114A is to perform Image Enhancement. Image Enhancement can include performing super resolution at step 114B and / or synthetic data generation at step 114C. Super resolution entails increasing the resolution of the images in order to make the gear damage more prominent, which will increase the performance of the Test Cycle Prediction Module 14. The improvement in image data that can be achieved with super resolution results from increasing pixel density of the image in order to enhance image sharpness. Synthetic data generation entails generating new data with the help of old data using Generative Adversarial Networks (GANs). Image Augmentation techniques can additionally be used during steps 114B and / or 114C to add variations in different characteristics of the image data, such as color and orientation, for example and without limitation. After steps 114B and / or 114C are performed, the method proceeds to step 115.

[0034] At step 115, image data is predicted for 100% completion of the test cycle based on either the unenhanced data output at step 113 or the enhanced data output at step 114. The prediction of image data at step 115 can be made by executing steps such as: performing basic image processing (such as, for example and without limitation, contrast and brightness adjustment, tilt correction, etc.) on the image data output at step 113 or step 114 to produce a processed image; creating masking of the portions in the image data where teething or pitting or a contact pattern exists; and superimposing the features extracted from the processed image on the original image. Optionally, additional features extracted from the non-image sensor data (i.e. from step 111) can be injected to reduce false positives either during training of the Test Cycle Prediction Module 14 or while performing non maximum suppression during post processing of the predicted results. The image for 100% completion of the test cycle is predicted using a Convolutional Encoder Decoder Module 14A included in the Test Cycle Prediction Module 14. FIG. 3 provides an illustrative example of step 115, showing images obtained for 12.5% completion of a test cycle and for 50% completion of a test cycle of a gear, and additionallyshowing image data predicted for 100% completion of the test cycle based on the images obtained for 12.5% completion and for 50% completion. The method then proceeds from step 115 to step 116, where the forecasted prediction for the 100% completion of the test cycle of the gears is stored in a database 15, labeled as the Stage 1 database 15 in FIG. 1.

[0035] At step 117, if the test cycle is not 100% complete, then the method can either return to step 110 to perform Stage 1 for the next phase of partial completion of the test cycle (indicated as Option 1 in FIG. 2) or the method can proceed to Stage 2 (indicated as Option 2 in FIG. 2). It will be appreciated that the Test Cycle Prediction Module 14 can be configured on a case-by-case basis to determine how regularly Stage 1 should be executed throughout the testing process. For example and without limitation, some users may want to execute Stage 1 for every additional 10% of the test cycle is completed, while other users may want to execute Stage 1 for every additional 12.5% of the test cycle completed. If the test cycle is 100% complete at step 117 or if Option 2 is pursued after step 117, then the data stored for Stage 1 in the database 15 is fed as input to Stage 2.

[0036] Stage 2 is executed by the RUL Module 16 and commences with step 120, where data for use in Stage 2 is gathered, said data being either: (1) data obtained for actual 100% completion of the test cycle, and / or (2) data that was forecasted during Stage 1 and stored in the Stage 1 database 15, said forecasted data being data predicted for 100% completion of the test cycle based on data gathered from at least one phase of partial completion of the test cycle, such as 12.5%, 25%, 50%, etc. As previously noted, when data is already available for 100% of a test cycle, Stage 1 can be skipped. For example and without limitation, a user may choose to skip Stage 1 if a test cycle has already been completed prior to using the system 10, and if the user desires to have an automated RUL assessment performed for the gear 1 rather than having to perform a manual inspection of the gear 1 to determine RUL. It is noted that FIG. 4 provides an alternative symbolic depiction of Steps 121-125 that may be helpful to view in conjunction with FIG. 2.

[0037] As in Stage 1, the data used in Stage 2 includes image data and ID sensor data. The image data comprises RGB channels of the image in 1000 x 1000 pixels. The ID sensor data includes measurements from various sensors (e.g., accelerometers, gyroscopes, etc.). The method proceeds from step 120 to step 121, where the data gets preprocessed. Preprocessing can comprise, for example and without limitation, scaling, interpolation of missing values, croppingof the image, etc. The method then proceeds from step 121 to steps 122A and 122B (which can be referred to generally or collectively as step 122), which can be performed concurrently by the Encoder Module 18.

[0038] The sensor data is encoded at step 122A while the image data is encoded at step 122B, with the basic steps of encoding being the same for both types of data. The Encoder Module 18 includes a number of modality-specific encoders 19, with each modality specific encoder 19 being dedicated to encoding exactly one modality of data, i.e. either the image data or the data originating from one of the sensors 12. As such, for the N-dimensional system 10, one modality specific-encoder 19 is used to encode the image data, and (N-l) modality-specific encoders 19 are used to encode the sensor data originating from each of the (N-l) sensors 12. The encoding steps 122 begin with extracting input-level features from each distinct modality. The term “input-level features” refers to features of each modality that are present after preprocessing at step 121. Each modality-specific encoder 19 transforms its input data into a set of feature vectors. By way of non-limiting illustrative example, if the system 10 is configured to collect image, sound, accelerometer, and gyroscope data, then the system 10 is implemented as a 4-dimensional system having four modality-specific encoders 19 to correspond on a one-to-one basis with the four aforementioned modalities, and input-level features are extracted and transformed into feature vectors for each of the image, sound, accelerometer, and gyroscope modalities.

[0039] Each modality-specific encoder 19 is composed of several layers of neural networks that use nonlinear transformations to extract increasingly abstract features from the input data. Two identical instances of the output of each modality-specific encoder 19 are then stored separately, with each instance constituting a representation that captures the relevant information from the modality. The encoder module 18 then uses the first instance of the output from each of the modality-specific encoders 19 to determine what underlying structures and relationships exist between the multiple modalities in the system 10. The second instance of the output from each of the modality-specific encoders 19 is stored in a Stage 2 database 25 in the controller 13, in case the data is needed at a later point in time. By capturing the underlying structure and relationships between the input data from multiple modalities, the encoder module 18 enables the controller 13 to make more accurate predictions or generate new outputs for use in later steps of Stage 2.

[0040] The method then proceeds to step 123 (comprising steps 123A and 123B shown in FIG. 2), where the data encoded at steps 122A and 122B is input to a fusion model in the Fusion Module 21, i.e. for each given modality, the first instance of the given modality’s output from the Encoder Module 18 is fed as input to the Fusion Module 21. At step 123 A, information from the different modalities is combined, and at step 123B, spatial attention is used to determine a weighted sum of the modalities’ features. The goal of the Fusion Module 21 is to capture complementary information from different modalities and create a more robust and informative representation for downstream tasks performed in Stage 2. To aid in understanding of step 123, FIG. 5 provides a symbolic depiction of a convolution based fusion block used by the Fusion Module 21 for step 123. The Fusion Module 21 combines information from different modalities (e.g., image, sound, accelerometer, and gyroscope) into a single representation for doing classification. In the Fusion Module 21, a weighted sum of the modalities’ features is used, where the weights are learned during training, and attention mechanisms are used to focus on relevant parts of each modality at each timestep. Specific regions in images are attended to based on the context of abnormal spikes observed in the image data waveform (and vice versa). A loss function is defined that considers all modalities. Gradients are backpropagated through the entire network. The fusion model in the Fusion Module 19 is regularized to prevent overfitting (e.g., dropout, weight decay). Appropriate optimizers (e.g., Adam, SGD) are used for joint training.

[0041] The method then proceeds to step 124 (comprising steps 124 A and 124B shown in FIG. 2), where Multi Class Classification with Object Detection is performed by a Classification Module 24. FIG. 6 provides a symbolic depiction of one non-limiting example of how multi class classification and object detection can be performed at steps 124 A and 124B respectively. The Classification Module 24 takes the joint representation generated by the Fusion Module 21 and uses it to make a prediction or decision, i.e. a classification. The Classification Module 24 takes the form of a neural network with an object detection head, where the joint representation is passed through one or more fully connected layers before the final prediction is made. These layers can include non-linear activation functions, dropout, and other techniques to help prevent overfitting and improve generalization performance. This module is trained using a supervised learning approach, where the input modalities and their corresponding labels or targets are used to optimize the parameters of the model. Thisoptimization is often done using gradient-based optimization methods such as stochastic gradient descent or its variants. The backbone network can consist of a pretrained model to accelerate the training by using a transfer learning approach. A loss function that considers all modalities (e.g., cross-entropy for classification tasks, Intersection over Union (loU) loss for localization of the faults) is defined. Gradients are backpropagated through the entire network. The model is regularized to prevent overfitting (e.g., dropout, weight decay). Non-Maximum Suppression Algorithms reduce false positives in the post processing stage.

[0042] The output of the classification module 23 at step 124 will fall into one of multiple classes, with non-limiting examples of the classes including: Healthy, Slightly Worn, Medium Worn, Broken Teeth, Pitting, Faulty. The classes will be represented in the form of bounding boxes highlighting the area of damage caused in the gear 1. The method then proceeds to step 125, where two predictions are made. The first prediction pertains to quantifying and otherwise determining the remaining useful life of the gear 1 under current stress and load conditions based on the severity of gear wear and tear. The second prediction pertains to identifying the type of failure that will occur in a gear based on the input data provided at different stages of the test cycle in Stage 1. That is, for each phase of partial completion of the test cycle during Stage 1, the data stored in the Stage 1 database 15 is analyzed by the Classification Module 23 to determine what complementary information exists between the classification made at step 124 and the data available from Stage 1 for each phase of partial completion.

[0043] The objective of Step 125 is to find a relationship between the ultimate classification of gear condition made at step 124 and the condition of the gear during each phase of partial completion of the test cycle. By determining these relationships at step 125, the RUL Module 16 is expected to be better able to make predictions in the future about the lifespan of other gears based on data collected from relatively early in a test cycle, so that a design flaw can be caught as early as possible rather than having to finish 100% of the test cycle to discover the flaw. The method then proceeds from step 125 to step 126, where the data collected and produced during Stage 2 is stored in the Stage 2 database 25. Every time the controller 10 executes the method 100 for a new gear, the controller 10 increases its accuracy in predicting, increasingly earlier in a test cycle: (1) what the condition of a monitored gear will be at 100% completion of the test cycle based on partial completion of the test cycle, and (2) what the RULof the gear will be at 100% completion of the test cycle based on the prediction of what the gear condition will be at 100% completion of the test cycle.

[0044] The advantages of the disclosed system 10 and method 100 over existing manualinspection based methods are likely abundantly clear after the discussion of the figures provided here. It is worth additionally noting that while most known existing methods for assessing transmission gear longevity are only image data based, some research and small experimental setups have employed vibrational data instead of image data, but those setups use vibrational data only to be able to classify gears as either healthy or faulty and are not capable of identifying categories of fault the way that the disclosed method 100 does. The leveraging of both image data and timeseries sensor data by the disclosed system 10 and method 100, as well as the fully automated nature of the disclosed system 10 and method 100, result in a much more robust and objective analysis and provide a significantly more detailed assessment than known systems and methods can.

[0045] While specific embodiments of the invention have been described in detail, it will be appreciated by those skilled in the art that various modifications and alternatives to those details could be developed in light of the overall teachings of the disclosure. Accordingly, the particular arrangements disclosed are meant to be illustrative only and not limiting as to the scope of disclosed concept which is to be given the full breadth of the claims appended and any and all equivalents thereof.

Claims

What is claimed is:

1. A method for determining the longevity of a gear used in a transmission, the method being controller-implemented and comprising: a first stage, the first stage comprising: forecasting, at a first time that is earlier than 100% completion of a test cycle for the gear, predicted image data corresponding to 100% completion of the test cycle using image data and sensor data collected for the gear up through the first time, the sensor data being time-variant; and a second stage, the second stage comprising: gathering completed test cycle multi modal data including completed test cycle image data and completed test cycle sensor data, the completed multi modal data being obtained for 100% completion of the test cycle for the gear, the completed test cycle sensor data being time-variant; encoding the completed test cycle image data and the completed test cycle sensor data to create encoded test cycle image data and encoded test cycle sensor data; running a fusion model that fuses the encoded test cycle image data and encoded test cycle sensor data into a single fusion representation; performing multi class classification with object detection on the single fusion representation; and based on a result of the multi class classification with object detection, classifying the gear as falling within one of a plurality of gear failure modes and predicting a remaining useful life of the gear, and wherein, when image data corresponding to actual 100% completion of the test cycle is available, the first stage is optional such that the second stage can be performed without first performing the first stage.

2. The method of claim 1, wherein the plurality of gear failure modes includes: healthy, slightly worn, medium worn, broken teeth, pitting, faulty.

3. The method of claim 1,wherein the predicting the remaining useful life of the gear is based on current stress and load conditions of the gear.

4. The method of claim 1, wherein the first stage further comprises: gathering partial test cycle multi modal data including partial test cycle image data and partial test cycle sensor data, the partial test cycle sensor data being time-variant; and inputting the partial test cycle image data and partial test cycle sensor data to a convolutional encoder decoder module to produce the predicted image data.

5. The method of claim 4, wherein the first time is less than or equal to 50% completion of the test cycle.

6. The method of claim 1, wherein the multi modal data includes data from a plurality of modalities, the plurality of modalities including a camera and a number of one-dimensional sensors, wherein encoding the completed test cycle image data and the completed test cycle sensor data comprises, for each given modality in the plurality of modalities, extracting input-level features from each given modality and transforming the input-level features into feature vectors, wherein the method further comprises executing nonlinear transformations on the feature vectors in a plurality of neural network layers.

7. The method of claim 6, wherein running the fusion model further comprises: using a weighted sum of the feature vectors for all modalities in the plurality of modalities; defining a loss function; and backpropagating gradients through the entirety of neural network layers.

8. The method of claim 7,wherein performing the multi class classification comprises passing the single fusion representation through a neural network with an object detection head such that the single fusion representation passes through one or more fully connected layers in the neural network before the gear is classified as falling within one of the plurality of gear failure modes and the remaining useful life of the gear is predicted.

9. The method of claim 8, wherein the one or more fully connected layers in the neural network include at least one of a non-linear activation function and dropout.

10. A system for determining the longevity of a gear used in a transmission, the system comprising: a number of cameras positioned to collect image data for the gear; a number of sensors positioned to collect sensor data for the gear, the sensor data being time-variant; and a controller, the controller being configured to execute a method having a first stage and a second stage, wherein the first stage comprises: forecasting, at a first time that is earlier than 100% completion of a test cycle for the gear, predicted image data corresponding to 100% completion of the test cycle using the image data and the sensor data collected for the gear up through the first time; and a second stage, the second stage comprising: gathering completed test cycle multi modal data including completed test cycle image data and completed test cycle sensor data, the completed multi modal data being obtained for 100% completion of the test cycle for the gear, the completed test cycle sensor data being time-variant; encoding the completed test cycle image data and the completed test cycle sensor data to create encoded test cycle image data and encoded test cycle sensor data; running a fusion model that fuses the encoded test cycle image data and encoded test cycle sensor data into a single fusion representation;performing multi class classification with object detection on the single fusion representation; and based on a result of the multi class classification with object detection, classifying the gear as falling within one of a plurality of gear failure modes and predicting a remaining useful life of the gear, and wherein, when image data corresponding to actual 100% completion of the test cycle is available, the first stage is optional such that the second stage can be performed without first performing the first stage.

11. The system of claim 10, wherein the plurality of gear failure modes includes: healthy, slightly worn, medium worn, broken teeth, pitting, faulty.

12. The system of claim 10, wherein the predicting the remaining useful life of the gear is based on current stress and load conditions of the gear.

13. The system of claim 10, wherein the first stage further comprises: gathering partial test cycle multi modal data including partial test cycle image data and partial test cycle sensor data, the partial test cycle sensor data being time-variant; and inputting the partial test cycle image data and partial test cycle sensor data to a convolutional encoder decoder module to produce the predicted image data.

14. The system of claim 13, wherein the first time is less than or equal to 50% completion of the test cycle.

15. The system of claim 10, wherein the multi modal data includes data from a plurality of modalities, the plurality of modalities including a camera and a number of one-dimensional sensors,wherein encoding the completed test cycle image data and the completed test cycle sensor data comprises, for each given modality in the plurality of modalities, extracting input-level features from each given modality and transforming the input-level features into feature vectors, wherein the method further comprises executing nonlinear transformations on the feature vectors in a plurality of neural network layers.

16. The system of claim 16, wherein running the fusion model further comprises: using a weighted sum of the feature vectors for all modalities in the plurality of modalities; defining a loss function; and backpropagating gradients through the entirety of neural network layers.

17. The system of claim 16, wherein performing the multi class classification comprises passing the single fusion representation through a neural network with an object detection head such that the single fusion representation passes through one or more fully connected layers in the neural network before the gear is classified as falling within one of the plurality of gear failure modes and the remaining useful life of the gear is predicted.

18. The system of claim 17, wherein the one or more fully connected layers in the neural network include at least one of a non-linear activation function and dropout.

19. The system of claim 10, wherein the number of sensors includes at least one of an accelerometer, a temperature sensor, gyroscope, or transducer.

Images