Systems and methods for using machine learning in fault detection of powertrains and vehicles

Abstract

System and methods are provided for leveraging machine learning for the fault detection and assessment of powertrains and / or vehicles and associated components. At least one image associated with a part of the powertrain and / or vehicle is generated from an input by a user. The at least one image is assessed for a fault condition based on machine learning, and the assessment of the part is then output to the use.

Description

SYSTEMS AND METHODS FOR USING MACHINE LEARNING IN FAULT DETECTION OF POWERTRAINS AND VEHICLESCROSS-REFERENCE TO RELATED APPLICATION:

[0001] The present application claims the benefit of the filing date of, and priority to,U.S. Provisional Application Ser. No. 63 / 649,560 filed on May 20, 2024, which is incorporated herein by reference.BACKGROUND

[0002] Technicians that repair vehicles, powertrains, and components associated with these devices often rely on audio and / or visual assessment of various parts to determine if the part should be repaired, replaced, or re-used. However, these assessments made by technicians may be inaccurate or incapable of detecting certain types of fault conditions. Images of the part or parts in question may also be provided to experts to assist in the review and assessment. However, the number of experts available to assist in such assessments is far exceeded by the number of technicians. Therefore, further improvements in this technology area are needed.SUMMARY

[0003] Embodiments are directed to unique systems, components, and methods for leveraging machine learning for identification and detection or assessment of faults in one or more subsystems, subassembly, components, or parts (hereinafter collectively referred to as “part” or “parts”) associated with powertrains and / or vehicles. Other embodiments are directed to apparatuses, systems, devices, hardware, methods, and combinations thereof for leveraging machine learning for the fault detection and assessment for one or more parts of powertrains and / or vehicles.

[0004] According to an embodiment, a method or system of detecting or assessing a fault condition of a part of a powertrain and / or vehicle may include generating at least one image associated with the part, analyzing the at least one image based on machine learning and a database of related images associated with the part, generating an assessment of the part based on the machine learning analysis, and outputting the assessment.

[0005] According to an embodiment, a method or system of detecting or assessing a fault condition of a part of a powertrain and / or vehicle may include capturing, by a camera of a mobile device, at least one image of the part, determining an identification of the part based on the image, selecting a machine learning algorithm based on the identification of the part, generating an assessment of the part based on the selected machine learning algorithm, the at least one image, and a database of related images of the part, and outputting the assessment.

[0006] In some embodiments, the identification of the part is determined using a first machine learning algorithm and the assessment is generated using a second machine learning algorithm.

[0007] In some embodiments, the at least one image is captured by a camera. In an embodiment, the camera is part of a mobile device of a technician. In an embodiment, the camera is a thermal camera.

[0008] According to an embodiment, a method or system of detecting or assessing a fault condition of a part of a powertrain and / or vehicle may include capturing an audible output of the part, generating at least one image based on the captured audible output, selecting a machine learning algorithm based on an identification of the part, generating an assessment of the part based on the selected machine learning algorithm, the at least one image, and a database of related images of the audible output for the part, and outputting the assessment.

[0009] In some embodiments, generating at least one image based on the captured audible output includes using a spectrogram or scalogram of the audible output to generate the at least one image of the captured audible output.

[0010] In some embodiments, the audible output is captured by a microphone. In an embodiment, the microphone is part of a mobile device of a technician. In an embodiment, the audible output is captured by a sensor, such as a vibration sensor, which is transformed to audible output that can be captured in at least one image.

[0011] In some embodiments, a first image of the part is captured by a camera of a mobile device and an identification of the part is made by machine learning based on the first image and a database of images of various parts. The assessment of the part is made by a second machine learning based on the at least one image that is based on the captured audible output and a database of related images of audible output for the part.

[0012] In some embodiments, the method or system may further include transmitting the at least one image to a server, wherein generating the assessment of the part may include generating the assessment by analyzing the at least one image with machine learning in conjunction with a database of images related to the part in faulty and non-faulty conditions, and the method or system may further include transmitting the assessment from the server to an output device.

[0013] In some embodiments, the assessment may include whether the part is to be repaired, replaced, or re-used in a current condition. In some embodiments, the assessment may include an output to a mobile device of a technician. In some embodiment, the technician can provide feedback on the assessment, such as whether the assessment was accurate or complete, which is then incorporated into subsequent machine learning analysis.

[0014] This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used as an aid in limiting the scope of the claimed subject matter. Further embodiments, forms, features, and aspects of the present application shall become apparent from the description and figures provided herewith.BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The concepts described herein are illustrative by way of example and not by way of limitation in the accompanying figures. For simplicity and clarity of illustration, elements illustrated in the figures are not necessarily drawn to scale. Where considered appropriate, references labels have been repeated among the figures to indicate corresponding or analogous elements.

[0016] FIG. 1 is a simplified block diagram of an embodiment of a system for leveraging machine learning for the fault detection and assessment of a part of a powertrains and / or vehicle;

[0017] FIG. 2 is a simplified flow diagram of an embodiment of a process for the fault detection and assessment of a part of a powertrain and / or vehicle;

[0018] FIG. 3 is a simplified block diagram of another embodiment of a system for leveraging machine learning for the fault detection and assessment of a part of a powertrain and / or vehicle;

[0019] FIG. 4 is a simplified flow diagram of an embodiment of a process for the fault detection and assessment of a part of a powertrain and / or vehicle; and

[0020] FIG. 5 is a simplified flow diagram of at least one embodiment of a method for leveraging machine learning for fault detection and assessment of a part of a powertrain and / or vehicle.DETAILED DESCRIPTION

[0021] Although the concepts of the present disclosure are susceptible to various modifications and alternative forms, specific embodiments have been shown by way of example in the drawings and will be described herein in detail. It should be understood, however, that there is no intent to limit the concepts of the present disclosure to the particular forms disclosed, but on the contrary, the intention is to cover all modifications, equivalents, and alternatives consistent with the present disclosure and the appended claims.

[0022] References in the specification to “one embodiment,” “an embodiment,” “an illustrative embodiment,” etc., indicate that the embodiment described may include a particular feature, structure, or characteristic, but every embodiment may or may not necessarily include that particular feature, structure, or characteristic. Moreover, such phrases are not necessarily referring to the same embodiment. It should further be appreciated that although reference to a “preferred” component or feature may indicate the desirability of a particular component or feature with respect to an embodiment, the disclosure is not so limiting with respect to other embodiments, which may omit such a component or feature. Further, when a particular feature, structure, or characteristic is described in connection with an embodiment, it is submitted that it is within the knowledge of one skilled in the art to implement such feature, structure, or characteristic in connection with other embodiments whether or not explicitly described.Additionally, it should be appreciated that items included in a list in the form of “at least one of A, B, and C” can mean (A); (B); (C); (A and B); (B and C); (A and C); or (A, B, and C). Similarly, items listed in the form of “at least one of A, B, or C” can mean (A); (B); (C); (A and B); (B and C); (A and C); or (A, B, and C). Further, with respect to the claims, the use of words and phrases such as “a,” “an,” “at least one,” and / or “at least one portion” should not be interpreted so as to be limiting to only one such element unless specifically stated to the contrary, and the use of phrases such as “at least a portion” and / or “a portion” should be interpreted as encompassing both embodiments including only a portion of such element and embodiments including the entirety of such element unless specifically stated to the contrary.

[0023] The disclosed embodiments may, in some cases, be implemented in hardware, firmware, software, or a combination thereof. The disclosed embodiments may also be implemented as instructions carried by or stored on one or more transitory or non-transitory machine-readable (e g., computer-readable) storage media, which may be read and executed byone or more processors. A machine-readable storage medium may be embodied as any storage device, mechanism, or other physical structure for storing or transmitting information in a form readable by a machine (e.g., a volatile or non-volatile memory, a media disc, or other media device).

[0024] In the drawings, some structural or method features may be shown in specific arrangements and / or orderings. However, it should be appreciated that such specific arrangements and / or orderings may not be required. Rather, in some embodiments, such features may be arranged in a different manner and / or order than shown in the illustrative figures unless indicated to the contrary. Additionally, the inclusion of a structural or method feature in a particular figure is not meant to imply that such feature is required in all embodiments and, in some embodiments, may not be included or may be combined with other features.

[0025] Referring now to FIG. 1, in the illustrative embodiment, an embodiment of a system 100 for leveraging machine learning for determining a fault condition of a part of a powertrain and / or vehicle is shown. The illustrative system 100 includes a mobile device 102, a network 104, and a server 106. In an embodiment, server 106 is a cloud server. Further, the illustrative mobile device 102 includes at least one camera 108, at least one display 110, and at least one application 120. The techniques described herein allow a user (e.g., technician or end user) to use the camera 108 of the mobile device 102 to capture images of a part associated with a component of a powertrain and / or vehicle, use application 120 to leverage machine learning technologies to analyze the images, and use display 110 to present an assessment to the user regarding a fault condition of the part based on the machine learning of the least one image. The assessment may also include an identification of whether the part can be re-used or required replacement.

[0026] As used herein, “part” or “parts” can be any component of combination of components of a powertrain and / or vehicle. Examples of parts include, but are not limited to, engine blocks, pistons, cylinder heads, valves, shafts, rods, connectors, joints, cylinder liners, axles, bearings, brakes and brake system components, aftertreatment components, catalysts, filters, plumbing, turbochargers, compressors, turbines, wastegates, etc.

[0027] In an embodiment of system 100, network 104 includes a machine learning workstation 112 that includes multiple machine learning algorithms from which to select for identification and / or fault analysis of the part. Network 104 also includes an image database 114with images of various parts that are used to identify a part, and / or images of each of the various parts in various faulty conditions and non-faulty conditions. Machine learning workstation 112 may include a first machine learning algorithm to be used to identify the part based on the image of the part by comparing the part to images of various parts in image database 114. Machine learning workstation 112 may also be configured to select a second machine learning algorithm 116 to perform the assessment of the identified part using images of the part depicted in various faulty and non-faulty conditions in image database 114.

[0028] Referring to FIG. 2, a flow diagram of an embodiment of an assessment method or procedure 200 using system 100 is provided. Procedure 200 includes a first step that involves a vehicle or powertrain 202 being identified for a service event and / or assessment of one or more parts thereof. The vehicle and / or powertrain 202 may be a truck, locomotive, genset, or other vehicle or a part thereof. Procedure 200 includes a second step in which at least one image 204 is taken of a part of the vehicle and / or powertrain 202.

[0029] Procedure 200 includes a third step that includes uploading 206 the at least one image of the part to server 106. Server 106 communicates 208 with network 104 at the next step to analyze the at least one image in order to identify the part and / or to assess a fault condition of the part using machine learning algorithm 116. The assessment 210 of the part is then provided to server 106 at the next step 210, and an output 212 is provided to the user at following step.

[0030] Referring now to FIG. 3, in the illustrative embodiment, another embodiment of a system 300 for leveraging machine learning for determining a fault condition of a part of a powertrain and / or vehicle is shown. The illustrative system 300 includes a mobile device 302, a network 304, and a server 306. In an embodiment, server 306 is a cloud server. Further, the illustrative mobile device 302 includes at least one microphone 308, at least one display 310, and at least one application 320. The techniques described herein allow a user (e g., technician or end user) to use the microphone 308 of the mobile device 302 to capture audible output of a part associated with a component of a powertrain and / or vehicle, use application 320 and / or server 306 to convert the audible output to at least one image to leverage machine learning technologies to analyze the at least one image of the audible output, and use display 310 to present an assessment to the user regarding a fault condition of the part based on the machine learning of the audible output.

[0031] In an embodiment of system 300, network 304 includes a machine learning workstation 312 that includes multiple machine learning algorithms. Network 304 also includes an audio database 314 with images of audible outputs associated with various parts in various faulty and non-faulty conditions. In an embodiment, machine learning workstation 312 may include a first machine learning algorithm be used to identify the audio signature of the fault in question on the system being analyzed for the analysis of audible faults.

[0032] Referring to FIG. 4, a flow diagram of an embodiment of an assessment method or procedure 400 using system 300 is provided. Procedure 400 includes a first step that involves a vehicle or powertrain 402 being identified for a service event and / or assessment of one or more parts thereof. The vehicle and / or powertrain 402 may be a truck, locomotive, genset, or other vehicle or a part thereof. Procedure 400 includes a second step in which audible output 404 is captured from the part of the vehicle and / or powertrain 402.

[0033] Procedure 400 includes a third step that includes uploading 406 the audible output and / or at least one image of the audible output part to server 306. Server 306 communicates 408 with network 304 at the next step to analyze at least one image of the audible output in order to identify the part and / or to assess a fault condition of the part using the at least one image of the captured audible output and machine learning algorithm 316. The assessment 410 of the part is then provided to server 306 at the next step, and an output 412 can be provided to the user at the following step.

[0034] The identification and / or assessment in the above embodiments may be customized to the particular part using computer vision and machine learning to provide automatic, prompt, and accurate feedback to the user during a service event and / or in a more controlled lab-based environment. Further, in some embodiments, the server 106, 306 may provide customized instructions or feedback to the user (e.g., how or where to repair the part, to replace the part, re-use the part, provide a failure mode alert, to maintain the part in a current condition, to service the part, etc.). In some embodiments, mobile device 102, 302 may be used to provide user feedback regarding the assessment made by the machine learning to network 104, 304 via server 106, 306 (e.g., to determine whether the assessment was accurate or complete, whether the instructions addressed the issue, etc.) Such user feedback may be used to incorporate the associated image of the part and corrected to actual fault condition into the appropriate databases 114, 314

[0035] It should be appreciated that the mobile device 102, 302, network 104, 304, and / or the server 106, 306 may be embodied as any type of device or collection of devices suitable for performing the functions described herein. More specifically, in the illustrative embodiment, the mobile device 102, 302 may be embodied as any type of device capable of capturing images or audio and displaying information for a user of the mobile device 102, 302. Although the camera 108, microphone 308, and the display 110, 310 are described herein as forming a portion of the mobile device 102, 302 it should be appreciated that in other embodiments the camera 108, microphone 308, and / or the display 110, 310 may be separate from but communicatively coupled to the mobile device 102, 302 (e.g., as a peripheral device). For example, in an embodiment, microphone 308 is an array of microphones positioned around the part.

[0036] The mobile device 102, 302 may include one or more applications 120, 320 that enable the mobile device 102, 302 to capture images, capture audio, process the images and / or audio, and provide an output of the assessment of the part to the user and / or to allow the user to provide feedback regarding the assessment. It should be appreciated that the application may be embodied as any suitable application for performing the functions described herein. For example, in some embodiments, the application may be embodied as a mobile application (e.g., smartphone application). In some embodiments, it should be appreciated that the application may serve as a client-side user interface for a web-based application or service of the server 106, 306. As such, in some embodiments, the server 106, 306 may process various data and / or perform various functions described herein in reference to the mobile device 102, 302. Further, as described herein, in some embodiments, the mobile device 102, 302 may be configured to transmit images, video, and / or audio to the server 106, 306 for further processing, and server 106, 306 may transmit results of the analysis from network 104, 304 (e.g., including fault assessment of the part) back to the mobile device 102, 302 for display to the user.

[0037] The camera 108 may be embodied as any type of device capable of capturing one or more images discretely or in a stream. For example, the camera 108 may include one or more two-dimensional (2D) cameras, three-dimensional (3D) cameras, and / or video cameras. Although the camera 108 is described herein generally as including image sensors that captured the visual light spectrum, it should be appreciated that the camera 108 may be configured to capture waves / signals in the non-visual light spectrum and / or may be a thermal camera in otherembodiments. Although the camera 108 is primarily described herein in the singular, it should be appreciated that the mobile device 102 may include multiple cameras 108 in other embodiments and the techniques described herein apply equally well to such an embodiment. In addition, camera 108 may be non-fixed in position, e.g. handheld, in order to take the image. Some embodiments also contemplate the ability to fix or control the position of camera 108 when capturing images of certain parts, such as a cylinder liner insert.

[0038] The microphone 308 may be embodied as any type of device capable of capturing audio output from the part discretely or in a stream. For example, the microphone 308 may include one or more microphones, an array of microphones, and / or microphones used alone or in conjunction with video cameras. The audio output can be captured in a non-controlled environment. Application 320 and / or server 306 can manipulate or transform the audio data captured by microphone 308 into at least one image using spectrograms or scalograms using various wavelet transform configurations that support the machine learning algorithm to be used. Examples of spectrograms and scalograms include mel spectrograms, mel-frequency cepstral coefficients (MFCC) spectrograms, linear-frequency cepstral coefficients (LFCC) spectrograms, and power spectrograms. Although the camera 108 and microphone 308 are shown separately in different embodiments, it should be appreciated that the camera 108 and microphone 308 may be provided on a single mobile device 102, 302 in other embodiments.

[0039] In still other embodiments, mobile device 102, 302 includes one or more sensors to obtain data from the part being assessed. For example, the one or more sensors may include a vibration sensor, thermal sensor, etc. to capture data from the part that is transformed into at least one image by server 106, 306 and / or application 120, 320 and provided to network 104, 304 for analysis by machine learning. Thermal sensors and / or a thermal camera may be used to detect, for example, cracks, leaks, hot spots, etc.

[0040] The display 110, 310 may be embodied as any type of device capable of displaying fault condition assessments and instructions on a graphical user interface for a user of the mobile device 102, 302. For example, in some embodiments, the display 110, 310 may include a touchscreen display. Although the display 110, 310 is primarily described herein in the singular, it should be appreciated that the mobile device 102, 302 may include multiple displays 110, 310 in other embodiments and the techniques described herein apply equally well to such an embodiment. It should be further appreciated that, in some embodiments, the mobile device 102,302 may include additional and / or alternative mechanisms to provide output and instructions to the user. For example, in some embodiments, the mobile device 102, 302 may provide audible installation output via a speaker of the mobile device 102, 302 and / or other types of feedback / instructions to the user (e.g., tactile feedback, haptic feedback, and / or other types of feedback / instructions).

[0041] The server 106, 306 may be embodied as any type of communication network capable of facilitating communication between the various devices of the system 100, 300. As such, the server 106, 306 may include one or more networks, routers, switches, computers, and / or other intervening devices. For example, the server 106, 306 may be embodied as a cloud server or otherwise include one or more cellular networks, telephone networks, local or wide area networks, publicly available global networks (e.g., the Internet), ad hoc networks, short- range communication links, or a combination thereof. In the illustrative embodiment, the server 106, 306 may be configured to process one or more images or audio files captured by the mobile device 102, 302 for example, for use by server 106, 306 using computer vision, artificial intelligence, and / or other techniques.

[0042] It should be further appreciated that the server 106, 306 described herein may function in a cloud computing environment. Server 106, 306 may be embodied as a cloud-based device or collection of devices within a cloud computing environment. Further, in cloud-based embodiments, the server 106, 306 may be embodied as a server-ambiguous computing solution, for example, which executes a plurality of instructions on-demand, contains logic to execute instructions only when prompted by a particular activity / trigger, and does not consume computing resources when not in use. That is, the server 106, 306 may be embodied as a virtual computing environment residing “on” a computing system (e.g., a distributed network of devices) in which various virtual functions (e.g., Lambda functions, Azure functions, Google cloud functions, and / or other suitable virtual functions) may be executed corresponding with the functions of the server 106, 306 described herein. For example, when an event occurs (e.g., data is transferred to the server 106, 306 for handling), the virtual computing environment may be communicated with (e.g., via a request to an API of the virtual computing environment), whereby the API may route the request to the correct virtual function (e.g., a particular server- ambiguous computing resource) based on a set of rules. As such, when a request for the transmission of images of the part to network 104, 304 is made (e.g., via an appropriate userinterface to the server 106, 306), the appropriate virtual function(s) may be executed to perform the actions before eliminating the instance of the virtual function(s).

[0043] The network 104, 304 may be embodied as any type of device(s) capable of performing the functions described herein using machine learning for identifying the part and / or to assess a fault condition of the part. In the illustrative embodiment, the network 104, 304 may be configured to store and process one or more images or audio files captured by the mobile device 102, 302 for example, using computer vision, machine learning, artificial intelligence, and / or other techniques. The network 104, 304 may also be configured to store images or audio image files of various parts in faulty and non-faulty conditions to use in the machine learning analysis of a part under assessment. In some embodiments, the mobile device 102, 302, the network 104, 304, and the server 106, 306 may cooperatively perform one or more of the functions of the processing system described herein. For example, in some embodiments, the processing system of the mobile device 102, 302 may perform some processing (e.g., less computationally- and / or data-intensive processing), whereas the processing system of the network 104, 304 and / or server 106, 306 may perform other processing (e.g., more computationally- and / or data-intensive processing).

[0044] The mobile device 102, 302, network 104, 304, and / or server 106, 306 may apply various computer vision algorithms, filters, and / or techniques to generate processed versions of the captured images and / or reformatted versions thereof. For example, in some embodiments, mobile device 102 302, network 104, 304, and / or server 106, 306 may utilize image filters (e.g., kernel-based convolution, masking, etc.), edge detection algorithms (e.g., Canny edge detection, Sobel filters, etc.), image segmentation algorithms (e.g., pyramid segmentation, watershed segmentation, etc.), blob detection algorithms, corner detection algorithms, features identification and / or matching algorithms (e.g., scale-invariant feature transform (SIFT), speeded-up robust features (SURF), etc.), morphological image processing algorithms (e.g., erosion, dilation, opening, closing, etc.), thresholding / voting algorithms, and / or other suitable algorithms useful in determining characteristics of the access control hardware and / or installation location.

[0045] Further, in some embodiments, network 104, 304 may leverage machine learning techniques to perform the functions described herein (e.g., to better ascertain the characteristics of the part and / or assessment of the fault condition of the part). For example, in someembodiments, the network 104, 304 may utilize one or more neural network algorithms, regression algorithms, instance-based algorithms, regularization algorithms, decision tree algorithms, Bayesian algorithms, clustering algorithms, association rule learning algorithms, deep learning algorithms, dimensionality reduction algorithms, and / or other suitable machine learning algorithms, techniques, and / or mechanisms. The machine learning algorithms can reduce noise in the image data and / or determine distance, orientation, and / or direction of the camera from the part and make adjustment therefore during comparison with the images from database 114. The machine learning algorithms can be trained using gaps in images, randomized dropout, image flips, warped images, etc. Generative Adversarial Networks (GANS) could also be used to support increasing database size.

[0046] It should be appreciated that each of the mobile device 102, 302, network 104,304 and / or the server 106, 306 may be embodied as a computing device / system. For example, in the illustrative embodiment, one or more of the mobile device 102, 302, network 103, 304, and / or the server 106, 306 may include a processing device and a memory having stored thereon operating logic for execution by the processing device for operation of the corresponding device.

[0047] Depending on the particular embodiment, the mobile device 102, 302, network104, 304 and / or the server 106, 306 may be embodied as a mobile computing device, server, desktop computer, laptop computer, tablet computer, notebook, netbook, Ultrabook™, cellular phone, smartphone, wearable computing device, personal digital assistant, Internet of Things (loT) device, control panel, router, gateway, and / or any other computing, processing, and / or communication device capable of performing the functions described herein.

[0048] The mobile device 102, 302, network 104, 304 and / or the server 106, 306 includes a processing device that executes algorithms and / or processes data in accordance with operating logic, an input / output device that enables communication between with one or more external devices, and memory which stores, for example, data received from the one or more external devices via an input / output device.

[0049] The input / output device allows the mobile device 102, 302, network 104, 304 and / or the server 106, 306 to communicate with the external device. For example, the input / output device may include a transceiver, a network adapter, a network card, an interface, one or more communication ports (e.g., a USB port, serial port, parallel port, an analog port, a digital port, VGA, DVI, HDMI, FireWire, CAT 5, or any other type of communication port orinterface), and / or other communication circuitry. Communication circuitry may be configured to use any one or more communication technologies (e.g., wireless or wired communications) and associated protocols (e.g., Ethernet, Bluetooth®, Wi-Fi®, WiMAX, etc.) to effect such communication depending on the particular mobile device 102, 302, network 104, 304 and / or the server 106, 306. The input / output device may include hardware, software, and / or firmware suitable for performing the techniques described herein.

[0050] The external device may be any type of device that allows data to be inputted or output from the mobile device 102, 302, network 104, 304 and / or the server 106, 306. For example, in various embodiments, the external device may be embodied as the mobile device 102, 302 and / or the server 106, 306. Further, in some embodiments, the external device may be embodied as another computing device, switch, diagnostic tool, controller, printer, display, alarm, peripheral device (e.g., keyboard, mouse, touch screen display, etc.), and / or any other computing, processing, and / or communication device capable of performing the functions described herein. Furthermore, in some embodiments, it should be appreciated that the external device may be integrated into the mobile device 102, 302, network 104, 304 and / or the server 106, 306.

[0051] The processing device may be embodied as any type of processor(s) capable of performing the functions described herein. In particular, the processing device may be embodied as one or more single or multi-core processors, microcontrollers, or other processor or processing / controlling circuits. For example, in some embodiments, the processing device may include or be embodied as an arithmetic logic unit (ALU), central processing unit (CPU), digital signal processor (DSP), and / or another suitable processor(s). The processing device may be a programmable type, a dedicated hardwired state machine, or a combination thereof. Processing devices with multiple processing units may utilize distributed, pipelined, and / or parallel processing in various embodiments. Further, the processing device may be dedicated to performance of just the operations described herein or may be utilized in one or more additional applications. In the illustrative embodiment, the processing device is of a programmable variety that executes algorithms and / or processes data in accordance with operating logic as defined by programming instructions (such as software or firmware) stored in memory. Additionally or alternatively, the operating logic for processing device may be at least partially defined by hardwired logic or other hardware. Further, the processing device may include one or morecomponents of any type suitable to process the signals received from input / output device or from other components or devices and to provide desired output signals. Such components may include digital circuitry, analog circuitry, or a combination thereof.

[0052] The memory may be of one or more types of non-transitory computer-readable media, such as a solid-state memory, electromagnetic memory, optical memory, or a combination thereof. Furthermore, the memory may be volatile and / or nonvolatile and, in some embodiments, some or all of the memory may be of a portable variety, such as a disk, tape, memory stick, cartridge, and / or other suitable portable memory. In operation, the memory may store various data and software used during operation of the computing device such as operating systems, applications, programs, libraries, and drivers. It should be appreciated that the memory may store data that is manipulated by the operating logic of processing device, such as, for example, data representative of signals received from and / or sent to the input / output device in addition to or in lieu of storing programming instructions defining operating logic. The memory may be included with the processing device and / or coupled to the processing device depending on the particular embodiment. For example, in some embodiments, the processing device, the memory, and / or other components of the mobile device 102, 302, network 104, 304 and / or the server 106, 306 may form a portion of a system-on-a-chip (SoC) and be incorporated on a single integrated circuit chip.

[0053] In some embodiments, various components of the mobile device 102, 302, network 104, 304 and / or the server 106, 306 (e g., the processing device and the memory) may be communicatively coupled via an input / output subsystem, which may be embodied as circuitry and / or components to facilitate input / output operations with the processing device, the memory, and other components of the computing system. For example, the input / output subsystem may be embodied as, or otherwise include, memory controller hubs, input / output control hubs, firmware devices, communication links (i.e., point-to-point links, bus links, wires, cables, light guides, printed circuit board traces, etc.) and / or other components and subsystems to facilitate the input / output operations.

[0054] The mobile device 102, 302, network 104, 304 and / or the server 106, 306 may include other or additional components, such as those commonly found in a typical computing device (e.g., various input / output devices and / or other components), in other embodiments. It should be further appreciated that one or more of the components of the mobile device 102, 302,network 104, 304 and / or the server 106, 306 described herein may be distributed across multiple computing devices. In other words, the techniques described herein may be employed by a computing system that includes one or more computing devices.

[0055] Referring now to FIG. 5, in use, the system 100, 300 (e.g., in conjunction with a user) may execute a method 500 for leveraging machine learning for detecting a fault condition of a part of a powertrain and / or vehicle. It should be appreciated that the particular blocks of the method 500 are illustrated by way of example, and such blocks may be combined or divided, added or removed, and / or reordered in whole or in part depending on the particular embodiment, unless stated to the contrary.

[0056] The illustrative method 500 begins with block 502 in which a user captures one or more images of or audible output from a part of a powertrain and / or vehicle using the mobile device 102, 302. For example, the user may capture image(s) of and / or audible output from the part in an operating state and / or non-operating state. The captured audible output can be used to generate an image associated with the audible output from the part, as discussed above. Machine learning can be used to identify the part based on the image(s) and a database of images of various parts.

[0057] In block 504, the mobile device 102, 302 transmits one or more images and / or captured audible output to the server 106, 306 for processing. In the illustrative embodiment, the mobile device 102, 302 may transmit the images to the server 106, 306 for processing due, for example, to the often computationally intensive nature of computer vision processing images and / or converting audio files to image files. However, it should be appreciated that, in other embodiments, the mobile device 102, 302 may perform some or all of the processing herein in reference to the server 106, 306 (e.g., in conjunction with or in the alternative to the server 106, 306 performing those processes).

[0058] In block 506, the network 104, 304 analyzes the image(s) received from the mobile device 102, 302 and / or server 106, 306 using machine learning. In block 508 the network 104, 304 generates an assessment of the part based on the machine learning algorithm and database of images of the part or images of audio output from the part. In doing so, in block 506, the network 104, 304 may analyze the image(s) using one or more computer vision, machine learning, and / or artificial intelligence techniques, such as those discussed above.

[0059] For example, in an embodiment, the machine learning algorithm of network 104 places the full image of the part captured by camera 108 through the machine learning algorithm by turning the image data into a multiple dimensional tensor. The tensor is then trained on using a visual transformer (ViT) network.

[0060] In another example, the audio data captured by microphone 308 is processed using mobile device 302 and / or server 306 to transform the audio data into a spectrogram and / or scalogram before being analyzed by machine learning in network 304. Network 304 uses computer vision-based machine learning algorithms to generate an assessment of the fault condition of the part based on the image of the audio data. For example, the machine learning workstation 312 can be configured to use computer vision techniques and vision transformer networks to analyze the image(s) created from the audio data.

[0061] In one embodiment, the machine learning employs an array of red-green-blue(RGB) image tensors representing each of the spectrogram / scalogram images in a three- dimensional format. The channels of the red, green and blue colors are individualized into their own table with the size of the table representing the number of pixels within the image.

[0062] In block 510, the server 106, 306 outputs the assessment of the fault condition to the mobile device 102, 302 for display to the user on display 110, 310. As indicated above, the mobile device 102, 302 may, additionally or alternatively, provide the assessment to the user via another mode of communication (e.g., speaker, tactile feedback, haptic feedback, etc.).

[0063] In some embodiments, the mobile device 102, 302 may prompt the user to provide additional feedback. For example, in some embodiments, the application 120, 320 of the mobile device 102, 302 may prompt the user to indicate whether the assessment generated by machine learning was correct, incorrect, incomplete, etc. This feedback can be used to train the machine learning algorithms in machine learning workstation 112, 312 to improve accuracy and robustness, and to correct the assessment of the part based on actual conditions observed in the field.

[0064] Although the blocks 502-510 are described in a relatively serial manner, it should be appreciated that various blocks of the method 500 may be performed in parallel in some embodiments. In addition, method 500 may be repeated by the user to provide additional assessments of the same part or of different parts.

[0065] The systems and methods disclosed herein provide near immediate feedback to the user about whether to replace, repair, re-use, or maintain a current condition of a part of the powertrain and / or vehicle. This allows the user to avoid strip downs and expensive evaluations of the powertrain and / or vehicle if the part is not faulty. Identification of the faulty parts also provides for tear downs that are directed only to the faulty parts.

[0066] Various aspect and embodiments are contemplated by the present disclosure. For example, one aspect includes a method of detecting a fault condition of a part of a powertrain and / or vehicle. The method includes generating at least one image associated with the part; analyzing the at least one image with machine learning and a database of images associated with the part; generating an assessment of the fault condition of the part based on the machine learning; and outputting the assessment.

[0067] In an embodiment, the at least one image is generated by a camera on a mobile device.

[0068] In a further embodiment, the method includes identifying the part using a first machine learning of a database of images of a plurality of different types of parts and the at least one image. The generating the assessment of the fault condition of the part is based on a second machine learning and the database of images associated with the part.

[0069] In yet a further embodiment, the method includes selecting a first machine learning algorithm for identifying the part; and selecting a second the second machine learning algorithm based on the identification of the part for analyzing the at least one image and generating the assessment of the fault condition.

[0070] In an embodiment, the assessment includes whether the part is to be repaired, replaced, or re-used in a current condition.

[0071] In yet a further embodiment, the assessment includes determining a cracked condition of the part, a wear condition of the part, a vibration condition of the part, a misfire condition of the part, oil degradation of the part, cleanliness of the part, foreign object detection on the part, and / or a non-genuine part identification of the part.

[0072] In an embodiment, the at least one image is generated based on audible output from the part captured by a microphone on a mobile device.

[0073] In yet a further embodiment, the at least one image is a spectrogram or scalogram generated from the audible output.

[0074] In an embodiment, the method includes transmitting the at least one image from a mobile device of a user to a server; analyzing the at least one image and generating the assessment of the fault condition in a machine learning network connected to the server; and transmitting the output of the assessment from the machine learning network to the server and from the server to the mobile device of the user. In a further embodiment, the server is a cloud server.

[0075] According to another aspect of the present disclosure, a system for determining a fault condition of a part of a powertrain and / or vehicle is provided. The system includes a mobile device configured to capture at least one image of the part and / or capture data associated with the part to generate the least one image during servicing of the powertrain and / or vehicle and a server comprising a processor and a memory having a plurality of instructions stored thereon. In response to execution by the processor, the instructions cause the processor to: receive the at least one image and / or the data associated with the part captured by the mobile device; transmit the at least one image to a machine learning network that analyzes the at least one image and generates an assessment of a fault condition of the part with machine learning and a database of images associated with the part; receive the assessment from the machine learning network; and output the assessment to the mobile device.

[0076] In an embodiment, the mobile device includes at least one camera operable to capture the least one image of the part. In an embodiment, the mobile device includes at least one microphone operable to capture the data associated with the part, and the data includes an audible output from the part and the at least one image is generated from the audible output.

[0077] In an embodiment, the assessment from the machine learning network includes whether the part is to be repaired, replaced, or re-used.

[0078] In an embodiment, the machine learning network is configured to identify the part using a first machine learning algorithm and a database of images of a plurality of different types of parts and the at least one image of the part.

[0079] In a further embodiment, the machine learning network is configured to generate the assessment of the fault condition using a second machine learning algorithm and the database of images associated with the part. In a further embodiment, the machine learning network is configured to select the second machine learning algorithm based on the identification of the part.

[0080] In an embodiment, the server is a cloud server. In an embodiment, the plurality of instructions cause the processor to receive feedback from a user of the mobile device regarding the assessment of the fault condition of the part. In an embodiment, the database of images associated with the part includes images of the part in non-faulty conditions and in various faulty conditions.

Claims

WHAT IS CLAIMED IS:

1. A method of detecting a fault condition of a part of a powertrain and / or vehicle, the method comprising: generating at least one image associated with the part; analyzing the at least one image with machine learning and a database of images associated with the part; generating an assessment of the fault condition of the part based on the machine learning; and outputting the assessment.

2. The method of claim 1, wherein the at least one image is generated by a camera on a mobile device.

3. The method of claim 2, further comprising identifying the part using a first machine learning of a database of images of a plurality of different types of parts and the at least one image, and wherein the generating the assessment of the fault condition of the part is based on a second machine learning and the database of images associated with the part.

4. The method of claim 3, further comprising: selecting a first machine learning algorithm for identifying the part; and selecting a second the second machine learning algorithm based on the identification of the part for analyzing the at least one image and generating the assessment of the fault condition.

5. The method of claim 1, wherein the assessment includes whether the part is to be repaired, replaced, or re-used in a current condition.

6. The method of claim 5, wherein the assessment includes determining a cracked condition of the part, a wear condition of the part, a vibration condition of the part, a misfire condition of the part, oil degradation of the part, cleanliness of the part, foreign object detection on the part, and / or a non-genuine part identification of the part.

7. The method of claim 1, wherein the at least one image is generated based on audible output from the part captured by a microphone on a mobile device.

8. The method of claim 7, wherein the at least one image is a spectrogram or scalogram generated from the audible output.

9. The method of claim 1, further comprising: transmitting the at least one image from a mobile device of a user to a server; analyzing the at least one image and generating the assessment of the fault condition in a machine learning network connected to the server; and transmitting the output of the assessment from the machine learning network to the server and from the server to the mobile device of the user.

10. The method of claim 9, wherein the server is a cloud server.

11. A system for determining a fault condition of a part of a powertrain and / or vehicle, the system comprising: a mobile device configured to capture at least one image of the part and / or capture data associated with the part to generate the least one image during servicing of the powertrain and / or vehicle; and a server comprising a processor and a memory having a plurality of instructions stored thereon that, in response to execution by the processor, causes the processor to: receive the at least one image and / or the data associated with the part captured by the mobile device; transmit the at least one image to a machine learning network that analyzes the at least one image and generates an assessment of a fault condition of the part with machine learning and a database of images associated with the part; receive the assessment from the machine learning network; and output the assessment to the mobile device.

12. The system of claim 11 , wherein the mobile device includes at least one camera operable to capture the least one image of the part.

13. The system of claim 11, wherein: the mobile device includes at least one microphone operable to capture the data associated with the part, the data including an audible output from the part; and the at least one image is generated from the audible output.

14. The system of claim 11, wherein the assessment from the machine learning network includes whether the part is to be repaired, replaced, or re-used.

15. The system of claim 11, wherein the machine learning network is configured to identify the part using a first machine learning algorithm and a database of images of a plurality of different types of parts and the at least one image of the part.

16. The system of claim 15, wherein the machine learning network is configured to generate the assessment of the fault condition using a second machine learning algorithm and the database of images associated with the part.

17. The system of claim 16, wherein the machine learning network is configured to select the second machine learning algorithm based on the identification of the part.

18. The system of claim 11, wherein the server is a cloud server.

19. The system of claim 11, wherein the plurality of instructions cause the processor to receive feedback from a user of the mobile device regarding the assessment of the fault condition of the part.

20. The system of claim 11, wherein the database of images associated with the part includes images of the part in non-faulty conditions and in various faulty conditions.

Images