SYSTEM AND METHOD FOR ASSESSING THE OPERATING CONDITION OF AN ASSET - Patent application

Abstract

An exemplary mobile device for assessing the operational state of an asset. The mobile device includes at least one sensor device for acquiring data related to one or more operational or environmental characteristics of an asset during operation. The mobile device further includes a processing unit encoded with a neural network architecture having one or more models trained to identify one or more operational states of the one or more assets according to asset type. The processing unit can determine the operational state of the asset by extracting features from the acquired data and comparing attributes of the extracted features with attributes of at least one known operational state determined through training of the model. The mobile device can also include an output interface for outputting results of the determination performed by the processing unit, and the output interface can include communicating the results to a remote device.

Description

[Technical field]

[0001] The present disclosure relates to determining an operational state of an asset using multiple sensor data. [Background technology]

[0002] Inspection and diagnosis of the health and condition of assets (such as mechanical systems, structures, structural components, humans and other objects, or the operational or environmental conditions of the asset) requires a significant amount of time, cost, and expertise, and failure to properly monitor assets can lead to asset degradation, inefficiency, downtime, or catastrophic failure, depending on the application. These labor-intensive activities, and the detriments caused by failing to perform these activities comprehensively or at all, can tax the resources of organizations that maintain numerous mechanical systems and have large amounts of infrastructure, for example. As a result, necessary inspections and diagnosis may not be sustained consistently, and ailing systems are not identified before they fail. Equipment and infrastructure failures can result in significant unnecessary costs or failure to achieve organizational goals, as well as potential injuries and fatalities. The present invention reduces the time, effort, and expertise required to assess and report on assets (such as the health of mechanical systems and infrastructure machines), allowing organizations to perform more comprehensive asset health assessments and reducing the likelihood of asset failure.

[0003] Sensor technology has been used to monitor equipment for many years. In existing implementations, sensors are typically fixed / stationary sensors that are physically set up to detect changes or out-of-spec conditions on a specific asset. Adding sensors to multiple assets requires duplication of the fixed sensor array and specific tuning for each individual asset or each component of an asset. This approach is often complex, bulky, and costly. Summary of the Invention

[0004] An exemplary mobile device for assessing the operational condition of an asset is disclosed, the device comprising at least one sensor device for acquiring data related to operational characteristics of the asset during operation or the operating environment of the asset; a processing device encoded with a neural network architecture having one or more models trained to identify one or more operational conditions of the asset by processing the acquired data depending on the type of sensor, the processing device configured to perform a determination of the operational condition of the target asset by extracting features from the acquired data and comparing attributes of the extracted features with attributes of at least one known operational condition determined through training of the one or more models; and an output interface configured to output a result of the determination performed by the processing device, which may include communicating the result to an external computing device.

[0005] An exemplary method for assessing an operational condition of an asset is disclosed, the method comprising: acquiring data relating to at least one of operational or environmental characteristics of an asset during operation via one or more sensors of a mobile device; extracting one or more features from the acquired data based on one or more of a sensor type and a sensor location, one or more operational conditions of the one or more sensors, and operational or environmental conditions having potential adverse effects on the one or more assets via a processing device encoded with a neural network architecture having one or more trained models, each having one or more layers trained to recognize at least one state and function of the one or more assets, extracting one or more features from the acquired data based on a type of sensor; comparing, via the processing device, attributes of the extracted features with at least one operational condition attribute known through training of the one or more trained models; determining, via the processing device, the operational condition of the asset based on a result of the at least one comparison; and outputting, via an output interface, a result of the determination, which may include communicating the result to an external computing device.

[0006] An exemplary computing device for assessing an operational condition of an asset is disclosed, the computing device comprising: a communication interface configured to receive sensor data associated with one or more assets from at least one remote sensor device over a network, the sensor data being associated with a plurality of sensor types; and a processing device encoded with a neural network architecture including one or more models, the processing device configured to train the one or more models based on the sensor data and identify at least one normal operational condition and at least one fault condition of the one or more assets as a function of the asset type based on the sensor data.

[0007] An exemplary method for assessing an operational condition of an asset is disclosed, the method comprising the steps of: receiving asset data from one or more mobile devices at a communication interface, the asset data including at least one sensor type and at least sensor data associated with the asset type and metadata associated with each of the one or more mobile devices; parsing the sensor data based on the respective sensor types at the one or more data interfaces; and processing the parsed data at a processing device encoded with a neural network architecture having one or more models to determine an operational condition of the one or more assets and training the model to identify at least one normal operational condition and one or more fault conditions of the one or more assets depending on the corresponding asset type, the sensor data, the associated sensor types, and the metadata.

[0008] The illustrative embodiments are best understood from the following detailed description when read in conjunction with the accompanying drawing figures. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 illustrates a system structure according to an exemplary embodiment of the present disclosure. [Diagram 2] FIG. 2 illustrates a mobile device and a processing server according to an exemplary embodiment of the present disclosure. [Diagram 3] FIG. 1 illustrates a method for assessing an operational condition of an asset by a mobile device according to an exemplary embodiment of the present disclosure. [Figure 4] FIG. 1 illustrates a method for evaluating an operational condition of an asset by a processing server according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Further areas of applicability of the present disclosure will become apparent from the detailed description provided herein. It should be understood that the detailed description of the exemplary embodiments is for purposes of illustration only and is not intended to necessarily limit the scope of the present disclosure.

[0011] Exemplary embodiments of the present disclosure are directed to systems and methods for analyzing the condition of an asset. Particularly useful are exemplary embodiments that enable use in remote or limited bandwidth environments. These embodiments of the method include obtaining sensor data, such as audio, image, and operational data, from an edge device and processing the data through a machine training model running on the device to assess the health of the associated asset when bandwidth is not available or limited in the operating environment. When sufficient bandwidth is available, processing can occur in the cloud or at a remote location. The results of the processing can be displayed to a user to build a diagnostic report or fed to other data systems and applications (e.g., augmented reality). The system can automatically link the identified condition to a maintenance manual and identify parts required to repair the asset. In particular, applications of the present invention can relate to the use of such models to identify potential or actual maintenance issues in mechanical systems (e.g., military systems, automobiles, fixed or mobile machines) and infrastructure (e.g., structural components, power lines, gas pipelines, bridges), as well as to report the inference results of the models.

[0012] FIG. 1 is a diagram illustrating a system structure according to an exemplary embodiment of the present disclosure.

[0013] As shown in FIG. 1, the system 100 includes a mobile device 110 for assessing the operational status of the asset 120. The mobile device 110 can include multiple sensor devices 112 for obtaining data regarding one or more operational characteristics of the asset 120 during operation (or its associated external or internal environmental characteristics such as wind conditions, oncoming ocean wave patterns, ambient and oncoming atmospheric pressure measurements, air density, pH or speed of fluid flow around the asset, depth of the asset in the fluid, cloud cover, rain, weight of objects interacting with the asset, environmental temperature, humidity, other objects within the radius of the asset, external sounds or noises, asset temperature, etc., etc., that may affect the operational characteristics of the asset). The operation of the asset 102 can include various operating modes in which the asset is stationary or in motion. The asset can include machines such as personal or industrial vehicles for locomotion by air, land, or water, any type of electromechanical engine, robotic machines, electromechanical valves, mechanical appliances, printing equipment, or any other object or device having mechanized motion, drive, and / or operating cycle. The mobile device 110 may be any portable device designed to be moved manually or autonomously between two locations and has the structural capability to support multiple sensor devices 112. For example, according to exemplary embodiments of the present disclosure, the mobile device 110 may be embodied in the form of a smartphone, an airborne or land vehicle, a handheld or portable electronic device or tool, or any other suitable electronic device that may be attached, fixed, or carried by a person or vehicle as needed.

[0014] The plurality of sensors 112 may include any combination of image sensors, acoustic sensors, motion sensors, heat sensors, gas sensors, humidity / moisture sensors, airflow sensors, any type of radiation sensors, carbon monoxide sensors, carbon dioxide sensors, airflow sensors, distance measurement sensors, pressure sensors, or any other suitable sensing devices for detecting physical or operational characteristics of the asset 102 or the asset's operating environment. According to another exemplary embodiment, the sensor devices may be configured to detect physiological characteristics of a human or animal. For example, the sensors may include heart rate sensors, pulse oxygen sensors, blood pressure sensors, or any other suitable sensors, or measurements of health characteristics, such as, for example, images of the body, including x-rays, CT, and MRI scans, ultrasound scans, electrocardiograms, photographs of the patient's body, etc., or various series of the above over time used by medical professionals to determine the health of a patient. The sensor devices 112 may be integrated into or attached to the housing 116 of the mobile device 110. According to another example embodiment, the sensor device 112 may include a port 118 for connecting to an external sensor device 112 via a direct wired or wireless connection or an adapter.

[0015] The mobile device 110 may include one or more data interfaces 114 configured to convert the acquired sensor data into a specified format depending on the type of sensor. The data interface may include a combination of hardware and software. For example, according to an exemplary embodiment, the sensor device 112 may be configured as an image sensor to detect electromagnetic radiation, including visible light or infrared, radio frequency, microwave, or other relevant signal emissions by the asset 102 as needed. The image sensor may be configured to generate video or still images from the detected electromagnetic radiation. The sensor device 112 may also be configured as an acoustic sensor to detect sound or audio recordings, and a motion sensor, such as a gyroscope, speed sensor, or accelerometer to detect vibration, acceleration, velocity, or overall motion data. The one or more data interfaces 114 are configured to process the sensor data by converting the data into a specific format required for analysis. For example, the conversion process may be specific to the type of sensor data and the type of sensor, and may include operations such as image whitening, audio signal separation, sample rate normalization, or any other suitable operation or process to normalize a particular signal as needed. According to an exemplary embodiment of the present disclosure, a transformation process performed by one or more data interfaces 114 can be used to generate an abstract (e.g., theoretically or mathematically derived) representation of the sensor data. For example, acoustic and vibration data can be processed with a spectrogram transform prior to further analysis.

[0016] According to an example embodiment, the sensor device 112 may also be configured to obtain data related to the environment surrounding the target asset 102. For example, the sensor device may be configured to obtain weather information, dew point, humidity, air temperature, wind speed, wind direction, air composition, or any other suitable environmental condition as desired.

[0017] As shown in FIG. 1, the mobile device may include a processing unit 120 enclosed within the housing 116. The processing unit 120 may be encoded with a neural network architecture 122 having one or more models trained to identify one or more operational states of the plurality of assets 102 according to the asset type. The processing unit 120 may be configured to determine the operational state of the target asset 102 by extracting features from the transformed data and comparing attributes of the extracted features to attributes of known operational states determined through training of one or more models. For example, the processing unit may apply a weighting kernel (e.g., a matrix) to the extracted features according to the type of sensor and determine whether the operational state includes one or more of a normal state, a mechanical failure, a structural failure, an electrical failure, or a potential adverse operating environment based on a comparison of the kernel with one or more elements of the extracted features. The one or more models of the neural network architecture 122 may be trained using data selected to adjust each model to be sensitive to a wide range of problems in mechanical systems and infrastructures as well as to recognize changing conditions of operation of these mechanical systems (e.g., high and low efficiency operation). Based on the determined operating conditions, the neural network architecture 122 may identify one or more components associated with repairing one or more of the mechanical faults, the structural faults, and the electrical faults.

[0018] The neural network architecture 122 receives sensor data from one or more data interfaces 114 that are converted into a condensed representation. The conversion process and method are related to model performance. According to an exemplary embodiment of the present disclosure, the audio and vibration feature conversion in particular includes a spectrogram conversion for input to one of the multiple models of the neural network architecture 122, such as a convolutional model. The spectrogram conversion includes that the audio data and vibration data are converted to have the characteristics of image data. For example, the audio data or vibration data can be parsed into smaller overlapping segments and converted from the time domain to the frequency domain. The frequency domain values ​​are converted to polar coordinates to obtain the magnitude and phase of different frequencies. The frequency domain data is sampled using a fast Fourier transform to convert the polar coordinate data to the time domain. The resulting time domain data can be used to show the changes in different frequencies over time.

[0019] The mobile device 110 includes an output interface 124 configured to output the results of decisions performed by the processing unit. The output interface 124 is integrated into the housing 116 and may include any suitable display or output device that allows a user or operator to receive and possibly input information. The output interface 124 may include one or more of a graphical display device, a printer, an audio output device, one or more light emitting diodes (LEDs), an external memory device, an external computing device, or other suitable electronic device or output type, as appropriate. For example, according to an exemplary embodiment, the output interface 124 may be configured to display images and text and allow commands or information to be input into the mobile device 110 by manipulating graphical icons, graphical objects, virtual keys, or any other suitable method of providing interactive control or commands, as appropriate.

[0020] As shown in FIG. 1, the mobile device 110 may include a communication interface 126 integrated into and / or enclosed by the housing 116. The communication interface 126 is configured to communicate with remote devices via a communication or data network 140. According to an exemplary embodiment of the present disclosure, the communication interface 126 may be configured to transmit transformed data to a remote computing device via the network 140, which may include one or more other mobile devices 150 and a processing (e.g., central) server 160. According to another exemplary embodiment, the communication interface 126 may be configured to receive updated models of the neural network architecture 122 from the processing server 160, and raw sensor data, transformed sensor data, or operational state determinations from other mobile devices 150 via the network 140. The communication interface 126 may include hardware components such as an antenna, a network interface (e.g., an Ethernet card), a communication port, a PCMCIA slot and card, or other suitable components or devices as desired. The hardware and software components of the communication interface 126 may be configured to receive data, e.g., images, according to one or more communication protocols and data formats. For example, the communication interface 126 may be configured to communicate over a network 140, which may include a local area network (LAN), a wide area network (WAN), a wireless network (e.g., Wi-Fi), a cellular network, a satellite network, the Internet, optical fiber, coaxial cable, infrared, radio frequency (RF), or any combination thereof. During a receive operation, the communication interface 126 may be configured to identify portions of the received data via a header and parse the data signal and / or data packet into smaller frames (e.g., bytes, words) or segments for further processing by the processing unit 120.

[0021] The mobile device 110 may also include a memory device 128 including a first memory structure 130 and a second memory structure 132. The memory device 128 may be enclosed in the housing 116 or may be configured to have either the first or second memory structures 130, 132 external to the housing 116. The first memory structure 130 is configured to store raw sensor data and / or transformed data of a plurality of sensors 112 integrated with or attached to the mobile device 110. The first memory structure 130 is also configured to store raw sensor data and / or transformed data received by the communication interface 126 from another mobile device 150 via the network 140. The second memory structure 132 may be configured to store a model of the neural network architecture 122, which may include an updated model of the neural network architecture 122 received by the communication interface 126 from the processing server 160 via the network 140. The updated model may include changes based on raw or transformed data sent by the mobile device 110 over the network to the processing server 160, depending on the application.

[0022] According to an exemplary embodiment of the present disclosure, the mobile device 110 may include a navigation system 170. The navigation system 170 may include a combination of hardware and software components capable of providing automatic or manual navigation or positioning of the mobile device 110 relative to specified coordinates and relative to the asset 102. According to an exemplary embodiment, the mobile device 110 configured with the navigation system 170 may include a propulsion system for moving the mobile device 110 on land, air, or water. For example, the navigation system 170 may include program code or software stored in the second memory structure 132 of the memory device 128 and executed by the processing unit 120. According to an exemplary embodiment, the mobile device 110 may include at least one other processing unit 180 for executing the program code or software stored in the second memory structure 132 for command and control of the navigation system 170. For manual navigation, the mobile device 110 may be configured to receive command and / or control signals from the remote control device 155 via the communication interface 126 over the network 140. According to an exemplary embodiment, the communication interface 126 may also be configured to transmit data of one or more of the sensors 112 to the remote control device 155, receive commands to position the mobile device 110 relative to the asset 102, and / or receive one or more sensor commands from the remote control device 155 to orient one or more of the sensors 112 to obtain data related to an operational condition of the asset 102. The processing unit 120 may communicate sensor data, such as image data, to the remote control device 155 to provide a visual orientation and position of the mobile device 110 to an operator. The navigation system 170 may also be configured for autonomous positioning of the mobile device 110 relative to the asset 102.The navigation system 170 can receive data acquired by one or more of the sensors 112 and control the motion system of the mobile device 110 to a position to acquire data from the asset 102 via the multiple sensors 112. Additionally, based on the data acquired from the one or more sensors 112, the navigation system 170 can be configured to autonomously orient the one or more sensors 112 to acquire data related to an operational condition of the asset 102.

[0023] The mobile device 110 may include a communications infrastructure 125, including a bus, message queues, a network, a multi-core message passing scheme, etc., for connecting the communications interface 126, the data interface 114, the processing unit 120, the memory device 128, and the output interface 124 for communication of data and control signals.

[0024] 1, the processing server 160 may include a communication interface 162 configured to communicate with the plurality of mobile devices 104 over the network 140. According to an example embodiment, the processing server 160 may be configured to receive asset data associated with one or more assets 102 from at least one of the plurality of mobile devices 104 over the network 140. The asset data may include raw sensor data (or converted sensor data, as appropriate) generated by a plurality of sensors of each mobile device 104. For example, the converted sensor data may include sensor data converted into a format specified by each mobile device 104 depending on the type of sensor. The asset data may also include a determination of an operational state of the one or more assets 102 and metadata associated with the mobile device 110. The metadata of the mobile device 110 may include the location of the mobile device 110 (e.g., geospatial coordinates), the operating system executed by the mobile device 110, the hardware configuration of the mobile device 110, application software executable by the mobile device 110, the signal strength of the network 140 when the asset data was communicated, the battery level of the mobile communication device 110, the time the asset data (e.g., sensor data) was obtained, the weather (e.g., temperature, humidity, air pressure, dew point, etc.), information regarding a registered user or owner of the mobile device 110, or any other suitable data or information as appropriate. It should be appreciated that portions of the metadata may be obtained through any number of separate applications or program code stored on and / or executed by the processing device 120 or stored in volatile or non-volatile portions of the second memory structure 132 and / or memory device 128.

[0025] Communications interface 162 may be comprised of the same hardware and software components as described above with respect to communications interface 126 of mobile device 110, and may be configured to connect to any of the same networks or communications links discussed with respect to communications interface 126.

[0026] The processing server 160 may include one or more data interfaces 164 for converting the raw sensor data received by the communication interface 162. The data interface 164 may include a combination of hardware and software and may be configured to process and / or manipulate the raw sensor data or the converted sensor data as appropriate for the application, with respect to the data interface 114 of the mobile device 110.

[0027] The processing server 160 may include a processing unit 166 encoded with a neural network architecture 163 including one or more models, the processing unit 166 configured to train the one or more models based on the asset data to identify one or more normal operating conditions and one or more fault conditions of the one or more assets according to the asset type. According to another exemplary embodiment, if the asset data includes raw sensor data, the processing unit 164 is configured to determine an operating condition of the one or more assets 102 based on the transformed data generated by the one or more data interfaces 164. According to yet another exemplary embodiment, if the asset data includes the transformed sensor data and a determination of an operating condition of the one or more assets 102, the processing unit 166 is configured to train the one or more models of the neural network architecture 163 based on the determined conditions to generate an updated model. The processing unit 164 transmits the updated model to the mobile device 110 via the network 140.

[0028] The processing server 160 may include a memory device 168 having a first memory structure 170 configured to store raw or transformed sensor data, operational state determinations, and metadata received from the plurality of mobile devices 104. The memory device 168 also includes a second memory structure 172 configured to store one or more models associated with the neural network architecture 163. The first memory structure 170 may also be configured to store any additional information related to the plurality of assets 102. The additional information may be requested and / or accessed over the network 140 from manufacturer, industry, public, or government databases via the communication interface 162 of the processing device 166.

[0029] The processing device 166 may be configured to generate one or more of an operating condition trend analysis, a maintenance schedule recommendation and plan, a parts and inventory order, a repair plan, an inventory status report, or other suitable reports as appropriate, for a specified type or identified asset based on the information stored in the first memory structure 170 of the memory device 168. The reports may be generated automatically based on a specified schedule or may be generated in response to a request from at least one of the plurality of mobile devices 104. The reports may be communicated to the requesting mobile device 110 or may be broadcast to the plurality of mobile devices 104 via the communication interface 126 according to a desired schedule.

[0030] The processing server 160 may include an output interface 174 that may be connected to receive data from the processing unit 166 via an input / output (I / O) interface 176. The I / O interface 176 may be configured to receive signals from the processing unit 166 and generate data signals suitable for output by the output interface 174. The I / O interface 176 may include a combination of hardware and software, such as, for example, a processor encoded with program code, software, and / or firmware for communicating with the output interface 174 via a direct wired or wireless link, a circuit card, or any other suitable hardware device. The output interface 174 may include one or more of a graphical display device, a printer, an audio output device, one or more light emitting diodes (LEDs), an external memory device, an external computing device, or other suitable electronic devices or output types as appropriate.

[0031] The processing server 160 may include a communications infrastructure 165, including buses, message queues, networks, and multi-core message passing schemes, to connect the communications interface 162, the data interface 164, the processing unit 166, the memory device 168, and the output interface 174 for communication of data and control signals.

[0032] The processing server 160 is configured to aggregate assessments of equipment and infrastructure by the mobile devices 104, as well as assessment information and other data from a collection of electronic devices, databases, and mobile devices that report over the network 140 either continuously or periodically when the mobile devices 104 achieve an appropriate level of data connectivity with the network 140 if connectivity is unavailable or limited at the time sensor data related to the asset's operational condition is collected. Once the assessments from any number of the multiple mobile devices 104 have been aggregated and stored in the memory device 168 of the processing server 160, the processing server 160 can generate one or more reports that provide a high level of analysis and recommendations based on the aggregated content of the assessments from the mobile devices 104.

[0033] FIG. 3 illustrates a method for assessing the operational condition of an asset. The method of FIG. 3 can be executed by each of a plurality of mobile devices 104. In step 300, the mobile device 110 activates one or more of a plurality of sensors 112 to acquire data related to a physical quality of the asset 102 during operation. According to an exemplary embodiment of the present disclosure, acquiring the sensor data can include one or more of the plurality of sensor devices 112 generating a video or still image of the asset 102 or detecting sound radiation, temperature, gas emissions, and moisture for the asset 102. The acquired data is converted into a format specified depending on the type of sensor using one or more data interfaces 114 (step 302). The converted data is stored in a first memory structure 130 of the memory device 128 (step 304). If the mobile device 110 has suitable connectivity to the network 140 (step 306), the converted data can be transmitted to the remote computing device 104, 160 via the communication interface 126 (step 308). Otherwise, if network connectivity is limited or unavailable, the processing device 120 extracts one or more features from the transformed data based on the type of sensor (step 310) and applies one or more weighting kernels to the extracted features to compare with attributes of known operating conditions determined through training of one or more trained models (step 312). The weighting kernels include attributes of known operating conditions. The processing device 120 determines an operating condition of the asset 102 based on the results of the comparison (step 314), including identifying one or more of a mechanical fault, a structural fault, and an electrical fault. Once the operating condition is determined, the processing device 120 can identify one or more parts associated with repairing one or more of the mechanical fault, the structural fault, and the electrical fault (step 316).An output of the determination result is provided to a user via the output interface 124 by compiling the converted data, the operational state determination result, and the metadata associated with the mobile device, and generating a report based on the compiled data (step 318). When connectivity to the network 140 is adequate or restored, the mobile device 110 can transmit its metadata, the operational determination, and the converted sensor data as asset data to the processing server 140 via the communication interface 126 (step 320). In response to this communication, the mobile device 110 receives an updated model of the neural network architecture 122 from the processing server 140 via the communication interface 126 (step 322). The updated model is stored in the second memory structure 132 of the memory device 128 (step 324).

[0034] 4 illustrates a method for evaluating the operational status of an asset, according to an exemplary embodiment of the present disclosure. In step 400, the processing server 160 receives asset data from one or more mobile devices 104 over the network 140 via the communication interface 126. The asset data includes sensor data associated with the applicable sensor type and metadata associated with each mobile device 110 of the one or more mobile devices 102. The sensor data is analyzed in one or more data interfaces 164 based on the type of each sensor of the mobile device 110 that acquired the data (step 402). The one or more data interfaces 164 convert the analyzed sensor data into a format specified according to the respective sensor type (step 404). The processing unit 166 of the processing server 160, encoded with a neural network architecture 163 having one or more models, processes the converted data to determine the operational status of one or more assets 102 and trains one or more models to identify normal operational status and multiple fault conditions of the one or more assets according to the asset type (step 406). The one or more updated models resulting from the training are transmitted to the mobile device 110 via the communication interface 162 over the network 140. The processing server 160 may receive asset data including an assessment of the operational condition of each asset from one or more other mobile devices 104 via the communication interface 162 over the network 140 and aggregate the asset data by storing it in a first memory structure 170 of the memory device 168 (step 408). The processing device 166 may generate one or more of an operational condition trend analysis, a maintenance schedule recommendation and plan, a parts and inventory order, a repair plan, and an inventory status report based on the information stored in the first memory structure of the memory device in association with the specified type or identified asset (step 410). The processing device 166 may transmit one or more of the generated reports to one or more of the mobile devices 104 via the communication interface 162 over the network 140 (step 412).

[0035] Computer program code for performing the specialized functions described herein may be stored on a computer usable medium, which may refer to memory, such as memory devices of the mobile device 104 and the processing server 160, which may be memory semiconductors (e.g., DRAM, etc.). These computer program products may be tangible, non-transitory means for providing software to the mobile device 104 and the processing server 160. Computer programs (e.g., computer control logic) or software may be stored in memory devices 155 resident on the mobile device 104 or the processing server 160. Also, computer programs may be received via the respective communication interfaces 126, 162. Such computer programs, when executed, may enable the mobile device 104 and the processing server 160 to perform the methods and exemplary embodiments discussed herein. Thus, such computer programs may represent controllers of the mobile device 104 and the processing server 160. If the present disclosure is implemented using software, the software may be stored in a computer program product or non-transitory computer readable medium and loaded onto the mobile device 104 and processing server 160 using a removable storage drive, interface, hard disk drive, or communication interface, and such software may be loaded onto the mobile device 104 and processing server 160, if applicable.

[0036] The processing units 120, 166 of the mobile device 104 and the processing server 160 may each include one or more modules or engines configured to perform the functions of the exemplary embodiments described herein. Each of the modules or engines may be implemented using hardware, and in some examples, may also utilize software, such as corresponding to program code and / or programs stored in memory. In such examples, the program code may be compiled by the respective processor (e.g., by a compilation module or engine) prior to execution. For example, the program code may be source code written in a programming language that is translated into a lower level language, such as assembly language or machine code, for execution by one or more processors and / or any additional hardware components. The process of compilation may include using lexical analysis, preprocessing, syntax analysis, semantic analysis, syntax-directed translation, code generation, code optimization, and any other techniques deemed suitable for translating the program code into a lower level language suitable for controlling the mobile device 104 and the processing server 160 to perform the functions disclosed herein. As a result of such processes, it will be apparent to those skilled in the relevant art that the mobile device 104 and processing server 160 are specially configured computing devices uniquely programmed to perform the functions described above.

[0037] It will be understood by those skilled in the art that the present invention can be embodied in other specific forms without departing from the spirit or essential characteristics thereof. The presently disclosed embodiments are therefore considered in all respects to be illustrative and not restrictive. The scope of the present invention is indicated by the appended claims, rather than the foregoing description, and it is intended to embrace all changes that come within the meaning, scope, and equivalents thereof. [Explanation of symbols]

[0038] 102 Assets 104 Mobile Devices 112 Sensors 140 Network 155 Memory Devices 160 Processing Server

Claims

1. 1. A mobile device for assessing an operational condition of an asset, comprising: at least one sensor device for acquiring data relating to operational characteristics of the asset during operation and attributes of an operating environment associated with the asset; a processing device programmed with a neural network having one or more models trained to identify one or more operational states of the asset by processing acquired data according to a type of sensor, the processing device being configured to perform a determination of the operational state of the asset by extracting features from the acquired data and comparing attributes of the extracted features with at least one known operational state attribute determined through training of the one or more models and attributes of the operating environment associated with the asset during operation; an output interface configured to output a result of the determination performed by the processing unit, the output interface may include communicating the result to an external device; Equipped with The processor is configured to assign a weight to the extracted features of each sensor based on a sensor type, a sensor location, and associated operational and environmental conditions of the sensor and the asset, and to determine whether the operational condition includes one or more of a normal condition, a mechanical failure, a structural failure, an electrical failure, and a hostile operating environment based on one or more of the extracted features.

2. 2. The mobile device of claim 1, comprising one or more data interfaces configured to convert the acquired data into one or more specified data formats depending on the type of sensor and the neural network for analysis by the processing unit depending on the type of sensor.

3. a memory device including a memory structure configured to store the acquired data or the converted data of at least one of the sensors; a communications interface configured to transmit the captured data or the converted data over a network to a remote computing device; The mobile device of claim 2 , comprising:

4. the at least one sensor includes one or more of an image sensor, an acoustic sensor, a motion sensor, a heat sensor, a gas sensor, a humidity sensor, a radio frequency sensor, and a fluid flow sensor; the communication interface is configured to receive an updated model of the neural network from the remote computing device; the memory device is configured to store a model of the neural network, including the received updated model, in the memory structure; the updated model includes changes based on at least the acquired data or the transformed data transmitted over the network to the remote computing device; the image sensor is configured to generate one or more of a video image and a still image; The mobile device of claim 3 , wherein the image sensor is configured to detect infrared, non-visible light, or visible light.

5. a housing for enclosing at least one or more of a sensor interface, the processing unit, and a graphical interface; The mobile device of claim 1 , wherein at least one sensor is integrated into the housing or connected to the housing via an associated one of the one or more sensor interfaces.

6. the mobile device is a vehicle; The vehicle, a navigation system configured to autonomously position the vehicle relative to the asset based on data obtained from the at least one sensor. The mobile device of claim 3.

7. the mobile device is a vehicle; The vehicle, a navigation system configured to position the vehicle relative to the asset; the communication interface is configured to transmit data obtained from one or more of the at least one sensor over a network to determine a location and position of the vehicle, and to receive navigation commands from a remote control device to position the vehicle relative to the asset and / or orient one or more of the at least one sensor. The mobile device of claim 3.

8. The processing device is configured to identify one or more components associated with repairing one or more of the mechanical failure, the structural failure, and the electrical failure, or identify an action to address the adverse operating environment. The mobile device of claim 1 .

9. 1. A method for assessing an operational condition of an asset, comprising: acquiring data relating to at least one of operational or environmental characteristics of the asset during operation via one or more sensors of the mobile device; extracting one or more features from the acquired data based on one or more of a sensor type and / or a sensor location, one or more operational states of the one or more sensors, operational or environmental conditions having a potential adverse effect on the one or more assets, or based on a sensor type, via a processing device programmed with a neural network having one or more trained models, each having one or more layers trained to recognize at least one state and function of the one or more assets; assigning, via the processing unit, a weight to the extracted features of each sensor based on sensor type, sensor location, and associated operational and environmental conditions of the sensor and the asset; determining, via the processing device, whether the operating condition comprises one or more of a normal condition, a mechanical failure, a structural failure, an electrical failure, and a hostile operating environment based on one or more of the extracted features by comparing attributes of the extracted features with attributes of at least one operating condition that is known through training of the one or more trained models; determining, via the processing device, an operational state of the asset based on a result of the at least one comparison; outputting a result of the determination via an output interface, which may include communicating the result to an external computing device; and A method comprising:

10. converting the acquired data via one or more sensor interfaces into a specified data format for analysis by the processing device according to the type of sensor and neural network; storing the acquired or converted data of the one or more sensors in a memory structure of a memory device; transmitting the acquired data or the converted data via a communication interface over a network to a remote computing device; 10. The method of claim 9, comprising: receiving an updated model for the neural network from a remote computing device via a communications interface; storing a model of the neural network including the received updated model in a memory structure of a memory device; 10. The method of claim 9, comprising:

12. The step of acquiring data comprises: generating a video or still image via said one or more sensors; or detecting via said one or more sensors sound emissions, temperature, gas emissions, humidity, motion, speed, acceleration, reflected light, heat, and fluid flow associated with said asset; 10. The method of claim 9, comprising:

13. The step of extracting one or more features comprises: The method of claim 10 , further comprising applying one or more weighting kernels to the acquired data or the transformed data, the weighting kernels including attributes of corresponding known operating conditions.

14. The step of determining an operating state includes: identifying, via the processing device, one or more components associated with an action to repair one or more of the mechanical faults, the structural faults, and the electrical faults, or to address the potentially adverse environmental or operating condition; 10. The method of claim 9, comprising:

15. The step of outputting a result of the determination includes: compiling, via the processing device, the captured data or the transformed data, the determination of the operating state, and metadata associated with the mobile device; generating, via the processing unit, a report based on the compiled data; The method of claim 10, comprising:

16. 1. A computing device for assessing an operational condition of an asset, comprising: a communications interface configured to receive sensor data relating to one or more assets from at least one remote sensor device over a network, the sensor data being associated with a plurality of sensor types; a processing device programmed with a neural network including one or more models, the processing device being configured to: train the one or more models based on the sensor data to extract features from asset sensor data for each sensor; assign weights to the extracted features for each sensor based on a sensor type, a sensor location, and associated operational and environmental conditions of the sensor and the asset; and determine and identify at least one of a normal operating state, a mechanical fault, a structural fault, an electrical fault, and a harmful operating environment of the one or more assets based on one or more of the extracted features according to a type of the asset based on the sensor data; A computing device comprising:

17. the sensor data includes raw sensor data for one or more sensor types; The computing device comprises: a data interface configured to convert at least a portion of the raw sensor data into one or more specified data formats according to the associated sensor type and the neural network for analysis by the processing unit; 20. The computing device of claim 16, wherein the processing unit is configured to determine an operational state of the one or more assets based on the transformed sensor data.

18. 17. The computing device of claim 16, further comprising: a computing device configured to generate one or more updated models by training the one or more models; and wherein the communications interface is configured to transmit the one or more updated models over the network to the at least one remote device.

19. the received sensor data includes asset data related to determining an operational state of the one or more assets and metadata related to the at least one remote sensor device; The computing device comprises:

20. The computing device of claim 16, comprising: a memory device including a first memory structure configured to store operational state determinations associated with a plurality of assets including the one or more assets and the metadata of a plurality of remote devices including the at least one remote sensor device.

20. 20. The computing device of claim 19, wherein the processing unit is configured to generate one or more of operating condition trend analyses, maintenance schedule recommendations and plans, parts and inventory ordering, repair plans, and inventory status reports associated with specified types of assets or identified assets based on the information stored in the first memory structure of the memory device.

Images